BallonsTranslator: a deep learning-assisted comic translation tool with automated inpainting and typesetting

What it solves

BallonTranslator is a deep learning-assisted tool designed to automate the translation of comics and manga. It solves the tedious process of manually detecting text in images, removing the original text (inpainting), translating the content, and typesetting the translated text back into the original speech bubbles while attempting to preserve the original layout, color, and alignment.

How it works

The tool integrates several AI modules to create a full pipeline:

• Text Detection & OCR: Uses models like manga-image-translator, manga_ocr, and PaddleOCRVLManga to locate text blocks and recognize characters (supporting Japanese, English, and Chinese).
• Image Inpainting: Employs deep learning models (AOT, lama) or traditional algorithms (PatchMatch) to erase original text and restore the background.
• Translation: Supports a wide array of translation engines, including LLMs (OpenAI-compatible APIs, Sakura-13B-Galgame), specialized translators (DeepL, Sugoi, Papago), and general services (Google, Caiyun).
• Typesetting: Automatically estimates the original layout (angle, orientation, alignment) to place translated text back into the bubbles.

Who it’s for

It is primarily for manga/comic translators, scanlation groups, and readers who want to translate comics from Japanese or English into other languages (particularly Chinese) with minimal manual effort.

Highlights

• One-click translation: Automates the entire pipeline from detection to rendering.
• Rich editing suite: Includes a WYSIWYG text editor, mask editing, and a repair brush for fine-tuning the inpainting results.
• Flexible translation backends: Supports both cloud APIs and local offline LLMs for privacy or cost reasons.
• Multi-platform support: Available for Windows, macOS, and Linux with GPU acceleration for Nvidia and Apple Silicon.
• Headless mode: Can be run via command line for batch processing without the GUI.