dmMaze/BallonsTranslator

BallonsTranslator – AI‑assisted manga translation tool

What it is – A desktop application that helps you translate comic‑book panels (Japanese, Chinese, English, etc.) by combining OCR, machine‑translation, and image‑inpainting. It lets you run a one‑click translation, then fine‑tune the result with visual and text editors.

Core capabilities

Feature	What it does
One‑click machine translation	Detects speech balloons, extracts the text, runs OCR, sends the text to a chosen translator (Google, DeepL, OpenAI‑compatible LLMs, Sugoi, Sakura‑13B, etc.) and writes the translated text back into the image, preserving font size, colour, orientation and alignment as best it can.
Image editing	Mask‑based erasing and a repair brush (AOT, PatchMatch, LAMA) to remove original characters and fill the background before re‑embedding the translation.
Rich text editing	WYSIWYG editor with font‑style presets, effects, deformation, find‑replace, and import/export of Word documents.
LLM‑enhanced translation	Optional context window that feeds previous pages and a user‑supplied terminology table to the LLM, reducing inconsistency of names and terms. Token‑budget controls how many past pages are kept.
Visual context & summarisation	When using a vision‑capable LLM, the model can look at the panel image; it can also generate per‑page summaries and a project‑level “memory” that is re‑used as context for later pages.
Batch / CLI mode	A head‑less command‑line interface (python -m ballontranslator --headless …) for automated processing of many folders.

How it works (high‑level pipeline)

1. Text detection – Detects speech‑bubble bounding boxes. Supports YSG‑Yolo, CTD, and the Stariver “团子” cloud OCR service. |
2. OCR – Uses models from manga‑image‑translator (including manga_ocr, PaddleOCR‑VL‑For‑Manga) to read Japanese/English/Chinese characters and optionally extract colour information. |
3. Translation – Sends the extracted strings to a selected translator. Users can pick free APIs (Google, DeepL, Papago, etc.) or self‑hosted LLMs (OpenAI‑compatible, Sakura‑13B, Sugoi). |
4. Inpainting – Removes the original text with an inpainting model (AOT, PatchMatch, LAMA) to create a clean background. |
5. Re‑embedding – Renders the translated text back onto the panel, using the original layout cues (angle, alignment, colour). |
6. Post‑edit – Users can tweak the result with the visual brush, rectangle eraser, or the rich‑text editor.

Installation & platforms

• Windows – One‑click PowerShell installer (install.ps1) or a pre‑built zip (Ballonstranslator_win_minium.zip). Requires Windows 10+ (no support for Windows 7 unless you install Python 3.8 manually). |
• macOS / Linux – One‑click Bash installer (install.sh). |
• GPU acceleration – Detects Nvidia CUDA or Apple‑silicon GPUs automatically; optional AMD ROCm support via ZLUDA (requires manual driver setup). |
• Dependencies – Relies heavily on the manga-image-translator project for detection, OCR, and inpainting models. Additional optional models are downloaded on first use.

Extensibility

• Add new translators – Implement a subclass of the base translator and expose two methods; PRs are welcomed. |
• Custom terminology tables – Load a UTF‑8 .json, .txt, or .tsv file; the tool will inject matching terms into LLM prompts. |
• Modular auto‑modules – Each stage (detector, OCR, translator, inpainting) can be toggled on/off from the settings panel.

Who might use it

• Hobbyist translators who want to speed up manga localisation.
• Small studios needing a quick prototype for comic translation.
• Researchers experimenting with multimodal LLMs on comic‑panel data.

Maturity & community

• Actively maintained (install scripts, issue tracker, Telegram & QQ groups). |
• Relies on several external models that must be downloaded separately, so initial setup can be heavy. |
• Documentation includes multilingual READMEs and detailed guides for OCR services, GPU setup, and adding new translators.

---

Bottom line – BallonsTranslator is a genuine AI‑driven desktop tool that chains together computer‑vision (text detection & inpainting) and large‑language‑model translation to automate comic‑book localisation, while still offering manual editing for high‑quality results.