ogkalu2/comic-translate

What it solves

It provides an automated way to translate comics, manga, and webtoons from various languages (including English, Korean, Japanese, French, Chinese, Russian, German, Dutch, Spanish, and Italian) into other languages, overcoming the limitations of traditional machine translation tools that often struggle with distant language pairs.

How it works

The tool uses a multi-stage pipeline to process comic pages:

1. Text Detection: An RT-DETR-v2 model identifies speech bubbles and text segments.
2. OCR: It extracts text using specialized tools like manga-ocr (Japanese), Pororo (Korean), and PPOCRv5 for other languages, with optional support for Gemini 2.0 Flash and Azure Vision.
3. Inpainting: It removes original text using a Manga/Anime fine-tuned LaMa checkpoint or an AOT-GAN based model.
4. Translation: It utilizes state-of-the-art LLMs (such as GPT-4, Claude, and Gemini) to translate the extracted text, feeding the LLM the entire page's text for better context.
5. Text Rendering: The translated text is wrapped and rendered back into the original bounding boxes.

Who it’s for

Readers of international comics and manga who want to translate content in languages they do not understand, as well as users who prefer the high-quality translations provided by LLMs over standard machine translation.

Highlights

• LLM-Powered Translation: Leverages GPT-4 and similar models for superior translation quality, especially for complex language pairs.
• Multi-Modal Support: Can provide the image itself to the LLM for additional translation context.
• Comprehensive Pipeline: Integrates detection, OCR, inpainting, and rendering in one application.
• Flexible Deployment: Available as a desktop app for Windows and macOS, or as a browser extension for Chromium-based browsers.
• Manual Correction Mode: Allows users to manually fix OCR or cleaning errors when automatic mode fails.