ciddwd/overlay-translator

无需 ROOT 的开源 Android 屏幕实时翻译工具，适合游戏、视觉小说和漫画。支持端侧与云端 OCR、离线 LLM、多种翻译服务和文字朗读（TTS），译文可直接显示在画面上。Open-source no-root Android real-time screen translator for games, visual novels, and manga. Supports on-device and cloud OCR, offline LLMs, multiple translation services, on-screen translations, and text-to-speech (TTS).

Screen Translator (屏译) – What It Is

Screen Translator is an Android app that lets you translate any text that appears on your phone’s screen in real‑time. It works by taking a screenshot, running OCR (optical character recognition) to extract the words, sending the text to a translation engine (which can be a local model, a cloud LLM, or a classic MT service), and then overlaying the translated result on top of the original image. The overlay can be a floating window or a block that sticks to the original text, and the app can also read the translation aloud.

Core Features (as described in the README)

Feature	How It Works
Screen capture	Uses Android’s MediaProjection API or the Shizuku service to grab the screen without needing root.
OCR (text extraction)	Multiple back‑ends: on‑device ML Kit, PaddleOCR (v5/v6), a specialized manga OCR, local HTTP services (Umi‑OCR, LunaTranslator), or cloud OCR (PP‑OCRv6, Baidu, Tencent, Youdao). The app auto‑chooses an engine that supports the detected source language.
Translation	Supports a wide range of engines: on‑device models (Sakura, Hy‑MT2), Google ML Kit, DeepL (official or self‑hosted deeplx), OpenAI‑compatible LLMs (ChatGPT, Claude, DeepSeek, etc.), Anthropic‑compatible services, and other commercial APIs (Baidu, Tencent, Volcano, MiniMax, MiMo).
Overlay display	Two rendering modes – BLOCKS (text blocks placed directly over the original words) or a draggable floating window. The overlay can automatically adapt colors, fonts, and layout to match the source image, even handling vertical Japanese manga panels and speech‑bubble detection.
Text‑to‑speech (TTS)	Uses the phone’s built‑in TTS, a self‑hosted voice service, or cloud providers (Volcano, MiniMax, MiMo). Users can read the original text, the translation, or dictionary entries.
Word‑level lookup	By dragging a selection box ("word‑select mode") you can get a dictionary card that shows translations, pronunciation, part‑of‑speech, synonyms, example sentences, and explanations (the richer data comes from LLMs).
Batch processing	Images can be imported from the gallery or shared from other apps; the app creates a task that processes many screenshots in the background, saving results and optionally exporting images with the translated text baked in.
Loop / auto‑translate	A loop mode repeatedly captures the screen at a fixed interval or when the text stabilises, skipping identical frames and prioritising dialogue regions.
Customizable UI	Themes (light/dark/follow‑system), colour pickers, font import (`.ttf`), button‑order editing, and per‑app translation presets.
Privacy & offline support	All OCR and translation can run entirely on the device; cloud services are optional. Sensitive settings (API keys, prompts) are stored encrypted.

Who Might Use It?

Gamers who want live subtitles for foreign‑language games.
Manga/visual‑novel readers needing on‑the‑fly translation of vertical Japanese text.
Language learners who want quick dictionary look‑ups while reading on a phone.
Developers or hobbyists who prefer an open‑source, self‑hostable pipeline (they can point the app at their own OCR or translation servers).

Technical Highlights

Written in Kotlin 2.x with Jetpack Compose UI.
Minimum Android version: API 26 (Android 8.0).
Modular pipeline: OCR, translation, and TTS are independent; you can mix‑and‑match local or remote components.
Model handling: Supports on‑device model download with background progress, resumable transfers, and automatic thread‑count scaling (up to 6 CPU threads).
Accessibility: Basic TalkBack support for the floating UI elements.

Getting Started (per the README)

Download the latest ScreenTranslator‑x.y.z.apk from the Releases page (arm64‑v8a only). Verify the SHA‑256 checksum if desired.
Install the APK (allow unknown sources) and grant overlay and notification permissions.
Start the screen capture service inside the app, switch to the target game/comic, and tap the floating ball to translate the whole screen or enter word‑select mode.
Optional: enable the accessibility shortcut (volume + & – pressed together for 0.3 s) for a global trigger, or install Shizuku to avoid the MediaProjection permission prompt.

Why It’s Relevant to AI/ML

Screen Translator is essentially a mobile AI pipeline: it combines state‑of‑the‑art OCR models (ML Kit, PaddleOCR, manga‑specific ONNX models) with large language models or traditional MT services, and optionally a TTS voice model. All of these components are configurable, and the project is open‑source under Apache‑2.0, making it a useful reference for anyone building on‑device AI or hybrid cloud‑edge translation systems.

TL;DR: Screen Translator is an open‑source Android application that captures the screen, runs OCR, translates the extracted text with either local models or cloud LLMs, and overlays the result (or reads it aloud). It is fully configurable, works offline, and targets gamers, manga readers, and language learners.