buxuku/SmartSub

SmartSub (妙幕) – One‑stop desktop tool for subtitles, translation and AI dubbing

What it does

• Takes a video (or an online link from YouTube, B‑站, etc.) and runs the whole pipeline:
  1. Download the source video and any existing subtitles.
  2. Speech‑to‑text (ASR) using local models such as whisper.cpp, faster‑whisper, FunASR, Qwen3‑ASR, FireRedASR, NVIDIA Parakeet, or optional cloud ASR services.
  3. Translate the generated subtitles with up to 20 translation back‑ends (free Bing/Google APIs, Baidu, Azure, DeepL‑X, Ollama, Gemini, etc.) – you can output pure translation or a bilingual “original + translation” subtitle.
  4. Proofread the subtitles in an interactive editor, with optional AI‑assisted polishing.
  5. Text‑to‑speech (TTS) and voice cloning – local models (Kokoro, VITS, ZipVoice) or cloud services (Edge TTS, OpenAI‑compatible, Azure Speech, ElevenLabs, etc.) turn the subtitle text into an audio track, optionally matching your own voice via zero‑shot cloning.
  6. Render the final video: hard‑burn subtitles into the picture or mux them as soft tracks, with full styling (fonts, colors, shadows, positioning) and real‑time preview.

All steps can be run independently or chained together for batch processing.

Why it matters

• Privacy‑first: The core ASR, translation (via Ollama) and TTS engines run entirely on the user’s machine; no audio or video leaves the computer unless you deliberately enable a cloud service.
• Hardware‑aware: Supports CPU, NVIDIA CUDA, AMD/Intel Vulkan, and Apple Core ML/Metal acceleration. The app ships its own GPU‑acceleration packs, so you don’t need to install CUDA yourself.
• Free‑to‑run: A fully functional workflow is possible without any paid API keys – local models are downloaded once, and the built‑in free translation/TTS services have no usage limits.
• Extensible: You can add your own OpenAI‑style API endpoints for translation, ASR, or TTS, and the UI lets you tweak request parameters without touching code.

Key features at a glance

Feature	Details
Video download	One‑click download via yt‑dlp (YouTube + 1800+ sites) or lux (Chinese platforms). Handles cookies, batch links, and auto‑matches official subtitles.
ASR engines	8 built‑in options, mixing local (whisper.cpp, faster‑whisper, FunASR, Qwen3‑ASR, FireRedASR, NVIDIA Parakeet) and cloud services (OpenAI, ElevenLabs Scribe, Deepgram, etc.).
AI subtitle polishing	Semantic re‑segmentation + batch correction (punctuation, homophones, filler removal) using a local LLM via Ollama.
Translation	20 providers, free Bing/Google, Chinese cloud vendors, and large‑model APIs (Gemini, DeepSeek, Azure OpenAI, etc.). Supports custom parameter JSON.
Proofreading UI	Side‑by‑side video and subtitle view, undo/redo, per‑line delete‑restore, one‑click AI rewrite.
TTS & voice cloning	Local Kokoro (103 voices) & VITS (174 Chinese voices) – free, offline. ZipVoice enables zero‑shot cloning from a short reference clip. Cloud options include Edge TTS, OpenAI, Azure, ElevenLabs, etc.
Subtitle rendering	Hard‑burn (permanent) or soft‑mux (switchable) with full style editor; live preview.
Cross‑platform	Windows x64, macOS (Apple Silicon & Intel), Linux x64. Install via GitHub releases or Homebrew (brew install --cask smartsub).
GPU acceleration	Automatic detection and download of CUDA, Vulkan or Core ML packs; fallback to CPU if needed.

Typical use cases

• Learning – add bilingual subtitles to foreign‑language lectures or tutorials.
• Content creators – generate subtitles and dubbed audio for YouTube or B‑站 videos without paying for third‑party services.
• Podcast/meeting archiving – batch‑transcribe audio files to searchable SRT files.
• Personal voice dubbing – clone your own voice and produce a fully narrated version of a video.

Getting started

1. Download the appropriate installer for your OS (GPU packs are optional).
2. Run the app, follow the onboarding wizard to fetch a speech model (or configure a cloud ASR).
3. Drag‑drop a video or paste a URL, set source/target languages, and press Start. After transcription you can edit, translate, TTS‑synthesize and finally export the finished video.

Community & contribution

• Open‑source under the MIT license; contributions via Issues and Pull Requests are welcome.
• Build locally with yarn install and yarn dev; native dependencies (whisper addon, sherpa‑onnx) are fetched automatically.
• Sponsorship options (DigitalOcean, Alipay/WeChat donations, QQ group) are listed in the README.

---

Bottom line: SmartSub bundles a complete, privacy‑preserving AI workflow for turning any video into a subtitled, translated and dubbed product, all from a single cross‑platform desktop application.