buxuku/SmartSub

SmartSub (妙幕)

Summary SmartSub is an open‑source desktop application (Electron‑based) that provides an end‑to‑end pipeline for video subtitles: download online videos, transcribe speech to text, translate subtitles, proofread, generate AI dubbing (including voice cloning), and burn or embed subtitles into the final video. It runs locally on Windows, macOS, and Linux, with optional cloud‑based services for transcription, translation, and TTS.

Purpose Enable users to turn any video or audio file into subtitles, translate them into other languages, optionally dub the video with synthetic or cloned voices, and produce a final video with hard‑coded or soft subtitles—all without needing to juggle multiple separate tools.

Key Features

• Online video download: paste B‑Station, YouTube, etc. links; uses yt‑dlp and lux engines; can fetch platform‑provided subtitles.
• Transcription (speech‑to‑text): 8 engines selectable per job – whisper.cpp (built‑in), faster‑whisper, FunASR, Qwen3‑ASR, FireRedASR, NVIDIA Parakeet, local Whisper CLI, or cloud ASR from 8 providers (OpenAI‑compatible, ElevenLabs Scribe, Deepgram, Volcengine, Tencent Cloud, Alibaba Cloud, iFlytek, Gladia). Local engines run fully offline; models can be imported manually.
• AI subtitle refinement: optional semantic re‑segmentation and batch correction via local LLMs (Ollama) or rule‑based fallback.
• Translation: 20 services including free Bing/Google fallbacks, Baidu, Aliyun, Tencent, iFlytek, Volcengine, Doubao, XiaoNiu, DeepLX, Azure, Google, Ollama, DeepSeek, Gemini, Tongyi Qianwen, SiliconFlow, Azure OpenAI, DeerAPI; supports custom OpenAI‑style endpoints and per‑service parameter tuning.
• Proofreading UI: side‑by‑side video preview, sentence‑level editing, undo/redo, AI one‑click polish.
• TTS & voice cloning: local offline engines – Kokoro (103 multilingual voices), VITS (174 Chinese voices), ZipVoice zero‑shot voice cloning from a short reference audio; cloud options – Edge TTS, OpenAI‑compatible, Azure Speech, Volcengine Doubao, ElevenLabs. Includes time‑axis alignment, speed control, ducking, per‑line preview, and export as wav/mp3, replaced audio track, mixed video, or dual‑audio MKV.
• Subtitle burning / muxing: hard‑coded subtitles (ffmpeg burn‑in) or soft subtitles (MKV track); full styling (font, size, color, outline, shadow, nine‑grid positions, presets) with real‑time preview.
• Hardware acceleration: GPU‑backed transcription via NVIDIA CUDA, AMD/Intel Vulkan, or Apple Core ML/Metal; automatic fallback to CPU.
• Privacy‑first: all local processing keeps files on the machine; cloud services are opt‑in and show a privacy confirmation on first use.
• Batch processing & workflow chaining: each step can be run independently or linked together; concurrency limits adjustable.

How It Works

1. Input – drag‑and‑drop a local media file or paste a URL; the app can download the video and any existing subtitles.
2. Transcribe – choose an engine; if a local model is selected, the audio is processed offline with the chosen Whisper‑based or sherpa‑onnx model; cloud ASR uploads audio to the user‑configured endpoint.
3. Translate – pick a translation service; the app sends the subtitle text (or original+translated pair) to the service, using free fallbacks or user‑provided API keys.
4. Proofread – open the built‑in editor to view the video alongside subtitles, correct timing or text, and apply AI‑driven polishing.
5. TTS – select a voice engine; for each subtitle line the app synthesizes audio, aligns it to the subtitle’s timing (using speed pre‑control, post‑hoc length check, and silent‑gap borrowing), and outputs a synchronized audio track.
6. Export – combine the original/video, the new audio track (or mixed audio), and the subtitles via ffmpeg: either burn subtitles into the video frames or mux them as a selectable subtitle stream. All steps can be performed sequentially in a wizard‑style pipeline or individually as needed.

Limitations / Considerations

• High‑accuracy transcription relies on larger models (e.g., Whisper large) which need more disk space, RAM, and GPU memory for fast processing.
• GPU acceleration requires the corresponding hardware; the app downloads acceleration packs internally but will fall back to CPU if loading fails.
• Cloud‑based transcription, translation, or TTS services require API keys and may incur usage fees or have rate limits; the app does not cover those costs.
• Voice cloning quality depends on the clarity and length of the reference audio; the app performs basic quality checks but cannot guarantee perfect results.
• The app does not provide real‑time live captioning; it works on pre‑recorded files.
• Some cloud services (e.g., Alibaba Cloud transcription) are commercial‑only with no free trial, as noted in the README.

All details above are taken directly from the provided README; no external information was added.