Huanshere/VideoLingo

What it solves

VideoLingo automates the complex process of translating and dubbing videos. It eliminates the need to manually synchronize subtitles, translate audio, and generate voiceovers, providing a streamlined pipeline to convert a video from one language to another while maintaining coherence and timing.

How it works

The project uses a multi-stage AI pipeline integrated into a Streamlit interface:

1. Transcription: It downloads videos (via yt-dlp) and performs word-level speech recognition and alignment using Qwen3-ASR and Qwen3-ForcedAligner.
2. Translation: It employs LLMs (via OpenAI-compatible APIs) to translate text, utilizing custom and AI-generated terminology to ensure consistency. It can also perform natural rewriting and reflection to improve quality.
3. Segmentation: The tool uses NLP and AI to segment subtitles based on configurable length limits for better readability.
4. Dubbing: It generates synthetic speech using various TTS providers such as GPT-SoVITS, OpenAI, Edge TTS, and F5-TTS to create dubbed audio tracks.

Who it’s for

Content creators, translators, and anyone looking to localize video content for a global audience without needing professional studio software or manual transcription services.

Highlights

• End-to-end workflow: Combines downloading, transcription, translation, segmentation, and dubbing in one tool.
• Flexible AI Backend: Supports multiple LLM providers, ASR engines (Qwen3, ElevenLabs, MAI), and a wide array of TTS options.
• AI-powered coherence: Uses terminology management and reflection-based rewriting to ensure translations are natural and contextually accurate.
• ** uma-click setup**: Offers a one-click installer for Windows and Docker support for Linux NVIDIA environments.