Huanshere/VideoLingo

What it solves

VideoLingo simplifies the complex process of translating and dubbing videos. It automates the workflow of downloading videos, transcribing speech, translating text with AI, segmenting subtitles for readability, and generating dubbed audio, all within a single interface.

How it works

The tool uses a pipeline of specialized AI models and services:

• Transcription: Uses WhisperX for word-level speech recognition and alignment.
• Translation: Employs LLMs (via OpenAI-compatible APIs) to translate text, utilizing custom terminology and a reflection process to ensure natural rewriting.
• Subtitle Management: Applies NLP and AI-powered segmentation to ensure subtitles fit length limits and are logically split.
• Dubbing: Integrates various Text-to-Speech (TTS) providers including GPT-SoVITS, Azure, OpenAI, and Edge TTS to generate voiceovers.
• Interface: Provides a Streamlit-based UI for managing the process, including task control (pause/resume) and model selection.

Who it’s for

Content creators and users who want to translate foreign-language videos into their own language with high-quality subtitles and optional dubbed audio.

Highlights

• End-to-end workflow: Combines YouTube downloading, transcription, translation, and dubbing in one app.
• AI-powered segmentation: Uses NLP to ensure subtitles are segmented naturally rather than just by time.
• Terminology Control: Supports custom and AI-generated terminology to maintain consistency across the video.
• Flexible TTS options: Supports a wide range of dubbing engines, including local options like GPT-SoVITS and cloud APIs.
• Progress Resumption: Detailed logging allows users to resume tasks from where they left off.