jianchang512/pyvideotrans

What it solves

It automates the complex process of translating videos from one language to another. Instead of manually transcribing, translating, and re-recording audio, this tool provides a unified workflow to handle speech recognition, subtitle translation, and AI dubbing while keeping audio and video synchronized.

How it works

The project implements a multi-stage pipeline:

1. Speech Recognition (ASR): Converts spoken audio into text subtitles using models like Faster-Whisper or various cloud APIs.
2. Subtitle Translation: Translates the text into the target language using LLMs (such as DeepSeek, ChatGPT, or local Ollama models).
3. Speech Synthesis (TTS): Generates new audio in the target language, supporting multi-role assignment and voice cloning (via F5-TTS, CosyVoice, etc.).
4. Video Synthesis: Merges the new audio and translated subtitles back into the original video.

Who it’s for

It is designed for content creators, developers, and anyone needing to localize video content efficiently, whether they prefer a simple Windows executable, a command-line interface for servers, or a web-based UI.

Highlights

• End-to-End Automation: One-click workflow from original video to translated dubbed video.
• Voice Cloning: Integration with advanced TTS models for zero-shot voice cloning.
• Speaker Diarization: Ability to distinguish between different speakers to assign unique AI voices to each role.
• Flexible Deployment: Supports local offline execution, cloud APIs, Docker, and a pre-packaged Windows .exe.
• Interactive Editing: Allows users to manually proofread and edit text at the recognition, translation, and dubbing stages.