jianchang512/pyvideotrans

pyVideoTrans – Quick Overview

What it does
pyVideoTrans is an open‑source Python application that lets you take a video in one language and automatically produce a version in another language. It strings together three AI‑driven steps:

1. Speech‑to‑text (ASR) – extracts a transcript from the original audio.
2. Machine‑translation – translates that transcript into the target language.
3. Text‑to‑speech (TTS) dubbing – synthesises a new audio track (optionally with different voices for each speaker) and re‑combines it with the original video.

The tool also supports:

• Batch subtitle generation (SRT) with speaker diarisation.
• Voice‑cloning models for a more natural dubbing voice.
• A GUI (WebUI/desktop) and a command‑line interface for automation or server‑side use.
• Local offline operation (using models like Faster‑Whisper, F5‑TTS, etc.) or cloud APIs (OpenAI, Azure, Google, etc.).

---

Core Features (as described in the README)

Feature	Details
One‑click video translation	ASR → translation → TTS → video synthesis in a single workflow.
Audio transcription / subtitle generation	Batch conversion to SRT, supports speaker diarisation.
Multi‑role AI dubbing	Different synthetic voices can be assigned per speaker.
Voice cloning	Integrates F5‑TTS, CosyVoice, GPT‑SoVITS for zero‑shot cloning.
Model support	• ASR: Faster‑Whisper (local), OpenAI Whisper, Alibaba Qwen, ByteDance Volcano, Azure, Google, etc. • Translation: DeepSeek, ChatGPT, Claude, Gemini, MiniMax, Ollama (local), Alibaba Bailian, etc. • TTS: Edge‑TTS (free), OpenAI, Azure, MiniMax, ChatTTS, etc.
Interactive editing	Pause at any stage to manually correct transcripts, translations, or dubbing.
Utility toolkit	Vocal separation, video/subtitle merging, AV alignment, transcript matching.
CLI	Headless operation for batch jobs or server deployment.
WebUI	Browser‑based interface (Gradio) for remote or internal‑network use.
Docker	Official Dockerfile for containerised deployment.
GPU acceleration	NVIDIA CUDA 12.8 + cuDNN 9.11; optional AMD via Whisper.NET.

---

How to Get Started

Windows (pre‑packaged)

1. Download the latest release from the Releases page.
2. Unzip to a folder without Chinese characters or spaces.
3. Run sp.exe. (GPU use requires CUDA/cuDNN installed.)

Source Installation (macOS / Linux / Windows developers)

1. Install Python 3.10+, FFmpeg, and uv (a fast package manager).
2. Clone the repo and sync dependencies:

   git clone https://github.com/jianchang512/pyvideotrans.git
   cd pyvideotrans
   uv sync   # add --all-extras for optional channels
   

3. Launch:
   • GUI: uv run sp.py
   • CLI: uv run cli.py … (see docs/cli.md for full options)
   • WebUI: uv run webui.py
   • Docker: docker build -t pyvideotrans-webui . && docker run -d -p 7860:7860 pyvideotrans-webui

---

Typical Use Cases

• Content creators who want to produce multilingual versions of tutorials, webinars, or podcasts without manual subtitle work.
• Enterprises needing internal video translation for training or compliance across regions.
• Researchers / hobbyists experimenting with voice‑cloning or multi‑speaker dubbing pipelines.
• Batch processing of large video libraries via the CLI or Docker container.

---

Limitations & Considerations

• Quality depends heavily on the chosen ASR, translation, and TTS models; offline models may be slower but keep data private.
• Some features (e.g., Whisper.NET for AMD GPUs) require extra setup steps.
• The software is GPL‑v3 licensed; any redistribution must also be open‑source under compatible terms.
• Users are responsible for complying with the terms of any third‑party APIs they call (e.g., OpenAI, Azure, etc.).

---

Where to Find More Information

• Official docs: https://pyvideotrans.com (tutorials, API config, FAQ)
• Community Q&A: https://bbs.pyvideotrans.com (error‑log analysis, tips)
• GitHub Wiki: architecture, extending translators, CLI/WebUI docs.
• Sponsor links: Recall.ai (meeting transcription API) and Infistar (multi‑model gateway) are mentioned for optional API usage.

---

TL;DR

pyVideoTrans is a full‑stack, open‑source video‑translation suite that combines speech‑recognition, machine‑translation, and voice‑synthesis (including voice cloning) into an easy‑to‑use GUI, WebUI, CLI, or Docker container. It supports both local models and a wide range of cloud APIs, making it suitable for anyone needing automated multilingual dubbing or subtitle generation.