R3gm/SoniTranslate

What it solves

SoniTranslate simplifies the process of translating videos into different languages while keeping the audio synchronized. It provides a comprehensive pipeline to handle transcription, translation, and voice generation (dubbing) within a single user-friendly web interface.

How it works

The application uses a Gradio-based web UI to orchestrate several AI models and tools:

• Transcription: Uses Whisper-based models (including faster-whisper and WhisperX) to convert speech to text.
• Translation: Translates the transcribed text into a target language using tools like deep-translator or OpenAI's GPT API.
• Voice Generation: Employs various Text-to-Speech (TTS) engines such as Piper, Coqui XTTS, BARK, and Facebook-mms to generate the translated audio. It also supports voice imitation via OpenVoiceV2 and RVC models.
• Processing: Uses FFmpeg for multimedia handling and pyannote-audio for speaker diarization (identifying who is speaking).

Who it’s for

Content creators, educators, and anyone needing to localize video content for a global audience without manually coordinating multiple separate transcription and dubbing tools.

Highlights

• Extensive Language Support: Supports a vast array of languages for both transcription and translation.
• Voice Cloning & Imitation: Integration with Coqui XTTS and OpenVoiceV2 allows for realistic voice cloning and imitation.
• Flexible Input/Output: Supports YouTube playlists, local files, and URLs; outputs can be video, audio, or subtitles (including ASS and soft subtitles).
• Customizable Pipeline: Allows users to choose their preferred ASR model, TTS engine, and translation method (including OpenAI API integration).
• Accessibility: Available via a local installation or a Google Colab notebook.