cyberofficial/Synthalingua

What it solves

Synthalingua is a self-hosted tool designed to break language barriers in real-time audio and video. It allows users to transcribe and translate audio from various languages into English (or other multilingual outputs) in near real-time, specifically targeting use cases like watching live streams (e.g., VTubers) or processing video files with synchronized captions.

How it works

The tool leverages AI models—including Whisper, FasterWhisper, and OpenVINO—to process audio input from microphones, HLS streams (YouTube, Twitch), or local files. It utilizes both GPU (Nvidia CUDA, AMD ROCm) and CPU resources for transcription and translation. It includes a web-based video player for real-time captioning and can export subtitles in SRT or VTT formats.

Who it’s for

It is primarily for hobbyists, students, and fans of foreign-language content (such as VTuber viewers) who want to understand live broadcasts or videos without relying on professional translation services.

Highlights

• Real-time Translation: Supports live HLS streams, microphone input, and local files.
• Interactive Video UI: A web-based player with synchronized captions and customizable styling.
• Flexible Hardware Support: Runs on CPU, Nvidia GPUs (CUDA), and Intel iGPU/dGPU/NPU.
• Noise & Spam Filtering: Includes a blocklist and repetition suppression to filter out hallucinated or unwanted phrases.
• Integration Options: Can send translation results to Discord via webhooks or display them via a local web server.
• Subtitle Generation: Capable of burning subtitles into video or embedding them as tracks, with optional vocal isolation using Demucs.