agan-j/xiaoniu

小牛视频翻译 是一款支持本地视频翻译、字幕翻译和 YouTube 视频翻译下载的 AI 工具，集成自动语音识别与多语言翻译功能，助力创作者高效完成视频翻译，应用于视频本地化与视频出海场景。

What it solves

Xiaoniu AI Video Translation is designed to help content creators, particularly those producing short dramas and promotional videos for platforms like TikTok, YouTube, and Douyin, overcome language barriers for global distribution. It solves the problem of creating high-quality localized versions of videos by automating the translation of speech and subtitles while maintaining the original audio atmosphere and visual synchronization.

How it works

The tool uses a multi-stage AI pipeline to process videos:

  1. Translation Pipeline: It employs a proprietary "5-Step Method" (Core Understanding $\rightarrow$ Contextual Translation $\rightarrow$ Cultural Adjustment $\rightarrow$ Reflection/Adjustment $\rightarrow$ Final Proofreading) and a custom-trained translation model based on 1 million video subtitles to ensure natural, context-aware translations.
  2. Audio Processing: It separates vocals from background music, clones the original speaker's voice (using models like CosyVoice, IndexTTS, and ElevenLabs), and generates new speech in the target language.
  3. Visual Synchronization: It uses deep learning to align the generated audio with the speaker's lip movements (lip-sync) and adjusts audio rates to match the visual pace without slowing down the video frames.
  4. Subtitle Management: It automatically generates and translates subtitles, offering a Web UI for real-time manual correction of text and speech.

Who it’s for

  • Short Drama Creators: Those exporting content to international markets who need precise multi-character voice cloning and lip-syncing.
  • Enterprise Marketers: Businesses promoting products on global social media platforms.
  • Video Editors: Users needing tools for subtitle erasure, voice-over generation, and high-fidelity audio purification.

Highlights

  • High-Fidelity Voice Cloning: Supports advanced cloning models (ElevenLabs, CosyVoice, IndexTTS) with emotional expression and multi-character identification.
  • Lip-Sync Technology: Precisely synchronizes translated audio with mouth movements and visual rhythms.
  • Proprietary Translation Model: A custom-tuned model trained on a million subtitles to avoid literal translations and handle cultural nuances.
  • Comprehensive Audio Tooling: Includes background music separation, noise reduction, and AI-driven waveform fingerprint repair.
  • Multi-Platform Support: Works with local videos and YouTube content, providing a Web UI for fine-tuning.
  • Cross-Platform Availability: Available for Windows (CPU/GPU) and Mac (Intel/M-series).