agan-j/xiaoniu

What it solves

Xiaoniu AI Video Translation is designed to help content creators, particularly those producing short dramas and promotional videos for platforms like TikTok, YouTube, and Douyin, overcome language barriers for global distribution. It solves the problem of tedious manual translation, unnatural voice-overs, and the lack of synchronization between translated audio and the original video's lip movements.

How it works

The tool employs a multi-step AI pipeline to transform videos into multiple languages:

1. Translation Pipeline: It uses a proprietary translation model trained on 1 million video subtitles, utilizing a "5-step method" (Core Understanding $\rightarrow$ Contextual Translation $\rightarrow$ Cultural Adjustment $\rightarrow$ Reflection/Adjustment $\rightarrow$ Final Proofreading) to ensure natural, culturally relevant phrasing.
2. Audio Processing: The system separates vocals from background music, clones the original speaker's voice (using models like CosyVoice, IndexTTS, and ElevenLabs), and generates new translated audio that retains the original emotion and tone.
3. Visual Alignment: It uses deep learning to synchronize the newly generated audio with the speaker's lip movements and overall video pace, preventing the "slow-motion" effect often seen in automated dubbing.
4. Editing Interface: A Web UI allows creators to manually refine subtitles and audio in real-time while watching the video.

Who it’s for

• Short Drama Creators: Those exporting content to international markets who need high-fidelity voice cloning and lip-syncing for multiple characters.
• Enterprise Marketers: Businesses promoting products on global social media platforms.
• ** uma Video Content Creators:** Users who need to translate YouTube videos or local files into multiple languages with professional-grade dubbing.

Highlights

• High-Fidelity Voice Cloning: Supports emotional cloning and multi-character identification (up to 100 characters) to maintain distinct voices in complex scenes.
• Lip-Sync Technology: Automatically adjusts audio speed and visual frames to ensure the translated speech matches the mouth movements.
• Advanced Audio Separation: Removes background noise and separates vocals from music, allowing the translated voice to be overlaid on the original BGM.
• Broad Language Support: Translates and dubs into dozens of languages, including English, Japanese, French, Korean, Spanish, Portuguese, Russian, Arabic, and more.
• Integrated LLM Support: Compatible with Gemini, DeepSeek, and GPT engines for high-accuracy semantic translation.