zhao-kun/VibeVoiceFusion

What it solves

VibeVoiceFusion is a full-stack web application designed to make high-quality, multi-speaker synthetic speech generation and voice cloning accessible. It removes the need for coding knowledge to perform complex tasks like multi-speaker dialog synthesis, custom voice adaptation via LoRA fine-tuning, and memory-optimized inference on consumer-grade GPUs.

How it works

The project is built on Microsoft's VibeVoice model, which uses a hybrid architecture combining an autoregressive (AR) language model backbone (Qwen) with a diffusion-based refinement head.

1. Processing: Text is tokenized and reference voice samples are encoded into embeddings.
2. Generation: The AR model generates speech tokens, which a DPM-Solver-based diffusion head then converts into high-quality audio waveforms.
3. Optimization: To run on consumer hardware, it implements Float8 quantization (reducing model size from 14GB to 7GB) and dynamic layer offloading, which moves transformer layers between the CPU and GPU to save VRAM.
4. Customization: It supports LoRA (Low-Rank Adaptation) fine-tuning, allowing users to train the model on specific voice datasets to achieve better style transfer and voice customization.

Who it’s for

• Content Creators: For producing podcasts, audiobooks, or narrated articles with multiple distinct voices.
• AI Enthusiasts: Users who want to clone voices and experiment with TTS (Text-to-Speech) without writing code.
• Developers: Those looking for a complete, Docker-ready deployment of a voice generation system.

Highlights

• Multi-Speaker Dialog Editor: A visual editor with drag-and-drop reordering for complex conversations.
• VRAM Optimization: Multiple offloading modes (Balanced, Aggressive, Extreme) allowing the model to run on GPUs with as little as 10GB VRAM.
• LoRA Training Suite: Integrated dataset management and training system with real-time metrics and TensorBoard integration.
• Bilingual Support: Full UI and backend support for English and Chinese.
• Batch Generation: Ability to generate 2-20 audio variations in a single batch using different random seeds.