vibevoice-community/VibeVoice

What it solves

VibeVoice is designed to generate expressive, long-form, multi-speaker conversational audio (such as podcasts) from text. It overcomes common TTS challenges like maintaining speaker consistency over long durations and managing natural turn-taking between multiple speakers.

How it works

The framework uses continuous speech tokenizers (Acoustic and Semantic) operating at a very low frame rate (7.5 Hz) to maintain audio quality while increasing efficiency. It combines a Large Language Model (LLM) to handle textual context and dialogue flow with a diffusion head to generate high-fidelity acoustic details via a next-token diffusion framework.

Who it’s for

It is intended for creators of long-form audio content, such as podcast producers, and developers building real-time TTS applications.

Highlights

• Long-form synthesis: Capable of generating audio up to 90 minutes long.
• Multi-speaker support: Supports up to 4 distinct speakers in a single session.
• Real-time capability: Includes a specialized 0.5B streaming model for low-latency TTS.
• Voice cloning: Supports cloning voices from audio samples (in the 1.5B and 7B models).
• Fine-tuning: Community-supported implementations for adapting the model to new languages or voices.