vibevoice-community/VibeVoice

VibeVoice: Expressive, longform conversational speech synthesis. (Community fork)

What it solves

VibeVoice is designed to generate expressive, long-form, multi-speaker conversational audio (such as podcasts) from text. It overcomes common TTS challenges like maintaining speaker consistency over long durations and managing natural turn-taking between multiple speakers.

How it works

The framework uses continuous speech tokenizers (Acoustic and Semantic) operating at a very low frame rate (7.5 Hz) to maintain audio quality while increasing efficiency. It combines a Large Language Model (LLM) to handle textual context and dialogue flow with a diffusion head to generate high-fidelity acoustic details via a next-token diffusion framework.

Who it’s for

It is intended for creators of long-form audio content, such as podcast producers, and developers building real-time TTS applications.

Highlights

  • Long-form synthesis: Capable of generating audio up to 90 minutes long.
  • Multi-speaker support: Supports up to 4 distinct speakers in a single session.
  • Real-time capability: Includes a specialized 0.5B streaming model for low-latency TTS.
  • Voice cloning: Supports cloning voices from audio samples (in the 1.5B and 7B models).
  • Fine-tuning: Community-supported implementations for adapting the model to new languages or voices.

Related

  • Project
    microsoft/VibeVoice

    VibeVoice is Microsoft’s open‑source suite of frontier voice AI models, offering a 7‑B long‑form ASR that handles up to 60 minutes of audio in one pass (speaker diarisation, timestamps, hot‑word support) and a 1.5‑B multi‑speaker TTS that can synthesize up to 90 minutes of speech. A lightweight 0.5 B real‑time streaming TTS runs with ~300 ms latency. All models share a low‑frame‑rate continuous speech tokenizer and a next‑token diffusion architecture, enabling efficient long‑sequence processing. The repo provides Hugging Face model links, Colab demos, CPU‑quantised ASR via BitNet, vLLM GPU inference, and fine‑tuning scripts. Intended for research; Microsoft warns about bias, deep‑fake misuse, and advises against production use without further testing.

  • Project
    OpenMOSS/MOSS-TTS

    A family of open-source speech and sound generation models providing high-fidelity TTS, expressive dialogue, text-to-voice design, real-time voice agents, and sound effect generation.

  • Project
    pnnbao97/VieNeu-TTS

    A high-fidelity Vietnamese text-to-speech system featuring bilingual support, instant voice cloning, and optimized on-device inference for CPU and GPU.

  • Project
    OpenMOSS/MOSS-TTSD

    MOSS-TTSD is a long-form spoken dialogue generation model that converts dialogue scripts into expressive, multi-party conversational speech for podcasts and audiobooks.

  • Project
    2noise/ChatTTS

    ChatTTS is a generative text-to-speech model optimized for conversational dialogue, providing natural, expressive speech with control over laughter and pauses.