microsoft/VibeVoice

Open-Source Frontier Voice AI

What it solves

VibeVoice addresses the challenge of processing and generating long-form audio while maintaining speaker consistency and high fidelity. It solves the problem of conventional ASR models slicing audio into short chunks (which loses global context) and provides a way to synthesize long, multi-speaker conversations that sound natural and expressive.

How it works

The framework uses continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz to maintain efficiency over long sequences. It employs a next-token diffusion framework that combines a Large Language Model (LLM) to handle textual context and dialogue flow with a diffusion head to generate high-fidelity acoustic details.

Who it’s for

It is designed for researchers and developers in the speech synthesis and recognition community, as well as those building applications for long-form conversational audio, podcasts, and multi-speaker dialogues.

Highlights

  • Long-form ASR: Processes up to 60 minutes of audio in a single pass, providing structured transcriptions with speaker identity, timestamps, and content.
  • Long-form TTS: Synthesizes speech up to 90 minutes long with support for up to 4 distinct speakers in one conversation.
  • Real-time Streaming: Includes a lightweight 0.5B model for real-time TTS with low latency (~300ms) and streaming text input.
  • Multilingual Support: ASR supports over 50 languages, and the Realtime TTS model includes experimental voices in nine languages.
  • Customized Hotwords: Allows users to provide specific names or technical terms to improve ASR accuracy for domain-specific content.
  • Efficient Inference: Offers a BitNet version for edge CPU inference, compressing the model from 4.62 GB to 1.58 GB.

Related

  • Dispatch
    Hugging Face FFASR Leaderboard: Benchmarking Far-Field ASR

    Hugging Face and Treble Technologies have launched the FFASR Leaderboard, the first open community-driven benchmark to quantify the performance gap between near-field and far-field Automatic Speech Recognition (ASR) in realistic acoustic environments.

  • Dispatch
    Fine-Tune MMS Adapter Models for Low-Resource ASR

    Hugging Face announced that fine‑tuning adapter layers on Meta AI's Massive Multilingual Speech (MMS) models yields state‑of‑the‑art word error rates for low‑resource languages after only minutes of training, while being more memory‑efficient than full model fine‑tuning.

  • Dispatch
    Fine-Tuning XLS-R for Low-Resource Automatic Speech Recognition

    Hugging Face provides a technical guide on fine-tuning XLS-R, a cross-lingual speech representation model, for low-resource Automatic Speech Recognition (ASR) using the Transformers library.

  • Project
    vibevoice-community/VibeVoice

    A framework for generating expressive, long-form, multi-speaker conversational audio from text, capable of producing podcasts up to 90 minutes long.

  • Dispatch
    Fine-Tuning Wav2Vec2-BERT for Low-Resource ASR

    Hugging Face demonstrates how to fine-tune Meta's Wav2Vec2-BERT model for Automatic Speech Recognition (ASR) in low-resource languages, achieving performance comparable to Whisper-large-v3 while being significantly faster and more resource-efficient.