microsoft/VibeVoice

VibeVoice – Open‑Source Frontier Voice AI

What it is – VibeVoice is a Microsoft‑maintained research suite that ships a family of state‑of‑the‑art speech models:

VibeVoice‑ASR – a long‑form automatic‑speech‑recognition model that can ingest up to 60 minutes of audio in a single pass and output a structured transcript (speaker diarisation, timestamps, and content) with optional hot‑word customization.
VibeVoice‑TTS – a long‑form, multi‑speaker text‑to‑speech model capable of generating up to 90 minutes of natural‑sounding speech, supporting up to four distinct voices and several languages.
VibeVoice‑Realtime (0.5 B) – a lightweight, streaming TTS model designed for real‑time use (≈300 ms latency) and able to continue generation for several minutes.

All models share a common architectural innovation: continuous speech tokenizers (acoustic + semantic) that operate at a very low 7.5 Hz frame rate, coupled with a next‑token diffusion approach where an LLM provides high‑level textual context and a diffusion head fills in fine‑grained acoustic detail. This design yields high fidelity while keeping compute and memory requirements manageable for long sequences.

Key Features

Model	Size	Main capability	Long‑form limit	Multilingual	Demo / Quick‑try
VibeVoice‑ASR‑7B	7 B parameters	Speech‑to‑text with speaker diarisation & timestamps	60 min (single pass)	50+ languages	Playground
VibeVoice‑ASR‑BitNet	7 B (quantised)	CPU‑only inference, heterogeneous quantisation (I8_S + I2_S)	60 min	50+ languages	VibeASR.cpp
VibeVoice‑TTS‑1.5B	1.5 B	Long‑form multi‑speaker synthesis	90 min	English, Chinese, others	(currently disabled)
VibeVoice‑Realtime‑0.5B	0.5 B	Real‑time streaming TTS	~10 min continuous	9 languages (EN, DE, FR, IT, JP, KR, NL, PL, PT, ES)	Colab

Unified tokeniser at 7.5 Hz reduces sequence length dramatically, enabling the 60‑minute ASR pass and 90‑minute TTS generation.
Next‑token diffusion merges LLM‑level language understanding with diffusion‑based acoustic generation for high‑quality audio.
Edge‑CPU inference via the BitNet‑quantised ASR model (real‑time on 3+ CPU threads, no GPU needed).
vLLM integration for faster ASR inference on GPUs.
Fine‑tuning scripts are provided for both ASR and TTS, allowing domain‑specific adaptation.

Typical Use‑Cases

Transcribing long meetings, lectures, podcasts with speaker attribution and timestamps in a single run.
Generating synthetic podcasts, audiobooks, or multi‑speaker dialogues without stitching together many short clips.
Real‑time voice assistants or streaming narration where low latency is required.
Research on long‑context speech processing, tokenisation strategies, or diffusion‑based audio generation.

Getting Started (quick‑start steps)

Pick a model – decide whether you need ASR, TTS, or real‑time streaming.
Clone the repo and install dependencies (see requirements.txt).
Run the provided Colab notebooks for a zero‑setup demo:
- TTS streaming: demo/vibevoice_realtime_colab.ipynb
- ASR playground: use the Gradio link in the README.
Load the model via Hugging Face (e.g., from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor for ASR, or the corresponding TTS loader).
Optional – fine‑tune using the scripts under finetuning-asr/ or finetuning-tts/ if you have domain‑specific data.
For edge deployment, follow the BitNet instructions in the VibeASR.cpp repo to run the quantised ASR model on CPU.

Risks & Limitations (as documented)

Bias & Errors – inherits biases from the underlying Qwen2.5 1.5B base model; outputs may be inaccurate or reflect unwanted stereotypes.
Deep‑fake potential – high‑quality synthetic speech can be misused for impersonation or misinformation; responsible disclosure is recommended.
Research‑only license – Microsoft advises against commercial or production deployment without extensive testing and compliance checks.
Model size & hardware – while the 0.5 B realtime model runs on modest hardware, the 7 B ASR model still requires a GPU (or the BitNet‑quantised version for CPU).

Where to Learn More

Project page – https://microsoft.github.io/VibeVoice
Technical reports – ASR paper (arXiv 2601.18184), TTS paper (ICLR 2026 oral), diffusion method (arXiv 2412.08635)
Hugging Face collections – https://huggingface.co/collections/microsoft/vibevoice-68a2ef24a875c44be47b034f
Playgrounds & Demos – ASR Playground (aka.ms/vibevoice-asr), Streaming TTS Colab notebooks.

Community & Contribution

The repository includes a CONTRIBUTING.md with guidelines for code, model, and documentation contributions. Issues, pull requests, and model cards are welcomed.