microsoft/VibeVoice

What it solves

VibeVoice addresses the challenge of processing and generating long-form audio while maintaining speaker consistency and semantic coherence. It provides a unified framework for both high-fidelity speech recognition (ASR) and text-to-speech (TTS) that can handle extended audio sequences without the need to slice audio into short, disjointed chunks.

How it works

The system uses continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz to efficiently preserve audio fidelity. It employs a next-token diffusion framework, combining a Large Language Model (LLM) to handle textual context and dialogue flow with a diffusion head to generate high-fidelity acoustic details.

Who it’s for

It is designed for researchers and developers working on voice AI, specifically those needing to process long-form conversational audio, podcasts, or multi-speaker dialogues.

Highlights

• Long-form ASR: Processes up to 60 minutes of continuous audio in a single pass, providing structured transcriptions with speaker identity, timestamps, and content.
• Customized Hotwords: Allows users to provide specific names or technical terms to improve recognition accuracy for domain-specific content.
• ** uma Streaming ASR**: A variant that transcribes audio in real-time as it arrives.
• Long-form TTS: Capable of synthesizing up to 90 minutes of speech with up to 4 distinct speakers in a single conversation.
• Real-time TTS: A lightweight 0.5B parameter model designed for low-latency streaming text-to-speech generation.
• Edge Deployment: Includes a BitNet version of the ASR model for real-time CPU inference without requiring a GPU.