breezeblue-ai/breeze-tts

What it solves

Breeze TTS 2 is designed for real-time, expressive text-to-speech (TTS) interaction. It addresses the need for high-quality, low-latency audio generation that can be precisely controlled via natural language instructions, allowing users to create or modify voices without needing extensive technical setup or multiple reference samples.

How it works

The model is a bilingual (English and Chinese) open-weight system that supports three primary modes of operation:

• Voice Clone: Uses a reference audio clip and its transcript to replicate a speaker's timbre and style.
• Voice Design: Generates a completely new voice based solely on a natural-language description (e.g., "a warm, thoughtful young woman").
• Voice Direction: Combines reference audio with specific instructions to steer the tone, emotion, and pace of the cloned voice.

It also supports "Vocal Events" (like laughing or sighing) inserted directly into the text using parentheses or brackets. To achieve ultra-low latency, it utilizes a "fast path" with CUDA Graphs and specialized decoding stages to minimize the time to first audio (TTFA).

Who it’s for

This project is aimed at developers and researchers building real-time AI voice agents, interactive applications, and content creators who need precise control over vocal delivery and emotion in both English and Chinese.

Highlights

• Instruction-Following: Supports reference-free voice design and reference-guided voice direction using natural language.
• Ultra-Low Latency: Achieves under 40ms time to first audio (TTFA) on NVIDIA H100 GPUs.
• Bilingual Support: A single model handles both English and Chinese speech naturally.
• Expressive Control: Allows the insertion of inline vocal events such as (laugh) or (sigh) for more human-like delivery.
• GPU Efficiency: Operates with approximately 7.7 GiB of GPU memory for eager inference.