fishaudio/fish-speech

What it solves

Fish Speech is a multilingual text-to-speech (TTS) system designed to generate highly natural, emotionally rich, and realistic speech. It addresses the challenge of creating voice generation that captures fine-grained prosody and emotion while maintaining high audio fidelity and fast inference speeds across dozens of languages.

How it works

The system uses a Dual-Autoregressive (Dual-AR) architecture. A large "Slow AR" (4B parameters) predicts primary semantic codebooks along the time axis, while a smaller "Fast AR" (400M parameters) generates residual codebooks to reconstruct acoustic details. The model is trained on over 10 million hours of audio data across 80+ languages and aligned using Group Relative Policy Optimization (GRPO) reinforcement learning to ensure human-like acoustic preferences and timbre similarity.

Who it’s for

It is intended for developers and creators who need state-of-the-art voice cloning, multi-speaker conversation generation, and precise emotional control over synthesized speech.

Highlights

• Fine-Grained Inline Control: Supports over 15,000 unique natural language tags (e.g., [whisper], [excited]) to embed emotions and prosody directly into text.
• Rapid Voice Cloning: Clones voices using short reference samples (10-30 seconds) without requiring additional fine-tuning.
• Multilingual Support: Natively supports over 80 languages without needing phonemes or language-specific preprocessing.
• Native Multi-Speaker Generation: Can handle multiple speakers in a single generation using speaker ID tokens from a single reference audio file.
• High Performance: Optimized for streaming via SGLang, achieving low time-to-first-audio (~100ms) and high throughput on NVIDIA H200 GPUs.
• Multi-Turn Generation: Leverages context from previous turns to improve the naturalness of dialogue.