fishaudio/fish-speech

SOTA Open Source TTS

Fish Speech – Expressive, multilingual text‑to‑speech

What it is – An open‑source speech synthesis system that combines large language models with neural TTS (VITS‑style) to produce high‑quality, expressive, multi‑speaker audio. It ships two flagship models:

  • Drama 3 (35 B parameters, 3 B active per token) – the newest model, optimized for long‑form, multi‑character scenes, on‑the‑fly editing, and strong emotional nuance.
  • S2‑Pro (4 B parameters) – a smaller, faster model that still supports multilingual voice cloning, inline prosody tags, and streaming generation.

Key capabilities

  • Natural‑language control – embed tags like [whisper], [angry], [pause] directly in the script to steer emotion, prosody, and timing.
  • Voice cloning – 10‑30 s of reference audio is enough to reproduce a speaker’s timbre, with cross‑lingual cloning that achieves state‑of‑the‑art CER/WER.
  • Multi‑speaker / scene generation – generate a whole dialogue or drama with several characters in a single pass.
  • Long‑form streaming – produce continuous audio for minutes of text; Drama 3 even supports “director revision” where you ask the model to re‑render a segment with new instructions.
  • Local editing / infilling – replace or insert words without re‑generating the whole clip, preserving timbre continuity.

How to use

  • Installation – detailed guides for environment setup, weight download, and Docker deployment are provided at https://speech.fish.audio/install/.
  • Inference – command‑line tools, an HTTP API server, and a Web UI let you synthesize speech locally or serve it as a service.
  • LLM agents – the same install steps can be called from autonomous agents that need on‑demand TTS.

Performance

  • S2‑Pro achieves sub‑1 % WER on English/Chinese Seed‑TTS benchmarks and leads on several multilingual and Turing‑test metrics.
  • Drama 3 outperforms S2‑Pro on new benchmarks for multilingual cloning, free‑form editing, and multi‑turn emotion revision (see the technical report PDF for numbers).

Licensing

  • Code and model weights are released under the Fish Audio Community License; S2‑Pro weights carry the Fish Audio Research License. The repository warns against illegal use and will enforce the license.

Resources

  • Model checkpoints on Hugging Face: fishaudio/drama-3 and fishaudio/s2-pro.
  • Technical reports (PDF & arXiv) detail architecture, training recipes, and evaluation.
  • Community: Discord, Docker Hub, and QQ channel for support.

Who might benefit

  • Developers building voice‑enabled applications, games, or virtual assistants that need expressive, controllable speech.
  • Researchers exploring large‑scale multilingual TTS or editing‑in‑the‑loop generation.
  • Content creators who want to generate multi‑character audio dramas without recording each voice.

All information above is taken directly from the repository’s README; no additional features are inferred.

Related

  • Project
    Edge0-AI/Audio8_TTS

    A compact multilingual text-to-speech model with zero-shot voice cloning, available in 0.6B and 0.1B parameter versions for high-efficiency speech synthesis.

  • Project
    KittenML/KittenTTS

    An open-source text-to-speech library featuring a 1.7B-parameter speech language model with in-context voice cloning and CPU-optimized inference.

  • Project
    breezeblue-ai/breeze-tts

    Breeze TTS 2 is an open-weight, high-performance text-to-speech model that supports real-time streaming, voice cloning, and instruction-based voice design in English and Chinese.

  • Project
    fishaudio/Bert-VITS2

    Bert-VITS2 is a text-to-speech system that integrates a VITS2 backbone with multilingual BERT to produce high-quality, natural speech synthesis.

  • Project
    resemble-ai/chatterbox

    A family of open-source text-to-speech models providing low-latency voice cloning and multilingual speech generation across various model sizes for GPU and CPU deployment.