netease-youdao/Confucius4-TTS

Confucius4-TTS: a Multilingual and Cross-Lingual Zero-Shot TTS Engine

What it solves

Confucius4-TTS addresses the challenge of high-quality, multilingual speech synthesis where a speaker's identity is preserved across different languages. It enables "zero-shot" voice cloning, meaning it can replicate a voice from a short audio sample without requiring additional training or a transcript of the reference audio.

How it works

The system uses a "speech encoder + large language model (LLM)" architecture. It operates in two main stages:

  1. Text2Semantic (T2S): An LLM-based stage that generates semantic token sequences from input text and speaker conditioning.
  2. Semantic2Acoustic (S2A): A flow-matching model that converts those semantic tokens into mel spectrograms, which are then turned into waveforms.

To improve performance, the project supports the vLLM backend to accelerate the LLM stage using PagedAttention.

Who it’s for

This tool is designed for developers and researchers working on speech synthesis, localization, and voice cloning across 14 supported languages (including Chinese, English, Japanese, Korean, German, French, Spanish, and others).

Highlights

  • Cross-Lingual Transfer: Synthesizes unaccented speech in a target language while keeping the original speaker's voice.
  • Zero-Shot Cloning: Clones voices instantly from a reference WAV file without needing a transcript.
  • Emotion Transfer: Capable of cloning the emotional tone of the reference audio.
  • Broad Language Support: Supports 14 languages out of the box.
  • Flexible Deployment: Provides a Gradio web UI, a Python API, and a FastAPI server for both streaming and non-streaming audio generation.