k2-fsa/OmniVoice

OmniVoice – Massive‑scale, zero‑shot text‑to‑speech

What it is – OmniVoice is a multilingual text‑to‑speech (TTS) system that can synthesize speech in more than 600 languages without any per‑language fine‑tuning. It uses a diffusion‑style language‑model architecture that delivers high‑quality audio at very low real‑time factor (RTF ≈ 0.025, i.e. 40 × faster than real time). The model supports three main use‑cases:

Voice cloning – generate speech that sounds like a short reference recording.
Voice design – specify speaker attributes (gender, age, pitch, accent, dialect, whisper, etc.) instead of providing a reference audio.
Auto voice – let the model pick a voice automatically.

It also offers fine‑grained control such as non‑verbal symbols ([laughter]) and explicit pronunciation hints (pinyin for Chinese, CMU phonemes for English).

Quick start

Web UI – omnivoice-demo --ip 0.0.0.0 --port 8001 launches a Gradio demo.
Hugging Face Space – try the model online at https://huggingface.co/spaces/k2-fsa/OmniVoice.
Colab notebook – a ready‑to‑run notebook is linked in the README.

Installation (pip)

# Install PyTorch that matches your hardware (CUDA, Apple Silicon, Intel XPU)
# Example for a recent NVIDIA GPU:
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 \
    --extra-index-url https://download.pytorch.org/whl/cu128

# Install the library (stable release from PyPI)
pip install omnivoice

Alternatively install directly from the repo (pip install git+https://github.com/k2-fsa/OmniVoice.git) or use the uv tool to sync dependencies.

Python API (core pattern)

from omnivoice import OmniVoice
import soundfile as sf, torch

model = OmniVoice.from_pretrained(
    "k2-fsa/OmniVoice",
    device_map="cuda:0",   # "mps" on Apple Silicon, "xpu" on Intel Arc
    dtype=torch.float16)

# 1️⃣ Voice cloning
audio = model.generate(
    text="Hello, this is a test of zero‑shot voice cloning.",
    ref_audio="ref.wav",
    ref_text="Transcription of the reference audio.")
sf.write("out.wav", audio[0], 24000)

# 2️⃣ Voice design
audio = model.generate(
    text="Hello, this is a test of voice design.",
    instruct="female, low pitch, british accent")

# 3️⃣ Auto voice
audio = model.generate(text="Just speak in a random voice.")

The same model.generate call also accepts diffusion steps (num_step), speed factor, fixed duration, and many other knobs described in docs/generation-parameters.md.

Command‑line tools

Command	Typical use
`omnivoice-demo`	Launch the interactive Gradio demo
`omnivoice-infer`	Single‑utterance inference (voice cloning, design, or auto)
`omnivoice-infer-batch`	High‑throughput batch inference, optionally distributed across GPUs

All CLI options mirror the Python API arguments (e.g., --ref_audio, --instruct, --num_step).

Speed tricks

FlashInfer kernels give a 2–2.9× speedup with no quality loss. Install with the provided wheel for your CUDA version and enable it via --enable_flashinfer true (CLI) or apply_flashinfer(model) (Python). CUDA graphs further improve latency for batch‑size 1.

Training & evaluation

The repository includes an examples/ folder that walks through data preparation, model training, evaluation, and fine‑tuning. Users can adapt the pipeline to their own data.

Community & support

Issues are handled on GitHub.
A WeChat group and official account are advertised for Chinese‑language community discussion.
A list of community projects is maintained in docs/community-projects.md.

Citation

If you use OmniVoice in research, cite the arXiv paper:

@article{zhu2026omnivoice,
  title={OmniVoice: Towards Omnilingual Zero‑Shot Text‑to‑Speech with Diffusion Language Models},
  author={Zhu, Han and Ye, Lingxuan and Kang, Wei and Yao, Zengwei and Guo, Liyong and Kuang, Fangjun and Han, Zhifeng and Zhuang, Weiji and Lin, Long and Povey, Daniel},
  journal={arXiv preprint arXiv:2604.00688},
  year={2026}
}

Ethical note

The authors explicitly forbid illicit voice cloning, impersonation, fraud, or any other illegal/ unethical use. Users are expected to comply with local regulations and responsible‑AI guidelines.