5uck1ess/tts-bench

Speed and samples benchmark: for all types of text to speech (TTS) models on Windows/Linux/Mac.

tts‑bench – a local benchmark suite for text‑to‑speech models

What it is – tts‑bench is a Python‑based tool that lets you run a large collection of open‑source TTS models on your own hardware and compare them along three dimensions:

Lens	What it measures	How it is shown
Speed	Cold‑ and warm time‑to‑first‑audio (TTFA), realtime factor (RTFx) and memory usage. Runs on CPU, CUDA GPUs and Apple‑Silicon (MPS).	Sortable tables per‑rig (e.g. Ryzen 9 + RTX 5090, Apple M4, RTX 3090).
Listen	Raw audio output for every model on every benchmark prompt, both the model’s built‑in voice and a zero‑shot voice‑cloned version.	Inline `<audio>` players on the demo site; you can browse by prompt or by model.
Scores	Objective quality numbers: UTMOS (naturalness), WER (intelligibility) and SIM (voice‑cloning fidelity).	Interactive sortable tables and a top‑15 chart; a Default/Cloning toggle lets you see the two groups separately.

The suite does not currently expose a single “overall quality” metric – the authors tried a NAQ score but removed it because it didn’t align with human preferences.

Quick start (from the README)

Install the tiny package manager uv and Python 3.11.
Run the provided install script (Windows PowerShell or Unix shell). You can install all models or pick a subset, e.g.
```
./install.sh kokoro piper miso   # Linux/macOS
.\install.ps1 kokoro,piper,miso # Windows
```
The full set occupies ~~165 GB (≈39 GB of virtual‑environment files in the repo + ~125 GB of model weights cached in `~~/.cache/huggingface`).
Execute the benchmark:
```
python bench.py
```
Only models whose virtual‑env folder exists are run, so a partial install works fine.
Optional helpers:
- python speak.py <model> – play a single utterance.
- python compare.py "your phrase" – generate a side‑by‑side A/B clip for two models.
- Add --reference <wav> to bench.py to evaluate cloning models with your own voice sample.

What’s being benchmarked?

The repository tracks 75 models (both pre‑defined‑voice and zero‑shot cloning models). A few highlights:

Fastest on a high‑end GPU: Kokoro (67 ms warm TTFA, 104× realtime).
Fastest on CPU: Piper (≈107 ms warm TTFA, 59× realtime on a Ryzen 9).
Top cloning candidates (based on a public blind A/B voting arena):
1. OmniVoice – best voice‑match but can drop words.
2. Echo‑TTS – close second, clean 44.1 kHz output.
3. IndexTTS‑2 – third place.
The Listen lens is the recommended way to decide “best sounding” because the objective scores (UTMOS, WER, SIM) are only back‑stops; human ears are the ground truth.

How the scores are computed

UTMOS – a neural predictor of naturalness.
WER – word‑error‑rate from an ASR model (the same approach used in the seed‑tts‑eval repo).
SIM – speaker‑verification similarity for cloning models.

All metrics are run on the same set of benchmark prompts; the code lives in the scoring virtual environment (add scoring or scoring_sim on Linux to the install command).

Community & extensibility

Human‑preference arena – a Hugging Face Spaces app (tts‑arena) where anyone can vote on pairs of clips. Votes feed a live Elo leaderboard that the README references for “best cloning”.
Adding a model – see docs/architecture.md for the runner protocol; you simply drop a new entry in the model table and provide a small wrapper that follows the same CLI.
Known issues & licensing – detailed per‑model notes are in docs/known‑issues.md and docs/expressive‑control.md.

Who might use this?

Researchers wanting a reproducible comparison of many open‑source TTS systems on their own machines.
Developers selecting a model for an application (e.g., low‑latency on‑device synthesis vs. high‑quality voice cloning).
Enthusiasts who enjoy listening to dozens of voices and voting on them.

Bottom line

tts‑bench is a genuine, fairly comprehensive benchmarking suite for local text‑to‑speech models. It provides speed numbers, objective quality metrics, and an easy way to listen to every model, plus a community‑driven voting system for subjective ranking. The project is well‑documented, supports multiple hardware back‑ends, and can be extended with new models.