5uck1ess/tts-bench

What it solves

TTS-Bench provides a standardized way to evaluate and compare local text-to-speech (TTS) models. It solves the problem of choosing the right model by providing objective speed and quality metrics, as well as a subjective "Listen" gallery where users can audition models across various prompts and hardware configurations.

How it works

The project implements a benchmarking suite that runs a set of 71 tracked TTS models across different hardware rigs (Windows, Linux, and Mac). It measures three primary "lenses":

• Speed: Tracks cold and warm Time to First Audio (TTFA), Realtime Speed (RTFx), and memory usage across CPU, CUDA, and Apple Silicon.
• Listen: A consolidated gallery of audio samples generated by every model on every prompt, allowing for side-by-side comparison of default voices and voice cloning capabilities.
• Scores: Objective metrics including UTMOS (naturalness), WER (intelligibility), and SIM (cloning fidelity) using ASR and speaker-verification tools.

Additionally, it includes a companion TTS Voting Arena for blind A/B testing to establish a human-preference Elo leaderboard.

Who it’s for

Developers and AI researchers who need to find the best performing local TTS models for their specific hardware (e.g., RTX 5090, Apple M4) and use cases (e.g., high-fidelity cloning or low-latency speed).

Highlights

• Extensive Model Coverage: Tracks 71 different TTS models, including both predefined voice models and zero-shot cloning models.
• Hardware-Specific Leaderboards: Provides performance data for multiple hardware configurations, including high-end NVIDIA GPUs and Apple Silicon.
• Hugging Face Integration: Simplifies installation of a subset of models via a script that manages virtual environments and model weights.
• Human-Preference Ranking: Integrates a blind A/B voting system to determine the best-sounding models based on human ears rather than just objective scores.