wavlab-speech/versa

What it solves

VERSA provides a unified toolkit for evaluating the quality of speech and audio. Instead of manually implementing or integrating dozens of different scoring methods, researchers and developers can use a single framework to access over 90 evaluation and profiling metrics across multiple dimensions of audio quality.

How it works

VERSA acts as a comprehensive wrapper and orchestrator for a vast array of audio metrics. It organizes these metrics into four categories: independent (no reference needed), dependent (comparing prediction to reference), non-match, and distributional (statistical properties of collections). The toolkit supports various input formats like file paths, SCP files, and Kaldi-style ARKs, and can be deployed on local machines or distributed across clusters using Slurm. It also integrates with LLM-informed perceptual metrics (such as those based on Qwen2-Audio) to provide more nuanced audio profiling.

Who it’s for

It is designed for speech and audio researchers, developers of neural codecs, and anyone building AI-driven audio generation or processing systems who needs a standardized way to measure performance.

Highlights

• Massive Metric Library: Access to 90+ metrics covering perceptual quality, intelligibility, and technical measurements.
• Scalable Infrastructure: Built-in support for distributed evaluation via Slurm and the ability to resume interrupted scoring runs.
• Flexible Input/Output: Compatible with standard speech toolkits like ESPnet and supports multiple audio input formats.
• LLM Integration: Supports perceptual metrics powered by Large Language Models for advanced audio quality assessment.
• Discovery Tools: Includes a CLI (versa-score) to list available metrics and generate recommended configuration files based on the task.