wavlab-speech/versa

🎧 What is VERSA?

VERSA (Versatile Evaluation of Speech and Audio) is an open‑source Python toolkit that lets researchers and developers score the quality of speech, music, or any audio signal with 90+ built‑in metrics. The metrics cover perceptual quality (e.g., PESQ, POLQA), intelligibility (e.g., WER, STOI), technical measurements, and even distribution‑level analyses such as Fréchet Audio Distance (FAD). VERSA acts as a thin wrapper around the original implementations, handling model downloads, input parsing, and distributed execution so you can focus on the evaluation itself.

🚀 Core Features (as described in the README)

Huge metric catalog – >90 metrics with many variants, organized into independent, dependent, non‑match, and distributional groups.
Toolkit integration – already used in ESPnet, Kaldi‑style pipelines, and several speech‑related challenges.
Flexible I/O – accepts plain file paths, Kaldi‑style SCP lists, or ARK archives.
Scalable execution – built‑in Slurm support for multi‑node, multi‑GPU runs; local multi‑process mode for CPUs.
Resume‑safe scoring – JSONL results contain a hidden completion record so interrupted jobs can be continued without re‑computing finished utterances.
Visualization – interactive visualizers and a CLI (versa-score) to explore metric results.
Extensible – optional extra‑dependency groups for heavy back‑ends (e.g., Qwen2‑Audio, SongEval, MAPSS) that can be installed on demand.

📦 Installation (quick start)

# Clone and install the core package
git clone https://github.com/wavlab-speech/versa.git
cd versa
pip install .

# Install optional groups only when you need them
pip install ".[audio,text,ml]"      # metrics that need large model stacks
pip install ".[songeval]"          # SongEval metric only
pip install ".[external]"         # back‑ends that wrap other toolkits
pip install ".[dev]"              # testing & linting tools

You can also install directly from the repo without cloning:

python -m pip install git+https://github.com/wavlab-speech/versa.git#egg=versa-speech-audio-toolkit --no-build-isolation

🛠️ Typical Usage

One‑liner scoring (CPU, simple metric set):

python versa/bin/scorer.py \
  --score_config egs/speech_cpu.yaml \
  --gt   test/test_samples/test1 \
  --pred test/test_samples/test2 \
  --output_file result.jsonl \
  --io dir

GPU‑accelerated run with explicit metric selection:

python versa/bin/scorer.py \
  --score_config egs/speech_gpu.yaml \
  --gt test/test_samples/test1.scp \
  --pred test/test_samples/test2.scp \
  --output_file result.jsonl \
  --io soundfile \
  --use_gpu True \
  --scoring_mode metric   # load one metric at a time to save GPU memory

Resume a long job (e.g., after a cluster pre‑empt):

python versa/bin/scorer.py ... --resume

Explore available metrics:

versa-score --list-metrics                # list everything installed
versa-score --describe-metric pesq        # see dependencies, paper refs, etc.
versa-score --recommend-config --task tts --device gpu

Distributed evaluation with Slurm (launch script provided):

./launch_slurm.sh <pred_scp> <gt_scp|None> <score_dir> <split_job_num>

The script handles resource checks, job array creation, and result aggregation.

📊 What kinds of metrics are covered?

Category	Example metrics
Independent	PESQ, POLQA, MOS‑Net, DNS‑MOS (no reference needed)
Dependent	STOI, WER, SI‑SDR (compare prediction vs. ground truth)
Non‑match	Audio‑LLM perceptual scores, Qwen2‑Audio summarizations
Distributional	Fréchet Audio Distance (FAD), Kernel Inception Distance (KID), MAPSS

The full list (including variant names) is on the online Metric Catalog page.

📚 Documentation & Resources

Website & docs: https://wavlab-speech.github.io/versa/
Interactive Colab demo (Interspeech 2024 tutorial): https://colab.research.google.com/drive/11c0vZxbSa8invMSfqM999tI3MnyAVsOp?usp=sharing
Paper: VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music (NAACL 2025 System Demo) – cite the provided BibTeX.
License: MIT (permissive, commercial‑friendly).

🙋‍♀️ Who should use VERSA?

Researchers needing reproducible, multi‑metric evaluation for TTS, VC, speech enhancement, or music generation papers.
Engineers building pipelines that must report standardized audio quality numbers (e.g., for product QA or benchmark suites).
Students learning about speech‑quality metrics; the CLI can generate starter YAML configs for quick experiments.

🤝 Contributing

The project welcomes pull requests. See the online Contributing guide for style, testing, and documentation expectations.

TL;DR

VERSA is a well‑maintained, MIT‑licensed Python library that aggregates >90 speech/audio quality metrics, supports distributed scoring, and provides a unified CLI and YAML‑based configuration system. Install the core, add optional back‑ends as needed, and run versa-score or versa/bin/scorer.py to obtain reproducible metric reports for any audio dataset.