OpenBMB/UltraEval-Audio

Your faithful, impartial partner for audio evaluation — know yourself, know your rivals. 真实评测，知己知彼。A unified benchmark framework for ASR/TTS/Audio Codec/audio LLM evaluation

What it solves

UltraEval-Audio provides a unified framework for the comprehensive evaluation of large audio foundation models. It addresses the difficulty of evaluating both speech understanding (Speech-to-Text) and speech generation (Speech-to-Speech) within a single system, while automating the tedious process of dataset acquisition and metric alignment.

How it works

The framework aggregates 34 authoritative benchmarks across four domains (speech, sound, medicine, and music) and supports 10 languages and 12 task categories. It automates the downloading and processing of benchmark datasets and binds them with official evaluation metrics such as WER, BLEU, and G-Eval. To prevent dependency conflicts between different models, it uses an isolated inference mechanism where model-specific environments are managed automatically and communicate with the main process via Inter-Process Communication (IPC).

Who it’s for

It is designed for researchers and developers working with audio foundation models, specifically those building or testing Text-to-Speech (TTS), Automatic Speech Recognition (ASR), and Audio Codecs.

Highlights

  • Unified Evaluation: Supports both speech understanding and generation in one framework.
  • Automated Management: One-click benchmark downloading and data processing.
  • Isolated Runtime: Eliminates dependency conflicts by running model inference in isolated environments.
  • Broad Coverage: Includes 34 benchmarks across speech, sound, medicine, and music.
  • Reproducibility: Provides detailed replication documentation and one-click commands for popular open-source models.
  • Flexible Execution: Features GPU parallel acceleration, error retries, and resume-from-breakpoint capabilities.

Related

  • Project
    AudioLLMs/AudioBench

    A universal benchmark for evaluating Audio Large Language Models (AudioLLMs) across 50+ datasets covering speech, audio-scene, and voice understanding.

  • Project
    EleutherAI/lm-evaluation-harness

    A unified framework to evaluate generative language models on a large number of standard academic benchmarks, serving as the backend for the Open LLM Leaderboard.

  • Project
    groq/openbench

    A provider-agnostic evaluation infrastructure that provides standardized, reproducible benchmarking for LLMs across 95+ evaluation suites.

  • Project
    wavlab-speech/versa

    VERSA is an open‑source Python toolkit that bundles more than 90 speech‑and‑audio quality metrics (perceptual, intelligibility, statistical, etc.) into a single, easy‑to‑use package. It supports flexible input formats, optional GPU/Slurm distributed execution, resume‑safe JSONL output, and interactive visualisation. Install the core with `pip install .` and add optional groups for heavy back‑ends (e.g., Qwen2‑Audio, SongEval). Use the `versa-score` CLI or `versa/bin/scorer.py` with a YAML config to evaluate a dataset, optionally resuming interrupted runs or scaling across a cluster.

  • Project
    EvolvingLMMs-Lab/lmms-eval

    A unified evaluation toolkit for Large Multimodal Models that provides reproducible and efficient benchmarking across image, video, and audio tasks.