AudioLLMs/AudioBench

AudioBench: A Universal Benchmark for Audio Large Language Models

What it solves

AudioBench provides a standardized, universal benchmark for evaluating Audio Large Language Models (AudioLLMs). It addresses the lack of a comprehensive evaluation framework that can test models across diverse audio tasks, including speech recognition, audio-scene understanding, and voice characteristics.

How it works

The framework allows users to run inference on a wide variety of audio datasets and evaluate the results using different metrics. It supports a "model-as-judge" approach, where a powerful LLM (like Llama-3-70B) is served via vLLM to grade the responses of the model being tested. It also supports traditional metrics like Word Error Rate (WER) and BLEU for ASR and translation tasks.

Who it’s for

It is designed for AI researchers and developers building or fine-tuning AudioLLMs who need to measure their model's performance across speech, music, and environmental audio understanding.

Highlights

  • Extensive Dataset Support: Covers over 50 datasets spanning ASR, speech translation, question answering, emotion recognition, and music understanding.
  • Diverse Task Coverage: Evaluates capabilities in speech, audio-scene, and voice understanding.
  • Model-as-Judge Integration: Built-in support for using LLMs to evaluate open-ended responses.
  • Extensible Architecture: Simple process for adding new custom datasets and models to the benchmark suite.

Related

  • Project
    MatthewCYM/VoiceBench

    VoiceBench is a benchmark suite for evaluating LLM‑based voice assistants. It supplies a Hugging‑Face dataset with audio‑driven QA and instruction‑following tasks, a command‑line toolkit to run a model, collect spoken responses, and automatically score them (via GPT‑4o‑mini for open‑ended sets). Results are posted on a public leaderboard, and the repo includes a curated list of recent voice‑assistant papers.

  • Project
    OpenBMB/UltraEval-Audio

    A unified open-source framework for evaluating large audio foundation models, supporting both speech understanding and generation across 34 benchmarks.

  • Project
    bytedance/SALMONN

    A suite of advanced multi-modal LLMs that provide large language models with generic hearing abilities and integrated audio-visual perception.

  • Project
    wavlab-speech/versa

    VERSA is an open‑source Python toolkit that bundles more than 90 speech‑and‑audio quality metrics (perceptual, intelligibility, statistical, etc.) into a single, easy‑to‑use package. It supports flexible input formats, optional GPU/Slurm distributed execution, resume‑safe JSONL output, and interactive visualisation. Install the core with `pip install .` and add optional groups for heavy back‑ends (e.g., Qwen2‑Audio, SongEval). Use the `versa-score` CLI or `versa/bin/scorer.py` with a YAML config to evaluate a dataset, optionally resuming interrupted runs or scaling across a cluster.

  • Project
    groq/openbench

    A provider-agnostic evaluation infrastructure that provides standardized, reproducible benchmarking for LLMs across 95+ evaluation suites.