AudioLLMs/AudioBench

What it solves

AudioBench provides a standardized, universal benchmark for evaluating Audio Large Language Models (AudioLLMs). It addresses the lack of a comprehensive evaluation framework that can test models across diverse audio tasks, including speech recognition, audio-scene understanding, and voice characteristics.

How it works

The framework allows users to run inference on a wide variety of audio datasets and evaluate the results using different metrics. It supports a "model-as-judge" approach, where a powerful LLM (like Llama-3-70B) is served via vLLM to grade the responses of the model being tested. It also supports traditional metrics like Word Error Rate (WER) and BLEU for ASR and translation tasks.

Who it’s for

It is designed for AI researchers and developers building or fine-tuning AudioLLMs who need to measure their model's performance across speech, music, and environmental audio understanding.

Highlights

• Extensive Dataset Support: Covers over 50 datasets spanning ASR, speech translation, question answering, emotion recognition, and music understanding.
• Diverse Task Coverage: Evaluates capabilities in speech, audio-scene, and voice understanding.
• Model-as-Judge Integration: Built-in support for using LLMs to evaluate open-ended responses.
• Extensible Architecture: Simple process for adding new custom datasets and models to the benchmark suite.