MatthewCYM/VoiceBench

VoiceBench – Benchmark for LLM‑based Voice Assistants

What it is – VoiceBench is a research‑grade benchmark that lets you evaluate how well large language models (LLMs) work when they are used as voice assistants. It provides a collection of audio‑driven question‑answering and instruction‑following tasks, a Hugging‑Face dataset, and a small command‑line toolkit that runs a model, gathers its spoken responses, and scores them automatically (using GPT‑4o‑mini for some subsets).

Why it matters – As LLMs gain audio capabilities (speech‑to‑text, text‑to‑speech, and end‑to‑end speech‑language models), there is no standard way to compare them. VoiceBench fills that gap by offering:

• Multiple test suites covering open‑ended QA, multiple‑choice QA, reasoning, safety, and instruction following.
• Both synthetic (Google TTS) and human‑recorded audio sources, so you can see how models handle clean TTS versus noisy real speech.
• A public leaderboard where researchers can submit results via GitHub issues.

How to get started

1. Create the environment (Python 3.10, PyTorch 2.1.2 with CUDA 12.1, xformers).

   conda create -n voicebench python=3.10
   conda activate voicebench
   pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 \
       --index-url https://download.pytorch.org/whl/cu121
   pip install xformers==0.0.23 --no-deps
   pip install -r requirements.txt
   

2. Load the data – the benchmark lives on Hugging Face.

   from datasets import load_dataset
   ds = load_dataset("hlt-lab/voicebench", "alpacaeval")
   

   Sub‑sets include alpacaeval, commoneval, wildvoice, openbookqa, bbh, advbench, etc.
3. Run a model – the provided main.py generates a JSONL of model responses.

   python main.py --model naive --data alpacaeval --split test --modality audio
   

   Replace naive with any model you want to test (e.g., qwen2, diva).
4. Automatic scoring – for the open‑ended QA sets the script calls the gpt-4o-mini API.

   python api_judge.py --src_file naive-alpacaeval-test-audio.jsonl
   

5. Aggregate results – finally compute the benchmark numbers.

   python evaluate.py --src_file result-naive-alpacaeval-test-audio.jsonl --evaluator open
   

   The --evaluator flag selects the appropriate metric (open‑ended, MCQ, safety, etc.).

Dataset snapshot

Subset	# samples	Audio source	Task
alpacaeval	199	Google TTS	Open‑ended QA
alpacaeval_full	636	Google TTS	Open‑ended QA
commoneval	200	Human	Open‑ended QA
wildvoice	1 000	Human	Open‑ended QA
openbookqa	455	Google TTS	Multiple‑choice QA
mmsu	3 074	Google TTS	Multiple‑choice QA
sd‑qa	553	Human	Reference‑based QA
mtbench	46	Google TTS	Multi‑turn QA
ifeval	345	Google TTS	Instruction following
bbh	1 000	Human	Reasoning
advbench	520	Google TTS	Safety

Leaderboard & community – Results are displayed on a public page (https://matthewcym.github.io/VoiceBench/). New submissions are encouraged through the repository’s issue tracker, and the leaderboard is updated automatically.

Related resources – The README also curates an “Awesome Voice Assistants” list that points to recent papers and codebases (e.g., MiniCPM‑o 4.5, Qwen 3.5‑Omni, VoxMind). This makes VoiceBench a handy hub for anyone tracking the rapid progress of speech‑enabled LLMs.

Citation – If you use VoiceBench in a paper, cite the accompanying arXiv pre‑print (arXiv:2410.17196).

---

All details above are taken directly from the repository’s README; no additional features have been inferred.