MatthewCYM/VoiceBench

What it solves

VoiceBench provides a standardized framework and dataset for benchmarking the performance of LLM-based voice assistants. It addresses the lack of consistent evaluation metrics for voice assistants across different tasks such as open-ended QA, reasoning, and instruction following.

How it works

The project consists of a comprehensive dataset hosted on Hugging Face and an evaluation pipeline. The pipeline operates in three steps:

1. Response Generation: The system prompts a voice assistant model (e.g., Qwen2, Diva) using either audio or text modality to generate responses.
2. Automatic Evaluation: For specific subsets like alpacaeval and wildvoice, gpt-4o-mini is used as an automated judge to score the responses.
3. Final Result Calculation: A dedicated evaluation script calculates final scores based on the evaluator type (e.g., mcq for multiple-choice, bbh for reasoning, or harm for safety).

Who it’s for

This tool is designed for researchers and developers building Large Audio Language Models (LALMs) or voice assistants who need to measure their model's capabilities and compare them against a leaderboard of other state-of-the-art models.

Highlights

• Diverse Datasets: Includes multiple subsets covering open-ended QA, multiple-choice QA, reasoning, safety, and instruction following.
• Human-Recorded Speech: Features crowd-sourced datasets like wildvoice and bbh containing human-recorded speech with diverse accents.
• Automated Judging: Integrates GPT-4 based evaluation for open-ended responses.
• Broad Coverage: Tests across 12 diverse domains including those from mmlu-pro.