Big Bench Audio Release

Artificial Analysis has released Big Bench Audio, a new evaluation dataset designed to assess the reasoning capabilities of audio language models. The dataset reveals a significant "speech reasoning gap," where models perform substantially worse when processing audio compared to text, even when the underlying reasoning task is identical.

The Big Bench Audio Dataset

Big Bench Audio consists of 1,000 audio questions adapted from Big Bench Hard, a benchmark known for rigorous testing of advanced reasoning. The dataset is balanced across four categories, with 250 questions each:

  • Formal Fallacies: Testing logical deduction from provided statements.
  • Navigate: Determining if navigation steps return to a starting point.
  • Object Counting: Counting specific items within collections.
  • Web of Lies: Evaluating Boolean logic expressed in natural language.

To ensure quality, the audio files were generated using 23 synthetic voices from top-ranked Text to Speech (TTS) models. Each generation was verified using Levenshtein distance against transcriptions and manual review of edge cases.

Evaluation Methodology and Configurations

To isolate the impact of audio on reasoning, Artificial Analysis tested models across four distinct configurations:

  1. Speech to Speech: Audio input $\rightarrow$ Audio output.
  2. Speech to Text: Audio input $\rightarrow$ Text output.
  3. Text to Speech: Text input $\rightarrow$ Audio output.
  4. Text to Text: Text input $\rightarrow$ Text output.

Experimental Setup

Eighteen experiments were conducted involving various versions of GPT-4o (including Realtime Preview and mini Realtime Preview), Gemini 1.5 Flash, Gemini 1.5 Pro, and Gemini 2.0 Flash (Experimental).

Evaluation was automated using an LLM Evaluator (Claude 3.5 Sonnet, Oct '24). The system transcribes audio responses via OpenAI's Whisper API and then prompts the evaluator to label the candidate answer as either "CORRECT" or "INCORRECT" based on the official answer and the original question context.

Key Findings: The Audio Reasoning Gap

The primary finding of the Big Bench Audio analysis is a substantial performance drop when transitioning from text to speech modalities.

Performance Degradation

Using GPT-4o as a primary example, the research shows a clear decline in accuracy:

  • Text to Text: 92% accuracy (GPT-4o Aug '24).
  • Text to Speech: 74% accuracy.
  • Speech to Speech: 66% accuracy (GPT-4o Realtime Preview Oct '24).

This data indicates that both speech input and speech output contribute to the degradation of reasoning performance.

Pipeline vs. Native Audio

Traditional pipeline approaches—which utilize a sequence of Whisper (transcription), GPT-4o (reasoning), and TTS-1 (voice generation)—currently outperform native Speech to Speech models for reasoning tasks. These pipelines show minimal performance degradation compared to pure text processing, suggesting that for applications where reasoning accuracy is the highest priority, pipeline architectures are currently more reliable than native audio models.

Sources