Hugging Face Open ASR Leaderboard: Private Datasets to Combat Benchmaxxing
Hugging Face Open ASR Leaderboard: Private Datasets to Combat Benchmaxxing
Hugging Face has integrated private evaluation datasets provided by Appen Inc. and DataoceanAI into the Open ASR Leaderboard. This update aims to prevent "benchmaxxing"—the practice of optimizing models specifically for public benchmark test sets—and provide a more trustworthy measure of how Automatic Speech Recognition (ASR) models perform across diverse, real-world audio conditions.
Preventing Benchmaxxing with Private Data
To ensure that leaderboard rankings reflect genuine model robustness rather than optimization for specific public datasets, Hugging Face is keeping new high-quality datasets private. This approach mitigates the risk of test-set contamination, where developers might explicitly use public test sets for training or seek data that closely resembles them to artificially boost macroaverage scores.
While the Open ASR Leaderboard maintains a commitment to openness through open-source UI code and evaluation scripts, the introduction of private sets creates a necessary balance. By using data that model developers cannot access, the leaderboard can more accurately identify gaps and biases in model performance, particularly in non-saturated settings like conversational speech and non-American accents.
New Evaluation Datasets and Coverage
Appen Inc. and DataoceanAI have contributed datasets covering various accents, styles, and transcription types. These datasets are categorized to allow for targeted performance analysis:
| Dataset | Accent | Duration [h] | Male (%) / Female (%) | Style | Transcription |
|---|---|---|---|---|---|
| Appen Scripted AU | Australian | 1.42 | 49 / 51 | Read | Punctuated, cased |
| Appen Scripted CA | Canadian | 1.53 | 52 / 48 | Read | Punctuated, cased |
| Appen Scripted IN | Indian | 1.02 | 49 / 51 | Read | Punctuated, cased |
| Appen Scripted US | American | 1.45 | 49 / 51 | Read | Punctuated, cased |
| Appen Conversational IN | Indian | 1.37 | 51 / 49 | Conversational, spontaneous | Punctuated, disfluencies |
| Appen Conversational US003 | American | 1.64 | 49 / 51 | Conversational, spontaneous | Punctuated, cased, disfluencies |
| Appen Conversational US004 | American | 1.65 | 49 / 51 | Conversational, spontaneous | Punctuated, disfluencies |
| DataoceanAI Scripted US | American | 2.43 | 54 / 46 | Read | Punctuated, cased (proper nouns), disfluencies |
| DataoceanAI Scripted GB | British | 2.43 | 47 / 53 | Read | Punctuated, disfluencies |
| DataoceanAI Conversational US | American | 8.82 | NA | Conversational, spontaneous | Punctuated, disfluencies |
| DataoceanAI Conversational GB | British | 5.96 | NA | Conversational, spontaneous | Punctuated, disfluencies |
Targeted Metrics and Leaderboard Functionality
The leaderboard now includes a "Private data" tab with specific macroaverage metrics to highlight performance nuances:
- Average WER: A macroaverage of the data provider averages, ensuring equal weighting between providers.
- Avg Scripted: A macroaverage of all scripted datasets.
- Avg Conversational: ASR performance across all conversational datasets.
- Avg US: A macroaverage of all datasets featuring American accents.
- Avg non-US: A macroaverage of all datasets featuring non-American accents.
To prevent developers from optimizing for a specific accent or provider, individual scores for each split are not provided. By default, the Average Word Error Rate (WER) on the leaderboard remains computed on public datasets only. Users can manually toggle the "Private data" splits on to see how including them affects the macroaverage and the resulting model ranking (Rank Δ).
Model Evaluation Process
Model developers can have their models evaluated on the private data by following these steps:
- Submit a pull request on the Open ASR Leaderboard GitHub repository.
- Report results on public datasets via a model checklist.
- Hugging Face verifies the public results and computes the metrics on the private datasets.
- The developer confirms the final results.
Alternatively, developers can self-report metrics on public sets by adding a YAML file to their model card, which places the model on an unverified leaderboard on the dataset page.
Standardization and Robustness
To maintain meaningful benchmarks, Hugging Face utilizes a normalizer based on the OpenAI Whisper normalizer. This tool standardizes model outputs and dataset transcripts by removing punctuation and casing and mapping text to American spelling. All test sets are consolidated into a single dataset on the Hub for easier access and previewing.