Open ASR Leaderboard adds Monsoon datasets for Hindi and Indian English

Hugging Face and Voice Arena have expanded the Open ASR Leaderboard by introducing the Monsoon evaluation sets for Hindi (hi-IN) and Indian English (en-IN). This update introduces the first Global South language to the leaderboard's multilingual tab, moving beyond aggregate Word Error Rate (WER) to expose how ASR performance varies across geography, age, gender, and device types.

Addressing Demographic Bias in ASR Evaluation

Aggregate WER often masks significant performance disparities across different speaker populations. Research has indicated that commercial ASR systems can perform significantly worse for Black speakers compared to white speakers, with further biases appearing across gender, age, and accent.

To address this, the Monsoon datasets provide a framework where the test set varies along nine specific axes to expose these failure modes:

  • Geography: Recruitment across hundreds of districts rather than a few locations.
  • Devices and Acoustic Environments: Use of contributors' own handsets and real-world connections (indoors and outdoors) instead of studio hardware.
  • Vocabulary, Speech Type, and Rate: Prompts designed to elicit opinion, disagreement, narration, and recall to capture unrehearsed phrasing and named entities.
  • Demographics: Verified age and gender for every speaker.
  • Reference Flexibility: Support for multiple valid transcripts for the same audio.

Dataset Composition and Speaker Diversity

The Monsoon collection consists of four splits (public and private for both languages) sourced from unscripted dual-channel spontaneous conversations.

Set Language Duration Speakers Mean/Median Clip Length M/F Districts Devices Style
Monsoon en-IN public Indian English 5.62 h 1,444 9.6s / 10.4s 50/50 428 556 Conversational
Monsoon en-IN private Indian English 5.58 h 1,405 9.6s / 10.4s 45/55 420 560 Conversational
Monsoon hi-IN public Hindi 1.33 h 468 6.4s / 5.0s 315 202 315 Conversational
Monsoon hi-IN private Hindi 4.47 h 1,571 6.6s / 5.3s 55/45 295 582 Conversational

Key Diversity Metrics

  • Low Speaker Concentration: The top 10 speakers account for only 2.8% to 6.8% of total duration, ensuring no single voice dominates the score.
  • Hardware Variance: Recordings span 315 to 582 distinct device models, preventing overfitting to specific microphone responses.
  • Regional Representation: The Indian English set covers all six zones of India, with 35% of segments from southern speakers, 18% from the East, 18% from Central, 16% from the North, and 11% from the West.

Regional Performance Disparities

The inclusion of detailed metadata (12 attributes per segment, including occupation, education, and native district) allows for granular analysis of model failure.

In an illustrative test on the public Indian English split, eight models showed nearly identical aggregate WER (between 4.81 and 4.99). However, when disaggregated by region, the variance became stark: mistralai/Voxtral-Mini-3B-2507 varied by 1.68 points across zones, performing significantly better in the Central zone (4.38) than in the East (6.06). This demonstrates that models which appear identical on a standard leaderboard can have vastly different reliability depending on the speaker's origin.

Handling Orthographic Variation in Hindi

Hindi presents a unique challenge because it lacks a canonical mapping for many spellings, particularly for code-mixed English words and compound forms. Standard WER penalizes models for choosing a valid orthographic variant that differs from the annotator's specific choice.

To solve this, the Hindi sets utilize a lattice—a list of all accepted correct spellings for each span of the transcript. Hugging Face now reports the Orthographically-Informed Word Error Rate (OIWER) for Hindi, which counts any admitted form in the lattice as correct.

Comparing WER against OIWER reveals that rankings can flip; a system may appear superior under a single reference simply because it happens to mirror the annotator's spelling preferences rather than because it has better acoustic recognition.

Integration and Evaluation Process

  • Indian English: Integrated into the main leaderboard as Voice Arena Monsoon, contributing to the headline Average WER.
  • Hindi: Available in the Multilingual tab and via the "Language dataset breakdown" dropdown.

To be evaluated on private splits, developers must submit their models via the Open ASR Leaderboard GitHub, providing results on public sets for verification before the Hugging Face team computes metrics on the private data.

Sources

Related

  • Dispatch
    Fine-Tuning Whisper for Multilingual ASR with Hugging Face Transformers

    Hugging Face provides a comprehensive guide on fine-tuning OpenAI's Whisper model for multilingual automatic speech recognition (ASR), demonstrating a 31.5% absolute WER improvement on Hindi using only 8 hours of data.

  • Dispatch
    Hugging Face Open ASR Leaderboard: Private Datasets to Combat Benchmaxxing

    Hugging Face has introduced private evaluation datasets from Appen Inc. and DataoceanAI to the Open ASR Leaderboard to prevent test-set contamination and provide a more robust measure of real-world ASR performance.

  • Dispatch
    Hugging Face FFASR Leaderboard: Benchmarking Far-Field ASR

    Hugging Face and Treble Technologies have launched the FFASR Leaderboard, the first open community-driven benchmark to quantify the performance gap between near-field and far-field Automatic Speech Recognition (ASR) in realistic acoustic environments.

  • Dispatch
    Hugging Face Open ASR Leaderboard Trends and Insights

    Hugging Face has updated the Open ASR Leaderboard with new multilingual and long-form tracks, revealing that Conformer encoders paired with LLM decoders currently lead in English accuracy while CTC/TDT decoders offer superior throughput.

  • Dispatch
    Fine-Tuning Wav2Vec2-BERT for Low-Resource ASR

    Hugging Face demonstrates how to fine-tune Meta's Wav2Vec2-BERT model for Automatic Speech Recognition (ASR) in low-resource languages, achieving performance comparable to Whisper-large-v3 while being significantly faster and more resource-efficient.