Hugging Face FFASR Leaderboard: Benchmarking Far-Field ASR

Hugging Face FFASR Leaderboard: Benchmarking Far-Field ASR

Hugging Face and Treble Technologies have introduced the Far-Field ASR (FFASR) Leaderboard, an open benchmark designed to evaluate Automatic Speech Recognition (ASR) models under realistic acoustic conditions. This leaderboard addresses the persistent gap between high performance on clean-speech benchmarks and the significant degradation observed when models are deployed in real-world environments characterized by reverberation and background noise.

The Necessity of Far-Field Evaluation

Standard ASR evaluations typically rely on clean, close-microphone (near-field) benchmarks, which fail to predict how a model will behave in complex acoustic environments. As AI voice agents, humanoid robots, and in-car assistants move beyond headsets and smartphones, they must operate in spaces where microphones are located several meters from the speaker.

Existing research efforts like CHiME, URGENT, and NOIZEUS provide valuable data, but the community lacked a standardized, open, and continuously updated leaderboard to measure acoustic degradation consistently across different models. The FFASR Leaderboard fills this gap by providing a systematic way to quantify how far-field conditions affect Word Error Rate (WER).

Benchmark Construction and Methodology

The FFASR Leaderboard utilizes a hybrid wave-based simulation engine from Treble Technologies to generate acoustic data. This engine combines a wave-based solver for low to mid frequencies with geometrical-acoustics modeling for higher frequencies to accurately capture physical phenomena such as diffraction, scattering, and interference.

Evaluation Conditions

Models are evaluated across nine conditions, with the primary ranking determined by four key metrics:

  • Near-field (dry): Clean speech recorded in an anechoic chamber.
  • Far-field high SNR: Signal-to-Noise Ratio (SNR) above 14 dB.
  • Far-field mid SNR: SNR between 8 and 12 dB.
  • Far-field low SNR: SNR below 6 dB.

Dataset and Environment

  • Room Diversity: The benchmark includes 14 fully furnished rooms (20 to 470 m³), including offices, classrooms, bathrooms, and restaurants.
  • Audio Composition: Each scene features one target speaker (recorded anechoically) and up to three noise sources, including both transient noises (e.g., coughing) and continuous noises (e.g., HVAC).
  • Validation: The benchmark includes "Lab Measured" and "Lab Simulated" tracks to provide sim-to-real validation.
  • Moving-Source Splits: Currently in beta, these evaluate models against audio where the speaker is in motion, simulating use cases like mobile voice assistants or humanoid robots.

Performance Metrics

Beyond Word Error Rate (WER), the leaderboard reports RTFx (audio seconds per inference second), measured on an NVIDIA L4 GPU. This allows developers to analyze the Pareto front of accuracy versus latency, which is critical for real-time deployment.

Key Findings and Data Insights

Initial data from the leaderboard reveals a substantial performance gap between near-field and far-field conditions. While near-field WER values are comparable to established benchmarks, far-field WER at low SNR is consistently several times higher across all submitted models.

By reporting near-field and far-field WER side-by-side, the leaderboard enables developers to distinguish between models that are inherently accurate and those that are brittle to acoustic conditions. This distinction helps teams decide whether to invest in architectural changes, speech enhancement preprocessing, or far-field fine-tuning.

Submission Process and Integration

The leaderboard uses a held-out evaluation set of 2,000 anechoic speech samples across 14 rooms at three SNR tiers (approximately 8 hours of audio per condition), with Whisper-style text normalization. To prevent test-set contamination, the audio is not exposed to submitters.

Supported Architectures

The pipeline supports a wide range of ASR architectures available on the Hugging Face Hub, including:

  • Whisper variants
  • IBM Granite Speech
  • Cohere Transcribe
  • Wav2Vec2 and HuBERT CTC heads
  • SpeechBrain ASR

For complex inference stacks that combine ASR with speech enhancement, a custom evaluate() function can be submitted for moderator review and execution via Hub Jobs.

Future Roadmap

The FFASR Leaderboard intends to evolve based on community feedback. Planned future tracks include:

  • Multi-talker scenarios: Evaluating performance when multiple speakers are active simultaneously.
  • Microphone array support: Testing beamforming and spatial filtering approaches.
  • Echo cancellation: Evaluating devices that must listen while simultaneously playing audio.

Sources