Hugging Face Open ASR Leaderboard Trends and Insights

Hugging Face has introduced new multilingual and long-form transcription tracks to the Open ASR Leaderboard to provide a more comprehensive evaluation of Automatic Speech Recognition (ASR) models beyond short-form English transcription. As of November 21, 2025, the leaderboard compares over 60 open and closed-source models from 18 organizations across 11 datasets.

English Accuracy: Conformer Encoders and LLM Decoders

Models that combine Conformer encoders with large language model (LLM) decoders currently achieve the highest accuracy for English transcription. Leading models in this category include NVIDIA’s Canary-Qwen-2.5B, IBM’s Granite-Speech-3.3-8B, and Microsoft’s Phi-4-Multimodal-Instruct, all of which demonstrate that integrating LLM reasoning significantly reduces word error rates (WER).

To improve efficiency within this architecture, NVIDIA utilizes "Fast Conformer," a variant that is twice as fast as the standard Conformer, used in the Canary and Parakeet model suites.

Speed vs. Accuracy Tradeoffs

While LLM-based decoders provide superior accuracy, they are generally slower than simpler architectures. The leaderboard measures efficiency using the inverse real-time factor (RTFx), where higher values indicate better performance.

For applications requiring high throughput—such as real-time transcription, offline batch processing of lectures, or podcasts—CTC and TDT decoders are the optimal choice. These decoders deliver throughput that is 10–100× faster than LLM-based alternatives, though they result in slightly higher error rates.

Multilingual Performance and Specialization

There is a documented tradeoff between general multilingual capability and single-language specialization. OpenAI’s Whisper Large v3 remains a strong baseline supporting 99 languages, but distilled or fine-tuned variants like Distil-Whisper and CrisperWhisper often outperform the original on English-only tasks.

Key observations regarding multilingual ASR include:

  • Specialization Tradeoff: Focusing on English performance typically reduces a model's multilingual coverage.
  • Self-Supervised Systems: Meta’s Massively Multilingual Speech (MMS) and Omnilingual ASR support over 1,000 languages but trail behind language-specific encoders in terms of accuracy.
  • Localized Benchmarks: Community-driven efforts have expanded into language-specific leaderboards, such as the Open Universal Arabic ASR Leaderboard (covering Modern Standard Arabic and regional dialects) and the Russian ASR Leaderboard.

Long-Form Transcription Challenges

Closed-source systems currently maintain a performance lead in long-form audio transcription (e.g., meetings and podcasts), likely due to custom chunking, production-grade optimization, or domain tuning.

Among open-source options, OpenAI’s Whisper Large v3 is the top performer for accuracy. However, for throughput, CTC-based Conformers are significantly more efficient. For example, NVIDIA’s Parakeet CTC 1.1B achieves an RTFx of 2793.75 compared to Whisper Large v3’s 68.56, with only a moderate increase in WER (6.68 vs 6.43). Notably, Parakeet is limited to English only, further illustrating the tradeoff between multilingual support and specialized performance.

Sources