Apple SpeechAnalyzer Benchmark: Performance vs Whisper and SFSpeechRecognizer

Apple SpeechAnalyzer outperforms Whisper Small and legacy APIs in on-device English transcription

Apple's new SpeechAnalyzer API is the most accurate on-device speech engine for English among those tested, surpassing Whisper Small in both clean and noisy audio environments while operating approximately three times faster. It also represents a massive leap over its predecessor, SFSpeechRecognizer, which exhibited significantly higher Word Error Rates (WER) across all tested datasets.

Performance Benchmarks

Testing conducted on an Apple M2 Pro (32GB, macOS 26.5.1) using the LibriSpeech dataset reveals that SpeechAnalyzer provides the lowest Word Error Rate (WER), where lower percentages indicate higher accuracy.

Engine test-clean WER test-other WER Model size
Apple SpeechAnalyzer (iOS/macOS 26) 2.12% 4.56% system
Whisper Small (WhisperKit CoreML) 3.74% 7.95% ~460MB
Whisper Base 5.42% 12.51% ~140MB
Whisper Tiny 7.88% 17.04% ~40MB
Apple SFSpeechRecognizer (legacy) 9.02% 16.25% system

Migration from SFSpeechRecognizer to SpeechAnalyzer

Developers should migrate from the legacy SFSpeechRecognizer to SpeechAnalyzer immediately. The new API reduces the word error rate by 3.5x to 4x on the same audio. Specifically, WER dropped from 9.02% to 2.12% for clean speech and from 16.25% to 4.56% for noisy speech. Beyond raw accuracy, SpeechAnalyzer produces superior punctuated and cased text compared to the rougher output of the legacy engine.

SpeechAnalyzer vs. OpenAI Whisper

For English transcription on current Apple hardware, SpeechAnalyzer is now the strongest on-device option, beating Whisper Small by a comfortable margin while requiring only a third of the compute time per second of audio.

Despite this, Whisper maintains two primary advantages:

  1. Language Support: Whisper covers significantly more languages, whereas SpeechTranscriber supports approximately 30 locales.
  2. Platform Agnostic: Whisper runs across various platforms, while SpeechAnalyzer is restricted to Apple platforms running OS 26.

Speed and Efficiency

All tested engines operated faster than real-time on the M2 Pro, with transcription speeds ranging from 12x to 40x. SpeechAnalyzer was roughly 3x faster than Whisper Small per second of audio.

Technical Methodology and Validation

To ensure the validity of the benchmarks, the following controls were implemented:

  • Reproducibility: The Whisper results were compared against OpenAI's published numbers for LibriSpeech, showing a consistent, small positive offset attributed to CoreML quantization and a stricter text normalizer.
  • Production Code Paths: Engines were run through actual production code rather than a lab harness to mirror real-world usage.
  • Text Normalization: All outputs passed through a normalizer (casing, punctuation, digits-to-words) to ensure engines were not penalized for formatting.
  • On-Device Verification: The harness was configured to refuse execution if it fell back to cloud recognition, ensuring all results were strictly on-device.

Developer and User Insights

Community feedback and developer reports highlight several practical advantages and limitations of the SpeechAnalyzer API:

  • Streaming Capabilities: Unlike many models that require a full recording before transcription, SpeechAnalyzer supports streaming, allowing users to see results in real-time.
  • App Bundle Size: Because the model is part of the system, developers do not need to bundle the model within their app, reducing the final application size.
  • Language Management: The API treats locales as on-demand resources, which saves disk space but requires separate model downloads per language, making multi-lingual audio harder to transcribe without prior knowledge of the languages present.
  • Comparison to Larger Models: Some users report that while SpeechAnalyzer is faster, larger models like Whisper Large-V2 may still be slightly more accurate for specialized use cases, such as math lectures.

"SpeechAnalyzer almost always gets it [diverse audio]. It can struggle with proper nouns but will return something phonetically similar. My main gripe is that it requires a separate model download per language."

Limitations of the Study

  • Language Scope: The benchmark is limited to English; it does not account for the 100+ languages supported by Whisper.
  • Audio Type: The LibriSpeech corpus consists of read audiobook speech, which may differ from the performance seen in accented, far-field, or multi-speaker meeting environments.
  • Hardware: Results were generated on a single M2 Pro machine; while accuracy should be consistent across Apple Silicon, speed will vary by chip.

Sources

Related

  • Project
    m-bain/whisperX

    A fast automatic speech recognition system that enhances OpenAI's Whisper with batched inference, word-level timestamps, and speaker diarization.

  • Project
    openai/whisper

    Whisper is OpenAI’s open‑source, transformer‑based speech‑to‑text system that handles transcription, language identification, and translation across many languages. It ships as a pip‑installable package with six model sizes (tiny → large, plus a fast “turbo” variant). The CLI (`whisper …`) and a simple Python API (`whisper.load_model(...).transcribe(...)`) let you turn audio files into text with just a few commands. The project runs on Python 3.8‑3.11, PyTorch, and requires ffmpeg (and optionally Rust for the tokenizer). All code and model weights are MIT‑licensed.

  • Dispatch
    OpenAI Whisper Release

    OpenAI has released Whisper, an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data to improve robustness across accents, noise, and languages.

  • Dispatch
    Fine-Tuning Whisper for Multilingual ASR with Hugging Face Transformers

    Hugging Face provides a comprehensive guide on fine-tuning OpenAI's Whisper model for multilingual automatic speech recognition (ASR), demonstrating a 31.5% absolute WER improvement on Hindi using only 8 hours of data.

  • Dispatch
    Hugging Face Inference Endpoints: Fast Whisper Transcriptions

    Hugging Face has introduced a new OpenAI Whisper deployment option on Inference Endpoints that delivers up to 8x performance improvements in transcription speed without sacrificing accuracy.