OpenAI Whisper Release

OpenAI has introduced Whisper, an automatic speech recognition (ASR) system designed for high robustness across diverse audio environments. By training on a massive dataset of 680,000 hours of multilingual and multitask supervised data collected from the web, Whisper improves transcription accuracy in the presence of background noise, various accents, and technical language.

Architecture and Technical Implementation

Whisper utilizes a simple end-to-end encoder-decoder Transformer architecture. The processing pipeline follows these steps:

  1. Audio Processing: Input audio is divided into 30-second chunks.
  2. Feature Extraction: These chunks are converted into a log-Mel spectrogram.
  3. Encoding: The spectrogram is passed through an encoder.
  4. Decoding: A decoder predicts the corresponding text caption.

To handle multiple tasks within a single model, Whisper uses special tokens. These tokens direct the model to perform specific functions, including language identification, phrase-level timestamps, multilingual speech transcription, and translation of non-English speech into English.

Performance and Robustness

Whisper's primary strength is its general-purpose robustness rather than performance on narrow, specialized benchmarks.

  • Zero-Shot Performance: When measured across diverse datasets, Whisper demonstrates significantly higher robustness than models trained on smaller, closely paired audio-text datasets, making 50% fewer errors in zero-shot scenarios.
  • Specialized Benchmarks: Because it was not fine-tuned for specific datasets, Whisper does not outperform models that specialize specifically in LibriSpeech performance.
  • Multilingual Capabilities: Approximately one-third of the training data is non-English. This allows the model to excel at both transcribing original languages and translating them into English. In zero-shot testing, Whisper outperforms the supervised state-of-the-art (SOTA) on CoVoST2 to English translation.

Open Source Availability

To support the development of voice interfaces and further research into robust speech processing, OpenAI has open-sourced the Whisper models and the associated inference code.

Sources