openai/whisper

Whisper – Open‑source speech‑to‑text model

What it is – Whisper is a transformer‑based, sequence‑to‑sequence model that can turn audio into text. It was trained by OpenAI on a massive, multilingual audio dataset, so a single model can:

• transcribe speech in many languages,
• translate spoken language into English,
• identify the spoken language, and
• perform voice‑activity detection.

Why it matters – Instead of a pipeline of separate components (VAD → language ID → ASR → translation), Whisper does everything in one pass, which simplifies development and works well out‑of‑the‑box for a wide range of languages and audio conditions.

---

Model sizes

Size	Params	English‑only	Multilingual	VRAM (approx.)	Relative speed*
tiny	39 M	tiny.en	tiny	~1 GB	~10× (vs large)
base	74 M	base.en	base	~1 GB	~7×
small	244 M	small.en	small	~2 GB	~4×
medium	769 M	medium.en	medium	~5 GB	~2×
large	1 550 M	–	large	~10 GB	1×
turbo	809 M	–	turbo	~6 GB	~8×
*Speed measured on an A100 transcribing English; real‑world speed varies with language, hardware, and audio.

Tip: English‑only (*.en) models are a bit more accurate for English, especially the tiny and base versions. The turbo model is a speed‑optimized version of large‑v3 with a small accuracy trade‑off.

---

Quick start (CLI)

# install the package and ffmpeg (required for audio decoding)
pip install -U openai-whisper
# on Ubuntu/Debian you also need ffmpeg:
sudo apt update && sudo apt install ffmpeg

# transcribe one or more files (default uses the fast "turbo" model)
whisper audio.flac audio.mp3 audio.wav --model turbo

Use --language <Lang> to force a language, or --task translate with a multilingual model to get English translations.

---

Using Whisper from Python

import whisper
model = whisper.load_model("turbo")          # any size name works
result = model.transcribe("audio.mp3")
print(result["text"])                        # plain transcript

For lower‑level control you can:

1. Load and pad/trim audio with whisper.load_audio / whisper.pad_or_trim.
2. Convert to a log‑Mel spectrogram (whisper.log_mel_spectrogram).
3. Detect language with model.detect_language(mel).
4. Decode with whisper.decode(model, mel, whisper.DecodingOptions()).

---

Installation notes

• Requires Python 3.8‑3.11 and a recent PyTorch version.
• The package depends on OpenAI’s fast tokenizer tiktoken; on some platforms you may need a Rust toolchain (rustc, cargo) and setuptools‑rust to build it.
• ffmpeg must be available on the system path for the CLI and for audio loading in Python.

---

License

The code and model weights are released under the MIT License.

---

Where to learn more

• Blog post – https://openai.com/blog/whisper
• Paper – https://arxiv.org/abs/2212.04356
• Model card – https://github.com/openai/whisper/blob/main/model-card.md
• Colab demo – https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb
• Community examples – see the Show and tell discussion category in the repo.