Speculative Decoding Enables 2× Faster Whisper Inference

TL;DR

Hugging Face shows that applying speculative decoding to OpenAI's Whisper model reduces inference time by roughly 2× (e.g., from 73 s to 33 s on a small benchmark) without any loss in transcription quality, because the method guarantees exactly the same token outputs as the original model.

What Is Speculative Decoding?

Speculative decoding, introduced by Leviathan et al. (2022), pairs a fast assistant model with a larger main model. The assistant generates a short candidate token sequence (e.g., five tokens). The main model then verifies this sequence in a single forward pass, accepting all tokens up to the first mismatch and discarding the rest. The corrected prefix becomes the new context for the assistant, and the cycle repeats. Because the assistant runs much faster than the main model, overall decoding speed improves while the verification step ensures identical final outputs to using the main model alone.

Key requirements:

  • The assistant must share the exact tokenizer/vocabulary with the main model.
  • It should be at least 3× faster than the main model, while correctly predicting the majority (≈70‑80 %) of “easy” tokens.

Baseline Whisper Inference Speed

The authors benchmarked Whisper large‑v2 on a 73‑sample LibriSpeech validation set (≈9 MB total) using:

  • float16 precision, FlashAttention (attn_implementation="sdpa"), and low‑CPU‑mem loading.
  • Generation time measured per sample.

Result: 72.99 seconds total (≈1 s per sample) with a word error rate (WER) of 3.5 %.

Applying Speculative Decoding to Whisper

Assistant Model Choice

  • Distil‑Whisper distil‑large‑v2 was selected as the assistant. It retains Whisper’s encoder but uses only 2 of the 32 decoder layers, yielding a 6× speed advantage while staying within 1 % WER of the full model on out‑of‑distribution data.
  • The encoder can be shared between main and assistant, so the VRAM overhead is modest (≈8 % extra).

Implementation Details

assistant_model = AutoModelForCausalLM.from_pretrained(
    "distil-whisper/distil-large-v2",
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
    attn_implementation="sdpa",
).to(device)

# Generation with assistant
outputs = model.generate(**inputs, assistant_model=assistant_model, **kwargs)

The assistant_model argument activates the assisted generation strategy in 🤗 Transformers, which implements speculative decoding.

Speedup Results (English)

  • Speculative decoding time: 32.70 seconds total (≈0.45 s per sample).
  • Speedup: 2.2× faster than the baseline.
  • WER: Identical at 3.5 %, confirming exact output preservation.

The same approach works with the high‑level pipeline API by passing generate_kwargs={"assistant_model": assistant_model}.

Multilingual Transcription

Distil‑Whisper checkpoints are English‑only, so for multilingual use the authors employed the smallest multilingual Whisper checkpoint (tiny) as the assistant.

Benchmark on 73 Dutch samples from VoxPopuli:

  • Baseline (large‑v2): 116.5 s, WER 12.8 %.
  • Speculative decoding (assistant = tiny): 62.1 s, WER 12.8 %.
  • Speedup: 1.9×.

The method works for both transcription and translation tasks by supplying the appropriate language and task arguments to generate.

Strategies for Maximizing Efficiency

Selecting an Assistant Model

  • Aim for ≥ 3× speed over the main model and ≥ 70‑80 % token agreement.
  • For a specific language, fine‑tune a large Whisper model (e.g., large‑v3) as the main model and distil the same architecture to create a fast assistant. This aligns token distributions and improves WER for both models.

Batch Size Considerations

  • Speculative decoding yields the greatest gains with batch size = 1.
  • Larger batches require all candidates across the batch to match the main model; mismatches cause early discarding, diminishing speedups.
  • Empirically, speedups persist up to batch size 4; beyond that, the overhead outweighs benefits (see Distil‑Whisper paper, Sec. D.3).

Practical Takeaways

  • Speculative decoding is a drop‑in replacement for existing Whisper pipelines: simply add the assistant_model argument.
  • It delivers free 2× inference acceleration while guaranteeing unchanged transcription quality.
  • The approach is model‑agnostic; any transformer‑based sequence model can benefit if a suitable faster assistant sharing the same tokenizer exists.

“Speculative decoding offers the perfect drop‑in replacement for existing Whisper pipelines, since it provides free 2× speed‑up while maintaining the same accuracy.” — Sanchit Gandhi, Hugging Face blog (2023‑12‑20)

Acknowledgements: The post credits Patrick von Platen, Pedro Cuenca, and Joao Gante for feedback and the assisted‑generation implementation in 🤗 Transformers.

Sources