vLLM speculative decoding on AMD GPUs: performance and methods

TL;DR

Speculative decoding in vLLM lets a fast draft component propose several future tokens that the target model verifies in one pass, and on AMD Instinct MI300X/MI355X GPUs this technique yields up to ~2.9× throughput gains for some model‑method combinations while preserving the original model’s output behavior.

---

How speculative decoding works in vLLM

Speculative decoding adds a draft‑and‑verify stage in front of the original (target) LLM. The draft component generates a sequence of candidate tokens; the target model then evaluates the whole candidate block in a single forward pass. Tokens accepted by the target are committed, and the first rejected token is replaced by the target’s own output. The process repeats until the generation finishes.

Key properties

• The target model remains the source of truth; no output token is emitted before verification.
• Multiple draft tokens can be committed from one target‑model pass, reducing the number of expensive target passes.
• Acceptance is evaluated left‑to‑right; a rejection aborts the rest of the candidate block.

Drafting methods evaluated

vLLM supports five concrete drafting approaches, each differing in how the draft network receives information from the target model and whether it generates tokens sequentially or in parallel.

Method	Draft component	Target‑model information used	Token generation style
Native MTP	Model‑native auxiliary prediction path	Hidden representation from the target model (or previous MTP step)	Sequential (repeated MTP calls)
Gemma 4 MTP	Separate MTP checkpoint paired with the target	Target activations + shared KV cache	Sequential
EAGLE‑3	Dedicated autoregressive speculator	Fused hidden states from early, middle, and late target layers	Sequential, each draft token conditions the next
DFlash	Dedicated parallel speculator	Target hidden states projected to KV pairs used in every draft layer	All positions predicted together in one forward pass
DSpark	DFlash backbone + lightweight Markov head	Same target context as DFlash	Parallel backbone + sequential correction of token selection

Native MTP

Implemented inside the target model, MTP predicts a fixed number of future tokens using an auxiliary head. Draft tokens are generated one after another, each step consuming the hidden state from the previous MTP output. When num_speculative_tokens exceeds the native depth, vLLM re‑uses the MTP path through additional forward passes.

Gemma 4 MTP

A separate checkpoint (the assistant model) shares the target’s KV cache and activations, allowing it to reuse already‑computed context. Draft tokens are still generated sequentially, but memory overhead is modest because the draft component re‑uses target tensors.

EAGLE‑3

EAGLE‑3 trains a speculator that receives a fused representation of three target‑layer hidden states. The speculator runs autoregressively: the first draft token combines the fused target feature with the sampled token embedding; subsequent tokens condition on the previous draft output. This yields sequential draft work proportional to the proposal length.

DFlash

DFlash predicts an entire block of future positions in parallel. An anchor token—already verified by the target—starts the block, and the remaining masked positions are filled simultaneously. Target hidden states are projected into additional KV pairs that every draft layer can attend to, preserving target context throughout the parallel pass.

DSpark

DSpark builds on DFlash by adding a lightweight Markov head that introduces token‑wise dependence after the parallel backbone. The backbone produces base logits for all positions; the Markov head adjusts each position using the previously selected draft token, achieving a hybrid of parallel speed and sequential coherence.

---

Enabling speculative decoding in vLLM

Speculative decoding is configured via the --speculative-config flag. The JSON payload specifies the method, optional draft checkpoint, and num_speculative_tokens (the proposal length).

# Native MTP (no extra checkpoint)
vllm serve <target-model> \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'

# Gemma 4 MTP (requires assistant checkpoint)
vllm serve <target-model> \
  --speculative-config '{"method": "mtp", "model": "google/gemma-4-26B-A4B-it-assistant", "num_speculative_tokens": 4}'

# EAGLE‑3
vllm serve <target-model> \
  --speculative-config '{"method": "eagle3", "model": "RedHatAI/<target>-speculator.eagle3", "num_speculative_tokens": 3}'

# DFlash (parallel block)
vllm serve <target-model> \
  --speculative-config '{"method": "dflash", "model": "z-lab/<target>-DFlash", "num_speculative_tokens": 15}'

# DSpark (parallel + Markov correction)
vllm serve <target-model> \
  --speculative-config '{"method": "dspark", "model": "RedHatAI/<target>-speculator.dspark", "num_speculative_tokens": 7}'

Memory considerations – Native MTP shares weights with the target model, so it adds little extra GPU memory. All other methods load a separate draft checkpoint; the required headroom depends on checkpoint size, precision (FP16/INT4), tensor‑parallel degree, and runtime buffers.

---

Benchmark methodology

The authors measured output‑token throughput (tokens / second) on AMD Instinct MI300X and MI355X GPUs using the ROCm™ stack. Benchmarks used task‑grounded datasets (GSM8K, MATH500, HumanEval, MBPP) to reflect realistic acceptance patterns. For each target‑method pair, they swept num_speculative_tokens (N) and recorded:

• Throughput ratio (speculative / baseline)
• Mean accepted length (MAL) – average number of draft tokens accepted per verification round
• Acceptance rate (AR) – proportion of proposed tokens that survive verification

All measurements are relative to a non‑speculative autoregressive baseline run on the same hardware and software stack.

---

Key performance findings

Highest observed speedups

Target model	Method	Dataset	Best N	Throughput ratio
gemma‑4‑26B‑A4B‑it	DFlash	MATH500	7	2.87×
gemma‑4‑26B‑A4B‑it	Gemma 4 MTP	MATH500	5	2.74×
gemma‑4‑26B‑A4B‑it	EAGLE‑3	GSM8K	5	2.27×
gemma‑4‑31B‑it	DFlash	MATH500	7	2.41×
Qwen3‑8B	DSpark	GSM8K	7	1.63×
Qwen3‑6‑35B‑A3B	DFlash	MATH500	7	2.06×
Kimi‑K2.5	DFlash	GSM8K	7	2.37×

Trends across methods

• Sequential drafts (Native MTP, Gemma 4 MTP, EAGLE‑3) tend to improve throughput up to a modest proposal length (N ≈ 3‑5). Beyond that, the extra sequential drafting work outweighs the saved target passes, causing a plateau or decline.
• Parallel drafts (DFlash, DSpark) often achieve the highest ratios at larger N (N ≈ 7‑11). Acceptance rates drop for later positions, but the parallel cost remains low, so overall throughput still rises.
• DSpark adds a lightweight Markov head; its throughput is usually a few percent lower than pure DFlash for the same N because of the extra sequential correction step.
• Model family matters – native‑MTP built into Qwen 3.5/3.6 models outperforms DFlash for the same model size, whereas for Gemma 4 the parallel methods dominate.
• Workload dependence – code‑heavy datasets (HumanEval, MBPP) often favor shorter proposals because later tokens become less predictable, while math‑heavy datasets (GSM8K, MATH500) tolerate longer proposals.

Acceptance behavior

Across all experiments, the first draft token enjoys > 90 % acceptance. Acceptance declines monotonically with position; for DFlash at N = 15, the 15th token may be accepted < 5 % of the time. Mean accepted length (MAL) correlates strongly with throughput: higher MAL → fewer target passes → higher speedup.

---

Practical tuning guidance

1. Start small – For any method, begin with num_speculative_tokens = 1 (no extra cost) and verify correctness.
2. Sweep N – Increment N (e.g., 1‑3‑5‑7‑11‑15) while measuring throughput, MAL, and AR on a representative workload.
3. Observe per‑position AR – If acceptance drops sharply after position k, reduce N to k to avoid wasted draft work.
4. Memory budgeting – Ensure enough GPU memory for the draft checkpoint; parallel methods (DFlash/DSpark) usually require the most.
5. Hardware‑specific tuning – AMD Instinct GPUs benefit from the ROCm‑optimized attention backends (triton_attn). Parallel drafts leverage the high memory bandwidth of MI300X/MI355X.
6. Batch size & token limit – Larger max-num-batched-tokens and longer context windows improve amortization of the draft cost, especially for parallel methods.

---

Where to obtain draft checkpoints

Publisher	Supported methods	Example checkpoints
Google	Gemma 4 MTP	google/gemma-4-26B-A4B-it-assistant, google/gemma-4-31B-it-assistant
LightSeek	EAGLE‑3, EAGLE‑3.1	lightseekorg/kimi-k2.5-eagle3-mla
Red Hat AI	EAGLE‑3, DFlash, DSpark	RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3, RedHatAI/gemma-4-31B-it-speculator.dspark
Z‑Lab	DFlash	z-lab/gemma-4-26B-A4B-it-DFlash, z-lab/Qwen3.8B-DFlash-b16
DeepSeek AI	EAGLE‑3, DFlash, DSpark	deepseek-ai/eagle3_qwen3_8b_ttt7, deepseek-ai/dflash_qwen3_8b_block7
Inferact	EAGLE‑3, DSpark	Inferact/MiniMax-M3-EAGLE3, Inferact/Kimi-K3-DSpark

---

Training a new speculator (high‑level workflow)

1. Collect representative prompts for the intended workload (chat, code, math, etc.).
2. Generate target model responses using the exact tokenizer, chat template, and sampling settings that will be used at inference time.
3. Choose a hidden‑state extraction mode – online (on‑the‑fly), offline (pre‑saved), or hybrid (cache first epoch).
4. Extract required target layers (e.g., early, middle, late for EAGLE‑3; all selected layers for DFlash/DSpark).
5. Train the speculator with the same vocab, hidden size, and token embedding as the target. Include method‑specific heads (parallel block, Markov correction, etc.).
6. Validate acceptance rate, MAL, and end‑to‑end throughput; iterate on prompt mix or training hyper‑parameters if acceptance is low.
7. Package the checkpoint and serve it alongside the target model using --speculative-config.

---

Future directions

• Non‑learned speculation (e.g., n‑gram suffix prediction) could complement learned speculators for highly repetitive code‑editing workloads.
• Broader concurrency studies – measuring speculative decoding under multi‑user batch loads, varying batch sizes, and different sampling temperatures.
• Speculator data impact – systematic analysis of how prompt diversity and domain‑specific training data affect acceptance across code, math, chat, and multilingual tasks.
• Deeper profiling – isolating draft‑generation, target‑verification, KV‑cache reuse, and graph‑execution overhead on AMD GPUs to guide kernel‑level optimizations.

---

Acknowledgements

Thanks to Hongxia Yang and Peng Sun (AMD) and Pin Siang Tan, Jun Kang Chow, and Ye Hur Cheong (Embedded LLM) for hardware access and collaboration.

---

Disclaimer

Measurements were performed on two AMD Instinct configurations:

• Hardware 1: 8 × MI300X GPUs (gfx942) with dual EPYC 9654 96‑core CPUs.
• Hardware 2: 8 × MI355X GPUs (gfx950) with dual EPYC 9575F 64‑core CPUs (used for MiniMax‑M3‑MXFP8). Software stack: Ubuntu 22.04.5 LTS, ROCm 7.2.53211, vLLM 0.23.1rc1.dev1120+g0f0f28b53, PyTorch 2.11.0, Transformers 5.13.1, Python 3.12.13. Results may vary with different drivers, kernel versions, or hardware configurations.