EAGLE-3 Speculative Decoding on AMD Instinct GPUs

EAGLE-3 Speculative Decoding on AMD Instinct GPUs

vLLM and the AMD Quark team have introduced an end-to-end pipeline for EAGLE-3 speculative decoding on AMD Instinct GPUs, enabling lossless inference acceleration for large mixture-of-experts (MoE) and attention-heavy models. This integration achieves throughput speedups ranging from 1.69x to 2.00x for Kimi-K2.5 and 1.38x to 1.79x for MiniMax-M2.5 on AMD Instinct MI355X GPUs.

Speculative Decoding and the EAGLE-3 Approach

Speculative decoding is a lossless acceleration technique that reduces the number of expensive target-model decode iterations. Instead of generating tokens one by one, a lightweight draft model proposes multiple future tokens, which the target model then verifies in a single forward pass. Tokens that match the target model's distribution are accepted, and generation continues from the first rejected token.

EAGLE-3 improves upon previous iterations by leveraging multi-layer features from the target model. Rather than using an unrelated small language model, EAGLE-3 trains a draft module closely aligned with the target model, utilizing low-, mid-, and high-level semantic features to increase the draft's acceptance rate and overall inference speed.

AMD Quark MXFP4 Quantization

To address memory-bandwidth and capacity constraints in large MoE models, the AMD Quark team provides "day-0" quantization for mainstream LLMs using the MXFP4 (OCP Microscaling 4-bit floating-point) format.

Key technical details of the MXFP4 implementation include:

  • Hardware Acceleration: AMD Instinct MI350-series GPUs (MI350X/MI355X) provide native FP4 matrix acceleration.
  • Memory Efficiency: MXFP4 groups 4-bit elements into small blocks with a shared scale factor, maintaining numerical behavior superior to INT4 while reducing memory footprint.
  • vLLM Integration: These checkpoints are consumed by vLLM on ROCm via the FP4 ASM GEMM path (VLLM_ROCM_USE_AITER_FP4_ASM_GEMM=1) and AITER MoE kernels.

vLLM-Centric EAGLE-3 Training Pipeline

Training a high-acceptance draft model is treated as a systems problem, with vLLM serving as the core of the training loop across five stages:

  1. On-policy Data Synthesis: vLLM serves the AMD Quark MXFP4 target to generate responses using the exact serving chat templates to ensure consistency between training and deployment.
  2. Hidden-State Extraction: vLLM's hidden-state extraction hook exposes auxiliary layers (low-, mid-, and high-level hidden states plus fc_norm) directly from the running target engine. This supports online, offline, and streaming modes, the latter enabling the training of a 420B MXFP4 MoE target on a single node.
  3. Cold-start FSDP2 Training: The single-layer EAGLE-3 draft head is trained from scratch using a training-time-test (TTT) loss and position-decay weighting under FSDP2.
  4. In-loop Serve-Eval: Current checkpoints are periodically exported and served under vLLM speculative decoding to measure true acceptance length, ensuring the selected checkpoint is optimized for the production engine.
  5. Export and Deployment: The final draft is exported to Hugging Face format and deployed via vLLM-ROCm.

Draft Quality and Context Scaling

Evaluation on SPEED-Bench shows the MiniMax-M3 EAGLE-3 draft averages an acceptance length (AL) of 2.77 across 11 domains. The highest acceptance rates are found in structured technical content:

  • Coding: 3.16 AL
  • Math: 3.12 AL
  • RAG: 3.11 AL
  • Multilingual: 3.07 AL

Notably, the acceptance length remains stable as prompt length increases from 1K to 32K tokens (2.64 to 2.63), indicating that speedup does not decay on long contexts.

Performance Benchmarks on AMD Instinct MI355X

Benchmarks conducted with 1K input and 1K output tokens (1K/1K) on AMD Instinct MI355X (TP=4) demonstrate significant throughput gains over non-speculative baselines:

Kimi-K2.5 Results

  • BF16 Draft Path: 1.69x to 1.90x throughput increase.
  • AMD Quark FP8 Draft Path: 1.76x to 2.00x throughput increase.

MiniMax-M2.5 Results

  • BF16 Draft Path: 1.38x to 1.79x throughput increase.

Across both models, the performance gains are most pronounced at lower concurrency levels.

Sources