EAGLE 3.1 Release Notes: Enhancing Speculative Decoding Robustness and Efficiency

EAGLE 3.1 Release Notes: Enhancing Speculative Decoding Robustness and Efficiency

EAGLE 3.1 is a major update to the speculative decoding framework designed to improve robustness, efficiency, and deployability. By addressing "attention drift," the update enables up to 2× longer acceptance lengths in long-context workloads compared to EAGLE 3.

Solving Attention Drift with Architectural Improvements

EAGLE 3.1 addresses a performance degradation phenomenon called "attention drift," where the drafter shifts attention away from sink tokens toward its own generated tokens as speculation depth increases. This instability is caused by two primary factors: an imbalanced fused input representation where higher-layer hidden states dominate, and growing hidden-state magnitude resulting from an unnormalized residual path.

To mitigate these issues, EAGLE 3.1 implements two key architectural changes:

  • FC Normalization: Adding Fully Connected (FC) normalization after each target hidden state and before the FC layer.
  • Post-Norm Hidden-State Feedback: Feeding post-norm hidden states into the subsequent decoding step.

These modifications transition the drafter's behavior to more closely resemble recursive invocation across decoding steps rather than the simple addition of layers to the target model. This results in higher resilience to chat template and system prompt variations, stronger long-context robustness, and better extrapolation from training-time to inference-time.

Training and Deployment via TorchSpec and vLLM

EAGLE 3.1 is integrated into the AI ecosystem through collaborations with TorchSpec and vLLM to streamline the path from research to production.

TorchSpec Training Support

TorchSpec now provides efficient training support for EAGLE 3.1, reducing training overhead and simplifying experimentation workflows. As a demonstration of this pipeline, an EAGLE 3.1 draft model for Kimi K2.6 has been open-sourced.

vLLM Integration

EAGLE 3.1 is implemented as a config-driven extension of the existing EAGLE 3 implementation in vLLM. The integration includes support for FC normalization, post-norm hidden-state feedback, and the removal of hardcoded assumptions regarding target hidden states.

Backward compatibility with EAGLE 3 checkpoints is fully preserved, allowing draft models to be plugged into the existing speculative-decoding code path. This support is available in the vLLM main branch, nightly releases, and the upcoming v0.22.0 release.

Performance Benchmarks

Benchmarks conducted on the SPEED-Bench coding dataset using the Kimi K2.6 EAGLE 3.1 draft model (running on Kimi-K2.6-NVFP4 with vLLM, TP=4, GB200, non-disaggregated) show significant throughput gains:

  • Concurrency 1: 2.03× higher per-user output throughput compared to the no-speculation baseline.
  • Concurrency 4: 1.71× higher throughput.
  • Concurrency 16: 1.66× higher throughput.

Ecosystem Collaboration

The development of EAGLE 3.1 is a joint effort between the EAGLE team (algorithmic research), the vLLM team (production inference optimization), and the TorchSpec team (training infrastructure), with GPU support provided by NVIDIA.

Sources