eagle-hq/EAGLE-4

What is EAGLE‑4?

EAGLE‑4 (Extrapolation Algorithm for Greater Language‑model Efficiency) is a speculative decoding framework that speeds up inference of large language models. It provides a single “drafter” that automatically adapts its speculative tree to the batch size, hardware (NVIDIA or AMD GPUs) and workload, while preserving the target model’s output distribution.

Core idea

• Speculative decoding: a lightweight draft model predicts several tokens ahead; the heavyweight target model only verifies them, reducing the amount of expensive forward passes.
• Dynamic tree sizing: EAGLE‑4 measures draft‑and‑verification costs at runtime and chooses the optimal number of speculative steps for the current batch and hardware.
• Plug‑in architecture: the eagle‑harness package registers hooks into supported inference engines (SGLang, vLLM, TokenSpeed) without patching their source code.

What the repository contains

• eagle_harness – a pip‑installable Python package that implements the harness logic and engine plugins.
• Documentation – quick‑start guides, calibration procedures, compatibility tables, and detailed architecture notes.
• Release assets – pre‑built wheels and Docker images for the supported engines, plus a support matrix with measured speed‑ups.

Supported engines & hardware

Engine	GPU family	How it’s delivered
SGLang	NVIDIA B300 (CUDA 13) or AMD MI355X (ROCm 7.2)	Wheels (cu130 / rocm) and Docker image eagle‑4‑sglang:0.1.0
vLLM	NVIDIA B300 (CUDA 13)	Wheels (cu130) and Docker image eagle‑4‑vllm:0.1.0
TokenSpeed (experimental)	NVIDIA CUDA only	Wheels (cu130) and Docker image eagle‑4‑tokenspeed:0.1.0

Quick installation (example for SGLang on CUDA)

# Clone the repo
git clone https://github.com/eagle-hq/EAGLE-4 && cd EAGLE-4
# Load version pins
. docs/release.env
# Create a virtual env for SGLang
python3.12 -m venv .venv && source .venv/bin/activate
# Install SGLang and its dependencies
pip install -U pip
pip install --extra-index-url "$EAGLE_CUDA_INDEX" \
    --extra-index-url "$EAGLE_TORCH_CUDA_INDEX" \
    "sglang==$EAGLE_SGLANG_VERSION" 'flashinfer-python>=0.7.2'
# Install the harness
pip install "eagle-harness==$EAGLE_VERSION"
# Verify the plugin can see the engine
eagle-harness-check

The same pattern works for vLLM or TokenSpeed, just swapping the engine name and Python version.

Running a server (SGLang example)

python -m sglang.launch_server \
  --model-path $MODEL \
  --tp-size 4 --dtype bfloat16 \
  --attention-backend flashinfer \
  ... (other performance flags) ... \
  --speculative-algorithm EAGLE_HARNESS \
  --speculative-draft-model-path $DRAFT \
  --speculative-num-steps 12 \
  --speculative-eagle-topk 4 \
  --speculative-num-draft-tokens 16 \
  --speculative-algorithm-config '{"mode":"serve"}'

Then a simple curl request checks that generation works.

Measured performance

The paper’s support matrix reports up to ~3× token‑per‑second speed‑up over vanilla decoding on 4× NVIDIA B300 GPUs (e.g., SGLang: 1479 → 2820 t/s, vLLM: 242 → 723 t/s). Speed‑ups vary with batch size and hardware, but the framework always falls back to correct greedy output if the speculative path diverges.

License & commercial terms

• The code is source‑available under the Eagle 4 Source‑Available License (not OSI‑approved). It permits non‑commercial use and limited commercial use up to $20 M revenue; larger commercial deployments require a separate license.
• The repository also contains third‑party notices for SGLang, vLLM, FlashInfer, etc.

Who might use this?

• ML engineers looking to squeeze more throughput out of existing LLM deployments without retraining.
• Cloud or on‑premise inference services that need a single drafter that works across varying batch sizes and GPU families.
• Research teams experimenting with speculative decoding algorithms, as the harness exposes calibration tools and detailed logs.

---

TL;DR: EAGLE‑4 is a plug‑in harness that adds adaptive speculative decoding to popular LLM serving stacks (SGLang, vLLM, TokenSpeed), delivering up to ~3× faster token generation on modern NVIDIA/AMD GPUs while keeping the model’s output distribution intact.