vllm-project/speculators

What it solves

Speculators is a library designed to speed up Large Language Model (LLM) inference without losing output quality. It does this by training "draft models" (speculators) that propose tokens which are then verified by a larger base model. This process, known as speculative decoding, reduces latency by allowing the base model to verify multiple tokens in a single forward pass rather than generating them one by one.

How it works

The library provides an end-to-end framework to train these draft models. It supports several advanced algorithms including EAGLE-3, P-EAGLE (which predicts multiple tokens in a single pass), and DFlash (which uses anchored-block drafting). It also allows for offline training data generation by extracting hidden states from a verifier model using vLLM, or online training via a real-time hidden extraction system. Trained models are saved in a Hugging Face-compatible format and can be deployed directly into vLLM inference servers for production use.

Who it’s for

ML engineers and researchers who want to optimize LLM serving latency in production environments, specifically those using vLLM as their inference engine.

Highlights

• Direct vLLM Integration: Models trained with Speculators can be served seamlessly using the vllm serve command.
• Diverse Algorithm Support: Supports EAGLE-3, P-EAGLE, DFlash, and Multi-Token Prediction (MTP) finetuning.
• Flexible Training: Offers both offline and online training modes for generating necessary hidden states.
• Broad Model Support: Compatible with Llama, Qwen3, gpt-oss, and Gemma 4, including MoE and Vision Language models.
• Memory Efficiency: Implements sliding window attention for DFlash and DSpark speculators to reduce KV cache allocation for long sequences.