vllm-project/speculators

What it solves

Speculators addresses the high latency of Large Language Model (LLM) inference. It provides a production-ready framework to train "draft models" (speculators) that can predict multiple tokens ahead of time, which a larger base model then verifies in a single pass. This reduces the number of expensive forward passes required by the main model without changing the final output quality.

How it works

The library standardizes the end-to-end process of creating these draft models. It allows users to generate training data (hidden states) using vLLM, train draft models using various algorithms, and deploy them directly into vLLM inference servers. It supports multiple training architectures, including single and multi-layer models, and is compatible with MoE, non-MoE, and Vision Language models.

Who it’s for

It is designed for AI engineers and researchers who want to implement speculative decoding to speed up LLM deployment in production environments.

Highlights

• Direct vLLM Integration: Models trained with Speculators can be deployed seamlessly using vllm serve.
• Diverse Algorithm Support: Includes support for EAGLE-3, DFlash, DSpark, and P-EAGLE training algorithms.
• ** uma Multi-Node Training:** Supports online training across multiple nodes via hs_connectors using shared filesystems or distributed stores.
• Flexible Model Support: Compatible with a wide range of architectures, including Llama, Qwen, Gemma, and NVIDIA Nemotron.
• MTP Finetuning: Supports finetuning native Multi-Token Prediction heads on domain-specific data.