Speculators – A Library for Training Draft Models for Speculative Decoding

What it does (in plain language)

Speculators lets you train a small “draft” model that can guess the next few tokens of a large language model (LLM). During inference, the draft model proposes several tokens at once, and the big LLM (the verifier) checks them in a single pass. If the draft’s guesses match what the big model would have produced, they are accepted, cutting the number of expensive forward passes and speeding up generation without losing quality.

Why it matters

• Latency reduction: Fewer full‑model passes mean faster responses, which is crucial for real‑time applications.
• Production‑ready: The library ships models in a Hugging‑Face‑compatible format that can be dropped straight into vLLM, a high‑throughput inference server.
• Research‑grade flexibility: Supports several speculative‑decoding algorithms (DFlash, DSpark, P‑EAGLE, MTP) and multi‑node online training, so you can experiment with the latest ideas while still being able to deploy them.

Core components

Component	Role
Training pipelines	Generate hidden‑state data with vLLM, then train draft models (single‑ or multi‑layer, MoE, vision‑language, etc.).
hs_connectors	Pluggable back‑ends for moving hidden states between vLLM and the trainer across multiple nodes (file‑based or Mooncake distributed store).
Algorithm implementations	DFlash, DSpark, P‑EAGLE, MTP – each adds different drafting strategies (anchored‑block, Markov head, parallel multi‑token prediction, etc.).
Speculators format	A Hugging‑Face‑style model directory with a speculator_config that tells vLLM how to run the draft model together with its verifier.
CLI / utilities	speculators command for version checks, scripts for response regeneration, and integration with GuideLLM for benchmarking.

Typical workflow

1. Data generation – Use vLLM to run the large verifier model on a corpus, saving hidden states to disk.
2. Train a draft – Pick an algorithm (e.g., DSpark) and run the Speculators training script, optionally across nodes via hs_connectors.
3. Export – The trained draft is saved in the Speculators format.
4. Deploy – vllm serve <speculator_model> launches the draft+verifier combo with speculative decoding enabled automatically.
5. Benchmark – Use GuideLLM or the built‑in metrics to compare latency and acceptance rates against the dense verifier.

Who should use it?

• LLM service operators who need to cut inference latency on production servers.
• Researchers exploring new speculative‑decoding strategies; the library provides ready‑made implementations and a unified training interface.
• Developers building custom AI products who want a drop‑in upgrade to faster generation without rewriting their serving stack.

Getting started quickly

pip install speculators          # install the library
speculators --version           # sanity check
# Train a draft (example command – see docs for exact flags)
python -m speculators.train \
    --algorithm dspark \
    --verifier Qwen/Qwen3-8B \
    --output_dir ./my_speculator
# Serve with vLLM
vllm serve ./my_speculator

Documentation & community

• Full docs: https://docs.vllm.ai/projects/speculators/en/latest/
• Slack: vLLM Community Slack (#speculators, #feat-spec-decode)
• Office‑hours video & slides linked in the README.

---

Speculators is a genuine, production‑focused open‑source project that bridges the gap between cutting‑edge speculative decoding research and real‑world LLM serving.