Speculators v0.5.0 release notes / what's new

Speculators v0.5.0 introduces support for the DFlash algorithm for single-pass draft token generation and unifies online and offline training workflows through vLLM's native hidden states extraction system. These updates improve training flexibility, reduce speculative decoding overhead, and enhance production readiness for speculative decoding workflows.

DFlash Algorithm Support

Speculators v0.5.0 adds training support for DFlash, a speculative decoding algorithm that generates all draft tokens in a single forward pass using block diffusion. This differs from autoregressive models like Eagle 3, which require multiple forward passes to generate draft tokens.

Key technical characteristics of DFlash include:

• Single-pass generation: DFlash produces a block of tokens of length B for each prefix, reducing overhead for longer draft sequences.
• Non-causal attention: Within a block, queries can attend to all other tokens in the same block, utilizing a specific attention mask.
• Optimized training anchors: To prevent the attention mask from growing too large and consuming excessive memory/compute, DFlash does not start prediction blocks at every point in a sequence. Instead, it randomly selects a smaller set of "anchor" positions that contribute to training loss, attaching predicted blocks only to these anchors.

Training a DFlash Speculator

Training DFlash models follows an online workflow similar to Eagle 3 but requires specific parameters. The training command utilizes torchrun and includes the following DFlash-specific flags:

• --speculator-type dflash: Specifies the algorithm.
• --block-size: Defines the number of tokens generated per diffusion block.
• --max-anchors: Sets the maximum number of anchor points for speculation during training.

Gemma 4 DFlash Performance

Using the new DFlash support, a Gemma 4 31B DFlash speculator was trained. Evaluation across diverse tasks shows strong performance, particularly in reasoning and code generation.

Performance benchmarks indicate that Gemma 4 DFlash achieves better inter-token latency than both Eagle 3 and a standalone FP8 quantized verifier. When combined with an FP8 quantized verifier, the inter-token latency gains are further increased.

vLLM Integration and Serving

DFlash models are integrated into vLLM's speculative decoding infrastructure via PR #38300, supported in vllm>=0.20.0.

Models include a speculators_config in their config.json file, which specifies the target model and the speculative algorithm. This allows DFlash models to be served using the standard vllm serve command:

vllm serve -tp 2 RedHatAI/gemma-4-31B-it-speculator.dflash

Unified Online and Offline Training

Speculators v0.5.0 migrates to vLLM's native hidden states extraction system (introduced in vLLM v0.18.0). This replaces previous lower-level utilities and removes vLLM as a direct Python dependency, decoupling the training pipeline from vLLM's internal APIs.

Training Modes

Both online and offline training now utilize the same vLLM-based extraction path:

• Online training: Hidden states are extracted on-the-fly during training. The process involves sending prompts to a vLLM server, extracting hidden states to disk or RAM disk, loading them for training, and then deleting the file.
• Offline training: Hidden states are pre-generated and cached to disk before training begins.

Because these modes are tightly coupled, they can be used in combination. Users can partially generate hidden states offline and load them during online training, or run online training without clearing files to reuse them in subsequent epochs.

By leveraging the native extraction system, Speculators inherits vLLM's inference optimizations, including hardware acceleration and efficient batching strategies, while communicating with the vLLM server via a standard REST API.

Documentation Updates

The release includes an updated documentation site featuring:

• Introductions to supported speculative decoding algorithms.
• Detailed tutorial walkthroughs for training speculator models.
• A developer guide for adding new speculative decoding algorithms to the library.
• A comprehensive API reference.