vLLM Kimi-K3 DSpark Speculative Decoding Implementation

Overview

vLLM has successfully trained and deployed a DSpark speculator for Kimi K3, a 2.8T-parameter frontier model. By utilizing the Speculators training library and GB300 NVL72 hardware, the implementation increases single-stream interactivity for math reasoning workloads from approximately 110 to 435 tok/s/user and provides up to 3.5× higher output throughput under concurrent load.

The DSpark Algorithm

DSpark is an extension of the DFlash block-level speculative decoding algorithm designed to solve "suffix decay," where a single mistake in a parallel-predicted block invalidates the remaining tokens. DSpark maintains the parallel backbone of DFlash but introduces two specific components to improve intertoken coherence:

  • Markov logit-bias head: This component samples tokens sequentially and uses a low-rank transition matrix to adjust logits based on the previously selected token, restoring local dependencies without requiring additional transformer passes.
  • Confidence head: This head estimates the probability of token acceptance, allowing a hardware-aware scheduler to verify longer prefixes during low load and trim unlikely suffixes during high system load.

Compared to DFlash, DSpark reports 16–18% longer accepted sequences on Qwen3 target models, and 27–31% longer sequences than EAGLE-3.

Inference Performance and Capabilities

The Kimi K3 DSpark speculator utilizes a five-layer, five-billion-parameter draft model proposing eight tokens per decoding step.

Benchmarks and Throughput

Across nine evaluation domains, the model achieved a macro-average acceptance length of 4.11 tokens per verification round. Performance was highest in structured tasks:

  • Mathematical reasoning: 6.42 tokens
  • HumanEval: 4.96 tokens
  • Translation: 4.65 tokens

Long-Context Performance

On the LongBench-v2 dataset, the speculator reached up to 5.31 output tokens per decode iteration on a 378K-token prompt. The top 10% of requests maintained at least 3.76 tokens per iteration, indicating that speculative decoding remains effective at extreme context lengths.

Concurrency and Latency

As concurrency increased from 1 to 16, aggregate output throughput rose from 177 to 683 tokens per second. Median time to first token (TTFT) remained stable, increasing by only 100 milliseconds (from 379 to 479 ms) despite the 16x increase in concurrent requests.

Hardware and Training Infrastructure

GB300 NVL72 Configuration

Training was conducted on a GB300 rack using Ubuntu 24.04.4 LTS and NVIDIA's 64K-page kernel 6.14. The environment utilized NVIDIA's 610.57.04 open-kernel GPU driver (R610) and CUDA 13.4.0 Developer Preview, the first toolkit to include Rubin support (sm_107).

Mooncake Hidden-State Transfer

Because draft models often require hidden states from the target model to align predictions, vLLM implemented the MooncakeHiddenStatesConnector. This system enables disaggregated training and hidden-state extraction, which is necessary for models as large as Kimi K3 (2.8T parameters) that exceed the VRAM limits of single-node configurations even with 4-bit quantization.

  • Mechanism: A master Mooncake proxy process manages communication between vLLM and training instances. Training processes request hidden states via the vLLM frontend, receive a Mooncake store key, and the Mooncake master brokers the transfer using RDMA or TCP.
  • Topology: The optimal configuration found for Kimi K3 involved sets of three nodes: two nodes dedicated to vLLM inference and one node dedicated to training.

Deployment

The Kimi K3 DSpark speculator is available via Hugging Face (RedHatAI/Kimi-K3-speculator.dspark) and can be deployed using vLLM with the following speculative configuration:

{
  "model": "RedHatAI/Kimi-K3-speculator.dspark",
  "num_speculative_tokens": 8,
  "method": "dspark",
  "draft_sample_method": "probabilistic",
  "rejection_sample_method": "block"
}

Sources