vLLM Adaptive Verification with DSpark

vLLM has implemented adaptive verification using DSpark's confidence-scheduled verification to optimize speculative decoding. By using a learned confidence head to score the survival probability of drafted tokens, vLLM can dynamically decide how many tokens to verify per step based on system load and token confidence, eliminating the need for manual tuning of num_speculative_tokens.

Optimizing Speculative Decoding for High Concurrency

Speculative decoding typically trades increased compute for fewer decode steps. While this is efficient at low batch sizes where GPUs are memory-bound, it becomes problematic at high concurrency (e.g., batch size 256) where draft tokens compete with real tokens for compute. When acceptance rates drop—such as the last token in a 7-token block on DeepSeek-V4-Pro-0813 surviving less than 10% of the time—rejected tokens waste critical compute and reduce overall throughput.

Adaptive verification solves this by replacing a static speculation length with a dynamic budget. Instead of verifying a fixed number of tokens, vLLM allocates verification slots to the most probable draft sequences across the entire batch, ensuring that compute is spent only on tokens with a high likelihood of acceptance.

The DSpark Confidence-Scheduling Mechanism

Adaptive verification utilizes the DSpark confidence head to assign a survival probability to each drafted token. The system determines the optimal draft budget ($B$) by maximizing the expected tokens produced per unit of step time using the following logic:

• Global Top-B Selection: The scheduler identifies the $B$ best draft slots across all requests. Because survival probability decreases as the position in the draft increases, the system simply admits a contiguous prefix of each request's draft.
• Budget Calculation: The budget $B$ is derived from a cost model that balances the expected bonus tokens (one bonus token per sampling request plus the survival of the $B$ best slots) against the profiled cost of the step (the sum of non-draft tokens $T$ and draft tokens $B$).
• Execution Pipeline: Budget sizing is performed on the CPU using a double-buffered confidence array from the previous step. The actual allocation of slots to requests is executed on the GPU via PyTorch and torch.compile (lowered to Triton), avoiding host-device read-backs.

Technical Implementation and CUDA Graphs

To support variable-sized verifications, vLLM integrated varlen decode CUDA graphs. This implementation relies on the following components:

• Attention Kernel Support: The system uses sparse MLA kernels and a varlen indexer kernel from DeepGEMM to handle variable lengths.
• Graph Capture: Decode graphs are captured with a maximum query length of num_speculative_tokens + 1. A single graph can serve any mix of 1 to num_speculative_tokens + 1 tokens per request.
• Cost Modeling: At startup, the engine profiles dummy steps to create lookup tables for verification and drafting costs. To handle profiling noise and kernel tile size variations, the cost curve is forced to be monotonic.
• CUDA Graph Padding: The cost model accounts for the "staircase" effect of CUDA graph padding, where a batch of 121 tokens may incur the cost of a 128-token graph. This encourages the budget algorithm to stay within the CUDA graph region for maximum efficiency.

Performance Results

Testing on DeepSeek-V4-Pro-0813 (TP=8 on 8×B300 SM100) demonstrated that adaptive verification consistently stays on the Pareto frontier of throughput versus interactivity across a concurrency sweep from 1 to 256.

The system effectively mimics a long fixed block at low concurrency and a short fixed block at high concurrency, providing the benefits of both without requiring prior knowledge of the workload shape.

Current Limitations

Adaptive verification currently has the following constraints:

• Hardware/Backend Requirements: Full varlen decode graphs require AttentionCGSupport.ALWAYS, currently reported by DSV4 sparse-MLA, sparse-SWA, and indexer backends on SM100.
• Unsupported Features: The feature is not compatible with --enforce-eager mode, LoRA, or pipeline parallelism.
• Logprobs: Output logprobs are not supported because verification compacts logits after the forward pass.

Configuration and Reproduction

Adaptive verification is enabled via the enable_adaptive_verification: true flag within the speculative-config JSON. For optimal performance, max_cudagraph_capture_size should be set to (num_speculative_tokens + 1) * max_num_seq to ensure verification batches remain within the captured graphs.