Kimi K3 Performance Optimizations in vLLM
vLLM has implemented a comprehensive series of performance optimizations for Kimi K3, achieving up to a 2.8× throughput increase and a reduction in Time to First Token (TTFT) by up to 85%. These improvements were achieved by addressing bottlenecks across the entire serving stack, including KDA recurrent state, LatentMoE, MXFP4 expert kernels, and speculative decoding.
Serving Performance Gains
Optimizations measured on a B300 node (CUDA 13.3) using an 8K/1K workload with TP8 and eight-token DSpark speculation show significant gains when comparing vLLM v0.27.1 to the September 13th main commit (82a85dc1).
| Concurrency | v0.27.1 Avg Latency (s) | 0913 main Avg Latency (s) | v0.27.1 Throughput (tok/s) | 0913 main Throughput (tok/s) | v0.27.1 Avg TTFT (ms) | 0913 main Avg TTFT (ms) |
|---|---|---|---|---|---|---|
| 1 | 12.37 | 5.30 (−57.2%) | 83.3 | 183.3 (+120.0%) | 2262.9 | 376.3 (−83.4%) |
| 4 | 23.67 | 10.50 (−55.6%) | 166.7 | 416.7 (+150.0%) | 2314.9 | 640.5 (−72.3%) |
| 16 | 55.90 | 22.17 (−60.3%) | 258.3 | 725.0 (+180.6%) | 7601.1 | 1121.0 (−85.3%) |
Key Technical Optimizations
Adaptive Scheduling Budget
To prevent low request counts from leaving max_num_batched_tokens unused, vLLM introduced an adaptive scheduled-token budget. This strategy ensures that single requests are not split across multiple forward calls when request counts are small, reducing TTFT by 55%–65% and increasing throughput by up to 41.5% for the 8K/1K workload.
Internal KDA Prefix Checkpoints
Previously, Mamba-style prefix cache splits required a second full-model pass for short suffixes. vLLM now supports checkpoint export inside a single prefill pass. For an 8K input, this allows one FlashKDA call to process all 8,000 tokens and export the checkpoint state at token 7,680 within the same recurrence, avoiding a second pass through attention, MoE, routing, and TP collectives. This reduced TTFT by 9%–25%.
Zero-Copy Mixed KDA Batches
Mixed batches containing both speculative and non-speculative tokens previously required multiple index_select and index_copy_ operations per layer. vLLM implemented contiguous zero-copy slices and direct output writes, which increased throughput by 5.2%–7.7% at concurrency 4 and 16.
Deferred MXFP4 Finalization
By fusing MXFP4 top-k finalization into the latent-tail kernel, vLLM removed one kernel launch and avoided the need to write and reread an intermediate tensor, reducing end-to-end latency by approximately 5%.
Advanced Memory and State Management
ReplaySSM for State Reconstruction
Speculative decoding typically requires writing KDA recurrent state at every draft position to allow for rollbacks. ReplaySSM optimizes this by buffering recent SSM inputs and reconstructing the accepted state only at the commit point. For Kimi K3 on Model Runner V2, this increased effective cache capacity by 10.97% under TP8 without impacting accuracy.
Prefill/Decode Disaggregation and Hybrid State Offload
vLLM now supports PD disaggregation and cache offload for Kimi K3, which requires transferring both MLA KV and KDA state. Because KDA state is sharded by head and dimension, and Mamba align block tables can be sparse and mutable, vLLM utilizes Mooncake to store boundary states selected by the scheduler and pin them until asynchronous writes complete on every rank.
Decode Context Parallelism (DCP)
Since Tensor Parallelism (TP) replicates MLA latent KV on every rank, it does not increase KV-cache capacity. Decode Context Parallelism (DCP) shards the KV-cache along the sequence dimension. For Kimi K3's fused MLA path, DCP uses Symmetric-memory A2A for output/LSE reduction and NVLS multicast for query gathering.
On a 120k-token workload (114k shared prefix, 6k suffix, 400 output tokens), KV-cache capacity increased from 1.93M to 19.75M tokens, while TPOT p50 dropped from 13.8 ms to 10.5 ms at concurrency 1.
Broader Implementation Efforts
The performance improvements were the result of a wider effort involving memory layout, sequence and pipeline parallelism, KDA prefill, and the tightening of small-batch GPU paths. This included the integration of DeepEPv2 with DeepGEMM MXFP4 and sequence-parallel GEMM paths, as tracked in GitHub issue #50587.