Kimi K3 in vLLM: Specialized CuTe DSL Decode on NVIDIA GB300 Hits 842 Tokens/s
vLLM has published a new decode path for Kimi K3 that replaces nearly all of the model's decode operations with custom CuTe DSL kernels tuned for NVIDIA GB300, reaching 842 tokens per second at batch 1 on 16 GB300 GPUs with DSpark speculative decoding — 1.73× faster than stock vLLM in the same setup. The post also introduces the process change behind the work: a proposed "fast-track model optimization RFC" (vllm-project/vllm issue #59665) that trades line-by-line human review for isolation, bounded scope, systematic correctness validation, and reproducible performance evidence.
The contribution targets NVIDIA Blackwell GPUs with 8 or 16 GPUs, decode batches from 1 to 64, and optional decode context parallelism for long contexts. The GPUs share one NVLink domain with multicast support. The Kimi K3 pull request is vllm-project/vllm #61867.
The Kimi K3 decode path: 13 kernels per layer instead of 23
The specialized decode path fuses and replaces almost all operations in the Kimi K3 decode step with customized CuTe DSL kernels. Measured per layer, stock vLLM uses 130 microseconds and 23 kernels, while the specialized decode path uses 69 microseconds and 13 kernels.
The optimization work was agent-assisted: coding agents were used to write complex GPU kernels, perform cross-operator fusions, overlap computation with communication, and tune execution across batch sizes and GPU configurations. The resulting path covers attention, MoE, and communication across 8 or 16 GPUs.
1. Tune kernel shapes for the workload
Different decode batch sizes require different kernel choices. For attention input projections, the Kimi K3 path uses specialized small-matrix kernels at batch sizes 1–2 and split-K kernels at larger batches, which divide the matrix multiplication across more GPU workers. These projections are tuned to launch well-sized kernels that fully utilize GPU resources and overlap memory reads, communication, and compute as much as possible.
KDA also uses an optimized CuTe DSL decode kernel, with scheduling tuned for the batch size and the number of attention heads on each GPU.
For long contexts, decode context parallelism (DCP) splits the KV cache across GPUs. Each GPU computes attention over its local portion, and a tuned fused kernel combines the partial results with v-proj and gating.
2. Overlap computation and communication
The Kimi K3 path fuses the attention output projection directly into its all-reduce and the following attention-residual computation. GPU kernels publish partial results into NVLink-accessible buffers, and receiving kernels combine them and continue the computation. Some small-batch configurations fuse the producer and consumer into a single kernel. Data exchange uses a fence-less approach with minimal overhead.
At larger batches, the implementation switches to a two-stage reduction: GPUs first combine assigned portions of the output, then distribute the results. This reduces data movement between GPUs.
3. Improve MoE scheduling and overlap
Although the experts already use tensor parallelism, the baseline computes the full router scores and latent input projection on every GPU. The Kimi K3 path splits these calculations across the TP ranks and combines each rank's share into one matrix multiplication. The GPUs exchange results over NVLink. This reduces repeated computation, weight reads, and kernel launches before expert execution begins.
The shared expert runs on a separate stream. Its work overlaps with routing or routed-expert computation, depending on the configuration. The schedule controls how many GPU resources each operation uses and is tuned to balance parallelism without contention. Implementation details live in experts_runtime.py.
4. Prefetch upcoming weights into cache
Small-batch inference is often memory-bound: limited by the speed at which weights can be loaded from memory. Some phases cannot fully use available memory bandwidth, such as communication. While communication runs, a dedicated kernel on a side stream prefetches the weights for the next operation, so that when communication completes those weights have a much higher cache hit rate.
Why agent-optimized inference code outpaces manual review
Agent-optimized inference code is outpacing vLLM's review and maintenance workflow. Manually understanding every line of every kernel, mapping out potential issues, and testing every supported configuration is very time-consuming; skipping that work risks shipping buggy software that could fail in production.
The proposed fast-track RFC addresses this by changing how performance work is organized: alongside agentic optimization, vLLM wants systems that automatically verify reliability and performance, and software structured to enforce isolation and risk minimization by design. The stated guiding principle is to decrease review burden while managing risk.
Our guiding principle is to decrease review burden while managing risk.
The approach rests on four principles.
Principle 1: Isolate changes
Model-specific optimizations stay within the model's code. This makes affected paths easier to identify and reduces the blast radius a reviewer must consider. Isolation must also hold at runtime: reviewers should be able to trace when the optimized path is selected, what state it changes, and how execution returns to the standard path.
Isolation deliberately excludes some optimizations. Input preparation, sampling, and KV cache management can cause overhead that isolation excludes. This is largely intentional: those components are shared among most vLLM deployments, and changing them to optimize for a specific workload risks breaking dozens of other stable vLLM configurations. Work on these components should involve core maintainers much more closely than the fast-track would permit.
Principle 2: Bound the scope, but preserve the user's features
An optimization does not need to improve every deployment to be useful. It can target a particular GPU, parallelism setting, or range of batch sizes. Clear boundaries make the work easier to tune and define a set of configurations to validate.
For Kimi K3, initial hardware validation covers GB300 with TP8 and TP16, decode batches up to 64, speculative decoding with various draft lengths, and decode context-parallel configurations within one NVLink domain.
These boundaries describe where the optimization applies, but they do not cut feature coverage. Production deployment needs fundamental features that make the endpoint useful. Features such as full-context serving, temperature sampling, and structured output cannot be "optimized away" or "opt-out"; they are first-class serving features and belong to the validation plan alongside kernel shapes and numerical checks.
Principle 3: Validate correctness
Kernel tests and model evaluations answer different questions. Kernel tests check numerical results across shapes and input values. Model evaluations check whether the complete execution path preserves the model's answers. Both are required.
Tests must actually exercise the changed code. A small-batch optimization needs evaluation at small batches. An attention kernel change needs long-context coverage. Tests that silently use the standard path provide little evidence about the optimization and are described as a common trap.
Kimi K3 development illustrates why this matters. An audit discovered that the compiler moved activation reads ahead of a synchronization wait; the fix restricted that load type to static weights. Such errors can survive simple functional checks and require targeted investigation.
Long-context and long-generation evaluations such as MRCR and AIME25 also surfaced sensitive numerical issues in attention and communication kernels that were not reproducible in standalone kernel tests using mock inputs and/or consistent launch timings.
Principle 4: Check performance broadly
A headline speedup is a starting point. Validation must show how a change behaves across the workloads it affects, which means sweeping batch sizes, context lengths, and supported parallelism configurations. MoE measurements should include real prompts because expert selection affects which weights are read and how much work can be reused. Pure decode results should be distinguished from workloads that mix prefill and decode.
Reliable validation takes GPU time. The related Bring-Your-Own-GPU evaluation RFC (vllm-project/vllm issue #61698) proposes standard evaluation images that contributors can run on their own hardware. Reproducible result bundles and submissions linked to GitHub PRs would make the evidence more accessible. Different evaluation tiers and shared community compute would expand coverage while managing GPU demand and reducing pressure on project-funded CI.
Performance results
Decode performance with DSpark
With DSpark, Kimi K3 reaches 842 tokens per second on 16 NVIDIA GB300 GPUs in the controlled AL=6 benchmark. That is 1.73× faster than stock vLLM with the same speculative decoding setup, which measured 488 tokens/s.
Setup: TP16, DSpark with seven draft tokens, synthetic acceptance length of 6, DL=7, ISL=1, OSL=1k, batch size 1. TPS measures aggregate decode throughput and excludes prefill.
Decode performance without speculation
The improvements span the tested batch range. TP16 retains a 1.62–1.71× gain at batches 16–64, while TP8 delivers 1.31–1.43× over the same range.
| Batch size | TP8 tokens/s | TP8 speedup | TP16 tokens/s | TP16 speedup |
|---|---|---|---|---|
| 1 | 193.3 | 1.58× | 256.4 | 1.88× |
| 2 | 317.8 | 1.50× | 436.7 | 1.92× |
| 4 | 540.6 | 1.55× | 702.5 | 1.81× |
| 8 | 820.1 | 1.45× | 1,157.9 | 1.79× |
| 16 | 1,274.8 | 1.43× | 1,758.3 | 1.71× |
| 32 | 1,702.6 | 1.33× | 2,550.9 | 1.64× |
| 64 | 2,375.9 | 1.31× | 3,606.1 | 1.62× |
These figures are without speculative decoding. Throughput is for the entire TP group.
Improvement on AgentX
The optimized Kimi K3 implementation delivers 1.8× higher P90 interactivity at concurrency 1 on AgentX: 403.2 versus 223.7 tokens/s/user. This compares the optimized run on 16 GB300 GPUs with the earlier vLLM result on 8 GB300 GPUs.
AgentX replays traffic patterns from real coding-agent sessions, including long contexts, multiple conversation turns, tool pauses, parallel subagents, and KV-cache reuse. It preserves these workload patterns while replacing private content with synthetic text, testing the serving system under realistic agent traffic.
What's next
DeepSeek-V4.1-Flash
DeepSeek-V4.1-Flash (vllm-project/vllm PR #61868) follows the Kimi K3 design patterns. Behind the same opt-in decode_backend: latency_specialized switch, the implementation supports up to 64 concurrent requests, seamless fallback for prefills and mixed batches, and speculative decoding. Instead of chasing a single hotspot, that executor cuts kernels and host work: it fuses all-reduce + mHC at every sublayer boundary, replaces the large-batch MoE and GEMMs with CuTe DSL kernels built for low latency, deduplicates and shards the indexer's work, and takes per-step work that stock vLLM runs eagerly on the CPU and replays it as a CUDA graph. DeepSeek AgentX results are reported on four GB300 GPUs.
Muse Spark Open
Within two days of Meta's Muse Spark Open release, vLLM's early results already exceeded 950 tokens per second per user, with seven draft tokens (K=7) and an acceptance length of six (AL=6). More optimizations are underway.
GLM5.3
vLLM is also releasing a similar mechanism to optimize decode latency for GLM5.3 under low-concurrency scenarios. Optimization details and performance results will follow in a separate post.
For each contribution, vLLM says it will continue to focus on concrete performance gains, production feature support, and reproducible correctness and performance checks.
Acknowledgements
vLLM thanked Rohan Varma (NVIDIA) and Alex Steiner (NVIDIA) for their contributions to this work, and Woosuk Kwon (Inferact), Michael Goin, and Lucas Wilkinson (Red Hat) for the model refactor and their feedback and support.