vLLM Kimi K3 Support Preview
vLLM Kimi K3 Support Preview
vLLM is preparing day-0 open-source serving support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter model. This integration enables the open-source community to deploy Kimi K3 immediately upon the release of its model weights, scheduled for July 27, 2026.
Kimi K3 Architecture and Serving Implications
Kimi K3 introduces several architectural shifts that change the requirements for inference engines. The model features a 1-million-token context window and a highly sparse Mixture-of-Experts (MoE) architecture with 896 routed experts (16 active per token) plus shared experts.
Key technical specifications and their serving impacts include:
| Property | Kimi K3 Configuration | Serving Implication |
|---|---|---|
| Model Scale | 2.8T parameters | Requires large-scale expert parallelism and high-bandwidth accelerator domains |
| Context Length | 1M tokens | Prioritizes cache capacity, prefix reuse, and prefill/decode disaggregation |
| Attention | Hybrid KDA and full attention | Requires simultaneous management of recurrent state caches and paged KV caches |
| Depth | Attention Residual (AttnRes) | Necessitates dedicated kernels for cross-layer representation reads/writes |
| Quantization | MXFP4 weights | Requires an efficient FP4 MoE path supporting Kimi K3’s SiTU activation |
| Multimodality | Native vision | Requires multimodal preprocessing and vision parallelism strategies |
Solving Prefix Caching for KDA
Kimi Delta Attention (KDA) is recurrent, meaning it advances a matrix-like recurrent state and a short convolution state rather than retaining per-token KV pairs. This creates a challenge for conventional prefix caching, as the engine must possess the KDA state at the exact prefix boundary to avoid replaying earlier states.
To resolve this, vLLM has implemented a design that decouples three previously linked concepts:
- Physical block size: The allocation of KDA state and full-attention KV on the GPU.
- Scheduler alignment: The point where execution stops to ensure cache group consistency.
- Prefix-match unit: The fine-grained token interval used for hashing and matching shared prefixes.
This separation allows vLLM to register a valid KDA state at a fine-grained boundary within a larger physical state block. When a subsequent request matches a partial block, the cached state is copied to a private destination (copy-on-write), preserving the shared prefix while allowing the new request to proceed.
Performance Optimizations and Kernel Work
Because Kimi K3 changes the "hot path" of inference, vLLM has implemented several hardware-specific and architectural optimizations:
KDA Prefill and Decode
Prefill integrates FlashKDA and Flash Linear Attention (FLA), fusing input projections and causal convolution. For decoding, vLLM uses a fused NVIDIA kernel that combines the short convolution, KDA state update, output gate, and normalization into a single operation to reduce the time-per-output-token (TPOT) penalty.
Attention Residuals (AttnRes)
To handle cross-layer representation reads without incurring excessive memory traffic, vLLM utilizes Triton and NVIDIA kernels that fuse residual update, AttnRes mixing, and output RMSNorm.
MLA Module and PD Disaggregation
Kimi K3 uses MLA attention every four layers. vLLM has replaced torch.compile custom-fusion with a manually fused MLA module. This module features separate prefill and decode paths optimized for Prefill-Decode (PD) disaggregated deployment and includes multi-stream support for the gate projection in the decode path.
MXFP4 MoE and Hardware Support
Kimi K3's release configuration uses MXFP4 weights and SiTU activation. vLLM now maps SiTU parameters into the optimized FP4 expert path via MXFP4 TRTLLM-Gen and DeepGEMM.
Hardware support includes:
- NVIDIA: Final tuning of specific kernels and MXFP4 MoE collaboration.
- AMD: Initial support via FlyDSL's MLIR Python kernel stack, featuring hardware-tuned A16W4/A8W4 quantized fused operators and SiTU activation.
Day-0 Release Package
The open-source release will include the vLLM model, parser, cache, and kernel integration, along with initial Docker images and validated launch recipes for NVIDIA configurations. It will also provide an initial AMD path using the FlyDSL MoE kernel, multimodal examples, tool-use, reasoning, and structured-output examples, and initial performance results.