vLLM Kimi K3 Support

vLLM Kimi K3 Support

vLLM has announced efficient day-0 support for Kimi K3, a 2.8-trillion-parameter open-weight multimodal Mixture-of-Experts (MoE) model. This integration enables high-performance serving of Kimi K3's unique hybrid architecture, achieving up to 370 tokens per second per user when utilizing DSpark speculative decoding on 16 NVIDIA GB300 NVL72 GPUs.

Kimi K3 Architectural Overview

Kimi K3 is a 2.8T parameter model that activates 16 of 896 experts per token. It is designed for long-context efficiency, supporting a window of up to 1 million tokens through several key innovations:

  • Kimi Delta Attention (KDA): A hybrid stack that interleaves linear-attention layers (which maintain a fixed-size recurrent state) with periodic full-attention layers to preserve global recall.
  • Attention Residuals (AttnRes): A mechanism that replaces standard residual accumulation with depth-wise attention, using learned pseudo-queries to weight residual states from preceding blocks.
  • Stable LatentMoE: A design that projects activations into a narrower latent dimension for routed-expert computation, utilizing Quantile Balancing to allocate experts based on router-score quantiles.
  • Native MXFP4: The model uses native 4-bit weights on the MoE path to reduce bandwidth requirements.

vLLM Serving Optimizations for Kimi K3

To handle Kimi K3's departure from standard Transformer architectures, vLLM implemented several specialized serving mechanisms:

Hybrid KV-Cache Management

vLLM uses a single hybrid KV-cache manager to handle both paged KV blocks for full-attention layers and compact recurrent-state blocks for KDA layers. To enable prefix caching across this hybrid design, vLLM decouples physical KDA state blocks from fine-grained prefix matching, allowing shared prompts to reuse both KDA state and paged KV.

Fused Kernels and Metadata Optimization

  • Attention Residuals: vLLM uses fused Triton and CUDA kernels to compute depth-wise attention logits, softmax, and hidden-state aggregation in one operation, reducing memory traffic.
  • KDA Decode: A specialized CUDA kernel fuses causal convolution, recurrent updates, and RMSNorm into a single launch to avoid intermediate tensors and repeated state traffic.
  • KDA Prefill: vLLM integrated FlashKDA and subsequent community optimizations (Flash-Flash-KDA) to accelerate the prefill phase.
  • Metadata Builder: A dedicated Kimi K3 KDA metadata builder replaces eager PyTorch operations with fused Triton kernels, reducing metadata-preparation latency by 96% (from 870°s to 34°s) at batch size 1.

Low-Latency Compute

vLLM employs skinnyGEMM for low-batch-size settings, bypassing shared-memory data staging to reduce end-to-end latency by approximately 10%. Additionally, a LatentMoE tail-fusion optimization reduces redundant compute in the linear projection phase, yielding a 7°8% end-to-end speedup.

Production Capabilities and Performance

Speculative Decoding with DSpark

vLLM supports DSpark, a block-diffusion speculative decoding algorithm. Using a DSpark speculator trained with vLLM and TorchSpec, vLLM achieves a 3.14× speedup for single-user requests, increasing throughput from 118 tok/s to 370 tok/s. Acceptance rates range from 4.73 tokens per step for low-entropy tasks (coding) to 2.61 tokens per step for high-entropy tasks (creative writing).

Prefill/Decode Disaggregation

For high-throughput environments, vLLM supports prefill/decode (PD) disaggregation. The NIXL connector manages the transfer of both token-level MLA cache and request-level KDA state between separate prefill and decode replicas.

Agentic Cache Retention Policies

To manage the memory footprint of KDA states in long-context agentic workloads, vLLM implements two retention policies:

  • Interval-based retention: Checkpoints KDA state at fixed intervals (e.g., every 32K tokens) and automatically at prompt boundaries.
  • Marconi-style selective retention: A demand-driven approach that caches KDA state only upon the second hit of a prefix, ensuring only frequently shared prefixes consume cache capacity.

Benchmarks and Hardware Requirements

Performance Metrics

On GB300 NVL72 GPUs, vLLM achieves the following single-user decode throughput (batch size 1):

Configuration Standard Decode DSpark Speculative Decode
TP8 111 tok/s 331 tok/s
TP16 118 tok/s 370 tok/s

Accuracy

Kimi K3 on vLLM scores 0.976 on GSM8K, 0.939 on GPQA-Diamond, 0.889 on OCRBench, and 0.818 on MMMU Pro Vision at maximum reasoning effort.

Hardware Requirements

Serving Kimi K3 requires a minimum of 8× B300 (or GB300 NVL72) GPUs or 16× B200 GPUs. Support is available for both NVIDIA (Hopper and Blackwell) and AMD (MI355X) hardware.

Deployment Guide Summary

To run Kimi K3, vLLM recommends the following configurations:

  • Prefix Caching: Must be explicitly enabled via --enable-prefix-caching.
  • MoE Backend: Use deep_gemm_mega_moe for disaggregated/expert-parallel (DEP) environments and flashinfer_trtllm for TP > 1.
  • All-to-all Backend: Use flashinfer_nvlink_one_sided for NVLink and deepep_v2 for RDMA.
  • Rust Frontend: Enable via VLLM_USE_RUST_FRONTEND=1.

Sources