vLLM DeepSeek V4 Support: Efficient Long-context Attention

vLLM DeepSeek V4 Support: Efficient Long-context Attention

vLLM has announced support for the DeepSeek V4 model family, including DeepSeek-V4-Pro (1.6T parameters) and DeepSeek-V4-Flash (285B parameters). These models utilize a specialized long-context attention mechanism designed to scale to one million tokens while drastically reducing memory overhead and computation costs.

DeepSeek V4 Attention Architecture

DeepSeek V4 addresses the linear growth of KV cache memory and the high computational cost of long-context attention through four primary architectural innovations:

  • Shared Key and Value Vectors: By sharing key and value vectors, the model achieves 2x memory savings. To maintain correctness, an inverse RoPE operation is applied to the attention output to ensure translation invariance.
  • KV Cache Compression: The model compresses the KV cache across multiple tokens to achieve 4x to 128x memory savings via two methods:
    • c4a: Compresses the KV cache by approximately 1/4; one compressed token represents a weighted sum of 8 uncompressed tokens with a stride of 4.
    • c128a: Compresses the KV cache by approximately 1/128; one compressed token represents a weighted sum of 128 uncompressed tokens with a stride of 128.
  • DeepSeek Sparse Attention (DSA): To keep computation bounded even after compression (e.g., 250k tokens for a 1M sequence using c4a), DSA is used to attend only to the top-k compressed tokens.
  • Short Sliding Window: A sliding window of size 128 is used for local information on uncompressed tokens, allowing query tokens to access local context before reaching the compression boundary.

These optimizations result in significant memory efficiency. A 1M context sequence in DeepSeek V4 requires only 9.62 GiB of KV cache per sequence (using bf16), which is approximately 8.7x smaller than the 83.9 GiB required for a 61-layer DeepSeek V3.2-style stack. Using fp4 for the indexer cache and fp8 for the attention cache further reduces this size by roughly 2x.

vLLM Implementation and Optimizations

Implementing DeepSeek V4 in vLLM required solving complex systems challenges related to heterogeneous attention types and memory management.

KV Cache Memory Management

vLLM employs three strategies to keep the KV cache packed and efficient:

  1. Single Logical Block Size: vLLM fixes the logical block at 256 native token positions for all compressed layers. This allows the allocator to use a consistent unit for slot mapping and prefix-hit detection, regardless of whether a layer is c4a (64 compressed entries per block) or c128a (2 compressed entries per block).
  2. Compressor State as Sliding Window: To avoid complex side buffers for rolling residuals (8-token for C4, 128-token for C128), vLLM treats compressor state as sliding-window KV. This allows the system to reuse existing abstractions for prefix caching and disaggregated prefill.
  3. Unified Page Sizes: By carefully choosing block sizes and compression ratios, vLLM collapses the five-way cache stack into three page-size buckets. This eliminates cross-pool fragmentation and removes the need for runtime repartitioning.

GPU Compute Saturation

To maximize GPU utilization and reduce HBM round-trips, vLLM implemented several kernel fusions and multi-stream partitioning:

  • Kernel Fusions:
    • Compressor + RMSNorm + RoPE + cache insertion: Fuses elementwise stages into one kernel, providing a 1.4-3x speedup.
    • Inverse RoPE + fp8 quant: Fuses these operations to avoid HBM round-trips, resulting in a 2-3x speedup.
    • Fused Q norm + KV RoPE + K insert: Horizontally fuses query and uncompressed SWA key work into a single kernel, delivering a 10-20x speedup.
  • Multi-stream Partitioning: vLLM overlaps independent operations across CUDA streams. For c4a layers, the indexer pipeline runs on a separate stream in parallel with main KV compression and SWA token insertion, reducing end-to-end latency by 5-6% at low batch sizes.

Deployment and Hardware Support

DeepSeek V4 is supported on NVIDIA Hopper and Blackwell architectures. vLLM provides optimized Docker configurations for single-node deployments:

  • DeepSeek-V4-Pro: Optimized for 8xB200 or 8xB300 GPUs.
  • DeepSeek-V4-Flash: Optimized for 4xB200 or 4xB300 GPUs.

Both configurations utilize fp8 KV cache, a block size of 256, and the deepseek_v4 tokenizer and reasoning parsers. Further optimizations, including the DeepGEMM MegaMoE kernel and paged prefill kernels, are currently in development.

Sources