vLLM Decode Context Parallelism for Long Context Workloads

TL;DR

vLLM has implemented Decode Context Parallelism (DCP), a technique that shards the KV cache across GPUs along the sequence dimension rather than just by attention head. This prevents KV cache duplication in Grouped-Query Attention (GQA) and Multi-head Latent Attention (MLA) models, allowing for significantly higher request concurrency and throughput during long-context inference.

The Memory Wall in Long-Context Inference

Standard Tensor Parallelism (TP) partitions the KV cache by attention head. This approach creates a memory bottleneck when the number of GPUs exceeds the number of KV heads, as the system must duplicate the KV cache across multiple GPUs.

  • Grouped-Query Attention (GQA): TP can only split the KV cache down to one head per GPU. Once TP exceeds the number of KV heads, the cache is duplicated.
  • Multi-head Latent Attention (MLA): MLA compresses KV into a single low-rank latent vector shared across all query heads. Because it effectively has only one KV head, the latent KV cache is replicated in full across every TP rank, severely limiting available GPU memory for concurrent requests.

This duplication consumes critical GPU memory, capping the number of concurrent requests and increasing the cost per token.

Decode Context Parallelism (DCP) Explained

Decode Context Parallelism solves the memory bottleneck by splitting the KV cache across GPUs by the sequence (context) dimension. Instead of each GPU holding a full copy of a specific head's cache for the entire sequence, each GPU is responsible for a chunk of token positions for the same sequence.

For example, in a 200K-token request, four GPUs using DCP might split the workload such that GPU 0 holds tokens 0–50K, GPU 1 holds 50K–100K, and so on.

The DCP Execution Process

DCP follows a specific communication rhythm to maintain correctness during the decode phase:

  1. AllGather Q: Since attention requires the full query vector, an all-gather operation assembles a complete copy of the query on every GPU. (For MLA models, this can be optimized via VLLM_DCP_Q_REPLICATE=1 to replicate query projections at load time).
  2. Compute: Each GPU performs attention between the gathered query and its local slice of the KV cache.
  3. AllGather + ReduceScatter: Partial results and Log-Sum-Exp (LSE) values are shared via AllGather. LSE values are used to reweight and merge partials using the online-softmax trick, and ReduceScatter sums them while returning only the relevant head-slice to each GPU.

Performance Benchmarks

vLLM tested DCP against a baseline TP deployment using Kimi K2.6 in NVFP4 on an 8×B200 node, utilizing an agentic long-context trace with a median input of ~67k tokens.

Throughput and Concurrency

DCP sustains significantly higher concurrency and throughput compared to baseline TP:

  • Baseline TP: Reaches 100% memory usage at a concurrency of 64, with throughput plateauing near 1,863 tok/s/GPU.
  • DCP: Scales up to a concurrency of 512, reaching 6,091 tok/s/GPU while utilizing only 82% of KV memory.

Stability Across Sequence Lengths

Performance remains stable even as context length increases. In the 200k+ token range, DCP maintains a high throughput-interactivity frontier, whereas the replicated-KV baseline often runs out of memory and fails to scale.

Implementation and Usage

DCP is enabled via the --decode-context-parallel-size argument.

Multi-head Latent Attention (MLA) Backend

Used in models like DeepSeek-V2, V3, R1, and Kimi K2.6. Because MLA has effectively one KV head, the sequence can be split up to the full TP degree.

  • Constraint: tensor_parallel_size >= decode_context_parallel_size and tensor_parallel_size % decode_context_parallel_size == 0.

Grouped-Query Attention (GQA) Backend

Used in models like Qwen3-235B and Llama-family models. TP first splits by KV heads; DCP then shards the remaining redundant copies.

  • Constraint: (tensor_parallel_size // num_key_value_heads) >= decode_context_parallel_size and (tensor_parallel_size // num_key_value_heads) % decode_context_parallel_size == 0.

Future Roadmap

vLLM plans to extend DCP through the following developments:

  • Finer-grained parallelism: More precise control over TP and DCP sizes to reduce over-provisioning.
  • Communication Kernels: Developing better all-to-all (A2A) kernels for single and multi-node settings to improve compute overlap.
  • Speculative Decoding: Integrating DCP with MTP and speculative decoding to maintain latency benefits.
  • Disaggregated Serving: Hardening support for prefill/decode (P/D) disaggregation.
  • Expanded Model Support: Extending support to GLM-5.2, Kimi K3, and developing Prefill Context Parallelism (PCP).

Sources

Related

  • Dispatch
    vLLM DeepSeek V4 Support: Efficient Long-context Attention

    vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.

  • Project
    lucienhuangfu/eLLM

    eLLM is an LLM inference framework for CPU servers that optimizes long-context performance to compete with or outperform multi-GPU systems by leveraging large CPU memory and cache.

  • Dispatch
    Optimizing LLM Performance: Solving Long Prompt Blocking and Decode Slowdowns

    Hugging Face explores how long prompts block request queues and slow down token generation, proposing request-parallel prefills and disaggregated prefill as solutions to reduce latency.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Dispatch
    vLLM AMD ROCm Attention Backends Optimization

    vLLM introduces optimized attention backends for AMD ROCm, delivering up to 4.4x higher throughput for MHA and 1.5x for MLA models on Instinct MI300X, MI325X, and MI355X GPUs.