vLLM v0.32.0 GLM-5.3 Serving: PCP, DCP, and Hybrid HiSparse on 16 H200s

Serving GLM-5.3 with long agentic contexts on 16 H200s requires two mechanisms that standard tensor and data parallelism cannot supply: Prefill Context Parallelism (PCP), which splits each prompt across GPUs by token, and Decode Context Parallelism (DCP), which shards the KV cache so a 1M-token context fits. Combined with Hybrid HiSparse decoding, this vLLM deployment (vLLM v0.32.0 or later) delivers 1.7-3.2x the per-user interactivity of a data-parallel deployment at the same throughput on the AgentX benchmark, and it is the only configuration the authors tested that can serve 1M-token requests at all.

This is Part 2 of vLLM's GLM-5.3 optimization series; Part 1 covered Hybrid HiSparse in an aggregated deployment on a single 8x H200 node. Part 2 covers prefill/decode disaggregation across two 8x H200 nodes.

What was announced

vLLM published a technical write-up describing how it serves GLM-5.3 long-context agentic workloads on 16 H200s split into one prefill node and one decode node. The post introduces three building blocks and shows how they compose:

Prefill Context Parallelism (PCP) splits each prompt across GPUs by token instead of by attention head, which suits GLM-5.3's sparse attention better than tensor parallelism.
Decode Context Parallelism (DCP) shards the prefill engine's KV cache by token, so a 1M-token context fits on the prefill node.
Hybrid HiSparse lets the decode engine keep KV in host memory, so a 1M-token request fits on the decode side.

On the SemiAnalysis AgentX benchmark, PCP8 with DCP8 prefill plus Hybrid HiSparse decode gave 1.7-3.2x the per-user speed of a data-parallel (DP8) deployment at equal throughput, and cut median time to first token (TTFT) from 1.5-5.2 s to 0.8-0.9 s.

Why long agentic contexts break standard parallelism on H200s

Agentic workloads have long contexts that grow with every turn, which pressures both GPU memory and prefill throughput. On H200s, GLM-5.3's weights consume most of each GPU's memory, leaving little room for the KV cache of long contexts. Because every turn adds more tool output and file contents to prefill, under load the prefill engines can become the bottleneck for the whole deployment. Prefill/decode disaggregation alone does not solve this.

Why DP8 and TP8 fall short

Two standard ways to run GLM-5.3 prefill on an 8x H200 node are data parallel (DP8) and tensor parallel (TP8), and both fail for different reasons:

DP8 assigns each prompt to a single GPU, so its attention cannot use the other seven. This limits time to first token for long prompts, and the prompt's KV cache must fit in that one GPU's memory.
TP8 partitions the attention heads across GPUs. Because MLA's latent KV is shared by all 64 heads, every GPU stores the full cache but reads it to serve only 8 heads. The indexer, which scores every earlier token to select the 2,048 tokens that sparse MLA attends to, is not partitioned at all, so its quadratic cost is repeated on every GPU.

Neither scheme spreads one long prompt's attention work across eight GPUs without also keeping the whole KV cache on each GPU that serves it. PCP addresses the query side; DCP addresses the KV side.

PCP: split the tokens, not the heads

Prefill Context Parallelism splits each prompt across the eight GPUs by token: every GPU computes the queries for one eighth of the prompt, using all 64 heads. The naive split would be unbalanced, because under causal attention a token attends to every token before it, so the GPU holding the end of the prompt would do far more work than the one holding the start. PCP instead cuts the prompt into 16 chunks and pairs them, so GPU g takes chunk g from the front and chunk 15 - g from the back, giving every GPU the same mix of cheap and expensive tokens. This is the load-balanced sharding from the paper "Context Parallelism for Scalable Million-Token Inference" (arXiv 2411.01783). Each GPU's new KV is then all-gathered so every query can attend to the full context.

The effect on arithmetic intensity is the headline number. Under PCP each GPU indexes only its own tokens, so each GPU does one eighth of the indexer work instead of replicating all of it. And each KV row read from memory now serves all 64 heads instead of 8, raising sparse MLA from about 26 to about 191 FLOPs per byte moved. The H200's BF16 ridge point is 206 FLOPs per byte, so PCP brings sparse MLA close to compute-bound.

Not replicating the indexer matters most for long prefills. The indexer's cost grows quadratically with prompt length, while sparse MLA's grows only linearly because each query attends to a fixed 2,048 tokens. Counting FLOPs for a prefill from scratch, the indexer overtakes sparse MLA under TP8 once the prompt reaches about 32K tokens; under PCP8, where no GPU replicates it, the crossover moves out to about 256K tokens.

PCP has two remaining costs. Each GPU must hold all the attention weights, while MoE experts stay split across the eight GPUs with EP8. And because new KV is all-gathered every layer, every GPU still ends up storing the full KV cache. That replicated cache, not compute, is what limits context length - and it is what DCP removes.

DCP: shard the KV cache too

Decode Context Parallelism splits the KV cache itself. A 1M-token GLM-5.3 request needs 53.4 GiB of FP8 KV, but each GPU has room for only about 15 GiB. DCP deals tokens round-robin across the eight GPUs, so each GPU stores every eighth token and the same request needs only 6.7 GiB (53.4 / 8) per GPU.

The naming is historical, not descriptive of the current deployment. DCP was built for decode, where each request has a single query token and the KV cache is the only thing worth splitting; PCP was built for prefill, where the prompt's many query tokens are the work. This deployment runs both inside the prefill engine, where PCP splits queries and DCP splits KV.

Because DCP shards the cache, most of the KV that each GPU's queries need now lives on other GPUs. In decode, DCP sends queries to the KV: each GPU attends over its own shard and the partial results are merged with their log-sum-exp. In this prefill engine the direction is reversed: every layer, each GPU all-gathers the shards into a temporary full-context copy, attends for its own queries, and frees the copy. Only the persistent cache is divided by eight.

The two layouts do not line up. A GPU stores every eighth token but computes the queries of chunks g and 15 - g, which is exactly why every layer needs the all-gather.

What the all-gather costs

The all-gather is not free: every layer it re-sends the whole cached context, so its cost grows with context length. Isolating the effect by running the same 2K-token prefill chunk on one PCP8 node with and without DCP8 at equal cached context, DCP8 adds about 6% per step up to 64K tokens of context, rising to 15% at 248K tokens. That is the price of an 8x larger KV pool. The authors state the overhead could be reduced by overlapping each layer's all-gather with the previous layer's compute, an optimization they are currently working on.

Beyond roughly 250K tokens, PCP8 alone cannot hold the KV cache, so DCP stops being an overhead and becomes what makes long prompts possible. Together, PCP8 with DCP8 gives the prefill engine both speed and room.

The deployment: prefill/decode disaggregation on 16 H200s

The deployment is a prefill/decode (P/D) disaggregated setup on two 8x H200 nodes. Both engines use an FP8 KV cache and MTP speculative decoding with three draft tokens (MTP3).

Prefill: one PCP8 x DCP8 x EP8 engine with 512 GiB of CPU KV offload, so later turns reuse their prefix.
Decode: eight DP ranks with EP8 and Hybrid HiSparse from Part 1 (48 GiB host pool plus 16 GiB offload per rank), so decode concurrency is bounded by host memory rather than HBM.

A prefix-aware llm-d router sends each request to the prefill engine, the NIXL connector moves its KV cache to a decode rank, and the decode rank streams the tokens back.

The comparison baseline uses the same hardware and the same total memory: PCP8 x DCP8 prefill is compared against DP8 prefill (eight DP ranks with EP8 and 64 GiB of CPU KV offload each, the same 512 GiB in total), and Hybrid HiSparse is compared against plain CPU KV offloading on the decode node, with 512 GiB of host memory either way. Every configuration runs the same 16 GPUs, host memory, and vLLM build.

Results on AgentX

The benchmark is SemiAnalysis AgentX: Claude Code traces replayed with prefix reuse, a fresh server per concurrency, one-hour runs, and MTP3 with a synthetic acceptance length fixed at 2.99 so every configuration gets the same speculative speedup. Throughput is measured as logical tokens per second per GPU including cached tokens - input plus output tokens, counting prefix-cached input, over all 16 GPUs. Interactivity is AgentX's P90 end-to-end-normalized output tokens per second per user.

Measurements were taken at two context limits: c8 through c64 at 142K, and c8 through c32 at 1M.

Up to c32 the deployment is mostly prefill-bound, so the prefiller makes the bigger difference. With HiSparse decode in both setups, PCP8 x DCP8 prefill gives 1.7-3.2x the interactivity of DP8 at the same throughput and cuts median TTFT from 1.5-5.2 s to 0.8-0.9 s. The decoder makes little difference at this load: at c8 and c16 the two PCP8 x DCP8 configurations are within a few percent of each other, and offload-only decode is slightly ahead at c8.

From c32 the deployment becomes more decode-bound, and HiSparse decode pulls ahead. At c64 it nearly doubles throughput, reaching 8,340 logical tokens per second per GPU at 29.6 tokens per second per user, while every other configuration drops to 1-2 tokens per second per user.

At 1M context, the deployment needs both mechanisms. A DP8 prefill rank cannot hold one 1M-token request, and neither can an offload-only decode rank. That is why the combined configuration is the only one tested that serves 1M-token contexts.

Making the features compose

The hard engineering work is making features like PCP, DCP, and Hybrid HiSparse compose, because each one changes how the KV cache is laid out, scheduled, or moved, and every combination has to stay correct and fast.

For prefill/decode disaggregation, the KV has to change layout between the two engines. The prefill engine shards each request's KV by token across eight GPUs, while each decode rank needs the whole KV - including the MTP layer's - in its own paged layout. The NIXL connector performs this reassembly between the two nodes, and prefix reuse from CPU offload has to keep working alongside it.

Every configuration in the post runs on Model Runner V2, vLLM's new modular core, which keeps model-specific state out of the common execution path and runs speculative decoding without CPU-GPU synchronization. Hybrid HiSparse is built on it.

How to reproduce and required flags

Everything in the post runs on vLLM v0.32.0 or later, using the vllm/vllm-openai:v0.32.0 image. The reported results were measured on vLLM main at commit 9e38b4111a, and the reproduction gist pins commit 2537629eb2.

The prefill engine's key flags are:

--prefill-context-parallel-size 8 \
--decode-context-parallel-size 8 \
--dcp-comm-backend ag_rs \
--enable-expert-parallel \
--attention-config '{"sparse_mla_force_mqa":true}'

One flag deserves explanation. Without sparse_mla_force_mqa, prefill batches whose prompts fit within the 2,048-token top-k take the dense MHA path. The flag keeps them on the sparse-MLA (MQA) kernel too, so every KV row serves all heads.

Why the numbers work out

Three footnote calculations underpin the post's claims about memory and compute. Sparse MLA reads 2,048 selected KV rows of 660 bytes each (656 bytes of FP8 KV in FlashMLA's fp8_ds_mla format plus a 4-byte index), about 1.35 MB per query. Each head adds 2,176 bytes for its BF16 query and output and performs 2,176 FLOPs per row (QK over 576 dimensions, PV over 512). So 8 heads do 35.7 MFLOPs over 1.37 MB, about 26 FLOPs per byte, while 64 heads do 285 MFLOPs over 1.49 MB, about 191 FLOPs per byte.

For the H200 SXM, the BF16 ridge point is 989 dense BF16 TFLOP/s over 4.8 TB/s of bandwidth, or 206 FLOPs per byte. The datasheet's 1,979 TFLOP/s figure assumes 2:4 sparsity.

The 53.4 GiB figure comes from per-token KV accounting: 78 layers x 656 bytes of MLA KV, 21 indexer layers x 132 bytes of indexer keys (128 FP8 values plus a 4-byte scale), and 788 bytes for the MTP layer, or 54,728 bytes in total. For N = 1,048,576 tokens, that is 54,728 MiB, approximately 53.4 GiB.

Who did the work

The work was done by Roberto L. Castro, Simon Veitner, Matthew Bonanni, and Lucas Wilkinson at Red Hat, and Nicolo Lucchesi at Mistral. Maroon Ayoub and Robert Shaw at Red Hat are credited with stress-testing the deployment, reporting errors, and fixing bugs.