vLLM GLM 5.3 Optimizations: Hybrid HiSparse Offloading

vLLM has introduced Hybrid HiSparse offloading to optimize the serving of GLM 5.3, specifically targeting memory-constrained environments such as a single 8× H200 node. This optimization allows GLM 5.3 to run at its full 1 million context length and significantly increases concurrency for agentic workloads where long contexts grow over time.

Solving KV Cache Pressure in Agentic Workloads

Agentic workloads typically involve many concurrent requests with long, growing contexts. In standard GPU serving, the fixed GPU block pool eventually runs out of space for the KV cache, leading to two traditional solutions with significant drawbacks:

  • Preemption: Dropping a request's KV cache and re-prefilling it later, which forces the request to pay the full Time to First Token (TTFT) penalty again.
  • Offloading: Moving blocks to host memory. However, dense attention requires all tokens to be resident on the GPU, meaning concurrency remains limited by GPU memory.

Hybrid HiSparse addresses these limitations by exploiting the sparsity of the sparse-MLA KV cache. While the indexer selects only top-K tokens for attention, Hybrid HiSparse keeps KV cache on the GPU as long as capacity exists. When the system faces KV cache pressure, it offloads the "coldest" pages to the CPU, keeping only the selected top-K tokens in GPU-resident "hot buffers."

Technical Implementation of Hybrid HiSparse

Hybrid HiSparse manages KV residency through three distinct states based on system pressure:

1. Full Residency

All sparse-MLA KV remains on the GPU. Completed prefix pages are proactively copied to host memory to prepare for potential future eviction without requiring additional copies during the pressure phase.

2. Mixed Residency

When GPU memory is tight, the tail of the request remains on the GPU, while older pages are moved to CPU memory. The system uses a fused kernel to resolve top-K tokens: resident tokens are read in place, tokens in hot buffers are read with their LRU entry refreshed, and misses trigger a copy of a single row from pinned host memory into an LRU slot. This process remains CUDA-graph-capturable as no decode path decisions wait on the CPU.

3. No Residency

For new requests reusing a prefix that exists only in CPU memory, the system starts with placeholders and a hot page. Rows are loaded only as the indexer selects them, ensuring the system only pays the memory cost for tokens the model actually attends to.

Memory Management and Integration

Hot buffers are not separate allocations but are ordinary KV-cache blocks leased from vLLM's Hybrid Memory Allocator (HMA) pool. This allows blocks freed by one request to be repurposed as hot-buffer capacity for another. To maintain efficiency, hot buffers default to 2× top-K rows per request.

Performance Benchmarks on 8× H200

vLLM benchmarked GLM 5.3 using an OpenHands multi-turn agentic workload (13-turn conversations, 74,160-token first turn, and 753-token subsequent turns). The setup used TP8, MTP3, FP8 KV cache, and a 142K admission limit.

Comparing Hybrid HiSparse (with a 384 GiB HiSparse pool and 128 GiB offloading pool) against a standard offloading baseline (512 GiB pool), the results demonstrate that Hybrid HiSparse sustains higher concurrency and better interactivity-throughput trade-offs. Unlike standard offloading, which may force a request to wait for a slot to free up, Hybrid HiSparse allows requests to continue decoding at partial residency.

Integration with vLLM Ecosystem

Hybrid HiSparse is designed as a residency policy over the shared HMA pool and integrates with existing vLLM features:

  • Prefix Caching: Other cache groups continue to use standard prefix caching and offloading.
  • P/D Disaggregation: Imports from Prefill/Decode disaggregation can land host-side if the prefix does not fit in resident memory.
  • Speculative Decoding: Works via per-step replayable resolver plans that share the request's hot state.

Hybrid HiSparse is planned for wide availability in vLLM v0.30. It is currently implemented for NVIDIA GPUs.

Sources