vLLM GLM 5.3 Optimizations: Hybrid HiSparse Offloading

vLLM has introduced Hybrid HiSparse offloading to make the serving of GLM 5.3 faster and more cost-effective. This optimization allows GLM 5.3 to run at its full 1 million context length on a single 8x H200 node—a feat previously impossible on this hardware—while substantially increasing concurrency across various context lengths.

Solving KV Cache Memory Pressure

Agentic workloads typically involve many concurrent requests with long, growing contexts. Because GPU block pools are fixed, the KV cache eventually exhausts available memory, forcing a choice between two traditional methods:

  • Preemption: A request's KV cache is dropped and must be re-prefilled later, incurring a full Time to First Token (TTFT) penalty.
  • Offloading: KV blocks are moved to host memory, but dense attention requires all tokens to be resident on the GPU to run, limiting concurrency to the GPU's memory capacity.

Hybrid HiSparse addresses this by leveraging the sparse-MLA (Multi-head Latent Attention) KV cache of GLM 5.3. In sparse-MLA, an indexer selects only the top-K tokens for attention. HiSparse offloads all KV cache to the CPU except for these selected tokens.

Hybrid HiSparse improves this by keeping the KV cache on the GPU as long as capacity exists. It only triggers offloading when the system is under KV cache pressure. This approach minimizes CPU-GPU memory transfers, paying the cost only during periods of high concurrency.

Technical Implementation of Hybrid HiSparse

Hybrid HiSparse manages residency through a shared GPU block pool using vLLM's Hybrid Memory Allocator (HMA). It tracks residency per page, moving requests through three states based on memory pressure:

  1. Full residency: All sparse-MLA KV remains on the GPU, while completed prefix pages are proactively copied to host memory.
  2. Mixed residency: The request tail remains on the GPU, but older pages reside in CPU memory. The indexer's required rows are stored in "hot buffers" on the GPU. A fused kernel handles the top-K resolution: reading resident tokens in place, reading hot tokens and refreshing their LRU entry, or copying a single row from pinned host memory into an LRU slot upon a miss.
  3. No residency: For requests reusing a prefix that exists only in CPU memory, the system starts with placeholders and a hot page, loading rows only as the indexer selects them.

Key Architectural Details

  • Hot Buffers: These are not separate allocations but ordinary KV-cache blocks leased from the HMA pool. They default to 2x top-K rows per request to maintain high hit rates with minimal size.
  • Proactive Copying: To prepare for pressure, HiSparse queues copies of completed prefix pages to CPU memory while they are still served from the GPU. This allows the GPU slot to be released immediately when pressure arrives without requiring another copy.
  • Lightweight Execution: The hisparse-glm branch copies all sparse-MLA layers together in a single launch after the forward pass, ordered on the model's GPU stream to simplify synchronization.

Integration with vLLM Stack

Hybrid HiSparse acts as a residency policy over the shared HMA pool. It integrates with existing vLLM machinery without disrupting other components:

  • Prefix Caching: Other cache groups continue to use standard prefix caching and offloading.
  • Indexer KV: The indexer KV is handled independently by the standard OffloadingConnector using block-granular storage.
  • P/D Disaggregation: Imports from Prefill/Decode disaggregation can land on the host side if a prefix does not fit in resident memory.
  • Speculative Decoding: Works via per-step replayable resolver plans that share the request's hot state.

Performance Benchmarks

vLLM benchmarked GLM 5.3 on 8x H200 GPUs using an OpenHands multi-turn agentic workload (13-turn conversations, 74,160-token first turn, 753-token subsequent turns, and 220-token outputs). The configuration used MTP3, FP8 KV cache, and a 142K admission limit.

Comparing Hybrid HiSparse (with a 384 GiB HiSparse pool and 128 GiB offloading pool) against a standard offloading baseline (512 GiB offload pool), Hybrid HiSparse demonstrated a superior interactivity-throughput Pareto curve and higher mean concurrent running requests.

Availability and Configuration

Hybrid HiSparse is planned for wide availability in vLLM v0.30. It is currently implemented only for NVIDIA GPUs. For current reproduction, the hisparse-glm branch (commit e8ef1e07bd) is required.

To enable Hybrid HiSparse, the following configuration is used in the vllm serve command:

--attention-config '{"hisparse_config":{"host_pool_gib":384}}'
--kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":137438953472}}'

Sources