Serving GLM-5.2 on NVIDIA B300 GPUs with vLLM

Serving GLM-5.2 on NVIDIA B300 GPUs with vLLM

vLLM has successfully deployed GLM-5.2-NVFP4 across 24 NVIDIA B300 GPUs (three 8-GPU servers) using a disaggregated Prefill/Decode (P/D) topology. By optimizing for Service Level Agreement (SLA) compliance rather than peak throughput, the team reduced mean Time Per Output Token (TPOT) from nearly 40 ms to 17 ms, meeting production targets of mean TTFT ≤ 2.5 s and mean TPOT ≤ 20 ms for context lengths between 16K and 256K tokens.

Prioritizing SLA Compliance over Peak Throughput

In colocated serving, prefill chunks can interleave with decode batches, causing long prompts to increase the inter-token latency of existing requests. vLLM utilizes P/D disaggregation to remove prefill work from the decode critical path, ensuring TPOT is determined solely by decode batch composition.

Production requirements for this deployment included:

  • Context Length: 16K–256K tokens.
  • Mean TTFT (Time to First Token): ≤ 2.5 s.
  • Mean TPOT (Time Per Output Token): ≤ 20 ms (approximately 50 tokens/s).
  • Throughput: Maximized only after the two latency constraints were met.

GLM-5.2 is a 744B-parameter MoE model with 40B active parameters, utilizing DSA sparse attention and MTP speculative decoding.

Optimizing Decode Performance

Initial configurations resulted in a mean TPOT of nearly 40 ms for 16K-token inputs. The following optimizations were implemented to bring this within SLA limits:

Speculative Padding for Mixed Batches

Profiling revealed that when a request transfers from a Prefill node to a Decode node, its first Decode step requires only one token, while existing requests using Multi-Token Prediction (MTP) require 1 + N tokens. This mismatch creates a mixed batch, forcing the system to fall back from the fast CUDA Graph path to expensive piecewise or eager execution.

To resolve this, vLLM implemented speculative padding on the Decode side, adding dummy tokens to the first step of new requests to match the 1 + N shape. This optimization (merged in PR #45237) reduced mean TPOT from 40 ms to 22 ms.

Model Runner V2 (MRV2)

Activating Model Runner V2 (VLLM_USE_V2_MODEL_RUNNER=1) provided an 11% reduction in TPOT. Key improvements included:

  • Warmup Kernels: The GLM-5.2 DSA indexer prefill-metadata kernel was added to startup warmup (PR #47285) to prevent cold-start latency spikes.
  • Local Argmax Reduction: Multi-GPU MTP now uses local argmax reduction (PR #46448), reducing TP communication volume from full-vocabulary logits to approximately 2 × TP size.
  • Dynamic Speculative Lengths: Full CUDA Graphs now support dynamic speculative lengths (PR #45953), reducing eager fallbacks.

Communication and Graph Configurations

  • All-to-All Backend: Replacing the default EP backend with the flashinfer_nvlink_two_sided backend reduced TPOT by 4%.
  • CUDA Graph Mode: The Decode instance uses FULL_DECODE_ONLY mode with --max-num-batched-tokens 1024, which provides full graph coverage while reducing startup compilation time.
  • MTP Configuration: The Decode side uses num_speculative_tokens=3 to amortize execution costs, while the Prefill side uses num_speculative_tokens=1 to prioritize fast KV cache handoff.

Prefill Parallelism and Capacity Trade-offs

When evaluating parallelism strategies for Prefill, the team compared TGS (throughput per GPU). While TP1 DP4 EP was not the absolute most efficient per-GPU configuration (TP1 DP2 EP was 8% more efficient), TP1 DP4 EP was selected to ensure sufficient KV-cache capacity for GLM-5.2's 1M-token context capability.

Stabilizing MTP Acceptance Rates

To ensure speculative decoding remained effective under high concurrency, vLLM implemented IndexerCache (PR #44420). This mechanism reuses Top-K sparse indices produced by the DSA indexer, preventing the system from rerunning the indexer for every MTP draft step.

Additional stability fixes included:

  • Indexer Initialization: Improved normalization loops and initialization when skipping Top-K layers (PR #45895).
  • Shared Index Buffer: Optimized the layout for batched requests to retain only indices for the final query token (PR #47238).
  • Post-Final-Norm Hidden State: Ensured MTP loop reuse of the post-final-norm hidden state (PR #47448).

Accuracy validation on AIME 2025 (86.67), GPQA (92.89), and LongBench V2 (64.01) confirmed that these optimizations did not degrade output quality.

Production Observability and Stability

Monitoring for disaggregated P/D deployments requires tracking metrics across two resource pools. Key metrics include per-pool TTFT/TPOT percentiles, MTP acceptance rates, and KV-transfer latency.

During long-term stability tests, the team discovered a host-memory leak where vLLM process RSS grew linearly over tens of hours. The root cause was an inconsistent gating mechanism in SingleTypeKVCacheManager.new_block_ids (PR #35219), which recorded block allocations for non-Mamba models but never drained them. This was fixed by unconditionally draining take_new_block_ids() on every scheduling step.

Deployment Recipe

Hardware and Topology

  • Hardware: 3 × 8 B300 GPUs (24 total).
  • Model: GLM-5.2-NVFP4.
  • Topology: 4 Prefill nodes (TP1 DP4 EP, 16 GPUs) and 1 Decode node (TP1 DP8 EP, 8 GPUs).
  • KV Transfer: NIXL.

Configuration Commands

Prefill Node:

export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve /mnt/model/glm/GLM-5.2-NVFP4 --trust-remote-code --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' --chat-template-content-format=string -ep -tp 1 -dp 4 --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 --gpu-memory-utilization 0.92 --enable-prompt-tokens-details --speculative-config='{"method":"mtp","num_speculative_tokens":1}' --shutdown-timeout 300 --fingerprint-mode=none

Decode Node:

export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve /mnt/model/glm/GLM-5.2-NVFP4 --trust-remote-code --chat-template-content-format=string --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' --max-num-batched-tokens 1024 -ep -tp 1 -dp 8 --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 --gpu-memory-utilization 0.90 --enable-prompt-tokens-details --all2all-backend=flashinfer_nvlink_two_sided --speculative-config='{"method":"mtp","num_speculative_tokens":3}' --shutdown-timeout 300 --fingerprint-mode=none

Sources