vLLM Qwen3.5 Performance Optimization

vLLM has achieved a total throughput of over 25,000 tokens per second (TPS) per GPU when serving Qwen3.5 on GB200 NVL72 systems. This performance milestone was reached by optimizing the disaggregated serving path for Qwen3.5's hybrid attention architecture, specifically addressing the challenges of Gated Delta Network (GDN) computation and state transfer between prefill and decode workers.

Technical Optimizations for Qwen3.5

Qwen3.5 utilizes a hybrid architecture combining full-attention layers with Gated Delta Network (GDN) layers. To maximize throughput, vLLM implemented three primary technical enhancements:

1. Blackwell-Optimized GDN Prefill

vLLM integrated a new GDN prefill kernel from FlashInfer (PR #3001), replacing previous FLA/Triton implementations. On an 8×B200 system running Qwen3.5-397B-A17B-NVFP4, this integration resulted in:

  • GDN Kernel Performance: Up to 5.92× higher performance in microbenchmarks.
  • Prefill Throughput: 1.13× higher end-to-end prefill throughput for prefill-only workloads (ISL/OSL = 8192/1).
  • Latency: A 12% reduction in mean Time to First Token (TTFT) for the same workload.

Users can enable this path by setting the GDN backend to auto or explicitly using --gdn-prefill-backend flashinfer.

2. Hybrid Cache and GDN-State Transfer

Serving hybrid SSM-attention models in a disaggregated mode requires transferring both full-attention KV cache and Mamba-style SSM state. vLLM implemented several key changes to support this:

  • Physical Memory Mapping: PR #35758 mapped HMA logical blocks to physical memory regions, reducing transferred descriptors from 4,284 to 1,650 and improving throughput by approximately 7% in small-scale H100 setups.
  • Dual Descriptor Views: PR #36687 introduced dual descriptor views and homogeneous-TP support, allowing prefill and decode workers to transfer heterogeneous states over NIXL.
  • GDN Extension: PR #41869 specifically extended this disaggregated path to support GDN layers for Qwen3.5.

3. Race-Free Async Scheduling

Async scheduling was identified as a critical feature for crossing the 25K tok/s/GPU threshold. vLLM resolved two critical race conditions in KV block transfers (PR #48481 and PR #45357) that previously caused accuracy to collapse to zero when async scheduling was enabled.

Performance Benchmarks

Environment and Setup

Performance was measured using the following configuration:

  • Hardware: GB200 cluster connected via NVLink72.
  • Model: Qwen3.5-397B-A17B-NVFP4.
  • Workload: ISL/OSL = 8192/1024, random dataset with random_range_ratio=0.8.
  • Topology: A fixed decode side using one endpoint with DEP8 (Data Parallel + Expert Parallel across 8 GPUs) and prefill side configurations ranging from 4 to 8 endpoints (each using DEP2).

Results

Accuracy was verified using the GSM8K benchmark, with all five tested configurations maintaining an accuracy of 88%, matching the aggregated Qwen3.5 run.

Total TPS per GPU reached 25,000 tokens per second, with concurrency swept from 64 to 5,120. The limit of 5,120 concurrency was reached due to KV cache capacity on the decode side; increasing this would require additional GPUs on the decode endpoint.

Deployment Recipes and Best Practices

To achieve these results, vLLM recommends the following settings:

  • State Layout: VLLM_SSM_CONV_STATE_LAYOUT=DS is mandatory for conv-state transfer in disaggregated serving.
  • Scheduling: Use --async-scheduling to maximize throughput.
  • Cache Optimization: Use --mamba-ssm-cache-dtype bfloat16 to increase effective KV cache capacity on the decode endpoint.
  • Model Mode: Use --language-model-only for textual workloads to disable multimodal inputs and unlock the fused QK-norm + RoPE + gate path in attention layers.
  • Prefill Batching: Set --max-num-batched-tokens 16384 (2× ISL) on the prefill side to prevent prefill from becoming a bottleneck, which provided an approximate +8% increase in total TPS per GPU at high concurrencies.
  • Frontend Overhead: Set --stream-interval 100 to reduce frontend overhead at high concurrency, though this increases per-token latency.
  • Monitoring: Use --api-server-count 1 to enable default stats logging for identifying bottlenecks and monitoring KV cache utilization.

Sources

Related