vLLM FP8 KV-Cache and Attention Quantization Update

vLLM FP8 KV-Cache and Attention Quantization Update

vLLM has implemented critical fixes and optimizations for FP8 KV-cache and attention quantization, enabling significant reductions in memory usage and inter-token latency (ITL) for long-context LLM serving. By utilizing the --kv-cache-dtype fp8 flag, users can halve KV-cache storage and run attention computations in FP8 (e4m3), which is particularly effective for memory-bound decoding workloads.

Technical Improvements and Bug Fixes

Recent updates to vLLM address critical accuracy and performance regressions identified during stress tests on Hopper and Blackwell GPUs.

Two-Level Accumulation for Accuracy

To resolve a severe accuracy regression on Hopper GPUs—where a 128k needle-in-a-haystack task dropped from 91% (BF16) to 13% (FP8)—vLLM introduced a two-level accumulation strategy. This approach writes partially accumulated results into actual FP32 registers to mitigate precision loss in Tensor Cores when contraction dimensions reach 100k or more. This fix restored accuracy to 89%.

Hybrid Attention and Layer Skipping

For models with hybrid attention architectures (e.g., GPT-OSS), where some layers use sliding-window attention with small windows (e.g., 128 tokens), the overhead of FP8 quantization often outweighs the memory benefits. vLLM now includes the --kv-cache-dtype-skip-layers sliding_window flag, allowing these specific layers to remain in BF16 to improve decoding speed.

Kernel and Fusion Optimizations

  • Per-Head Scales: The Flash Attention 3 (FA3) kernel now supports an array of scales for FP8 quantization, with each scale corresponding to one KV-head.
  • Query Quantization Fusion: Query quantization has been moved to a torch implementation that torch.compile can fuse, eliminating fixed per-token overhead.
  • Tiling Configurations: Prefill tiling for head_dim = 64 and head_dim = 128 was tuned to reduce register spills caused by two-level accumulation.

Performance Benchmarking

Quantizing the KV-cache from BF16 to FP8 halves memory traffic per attention step, which directly reduces the ITL slope (the rate at which latency increases with input length).

Single-Request Latency

On H100 GPUs using Llama-3.1-8B, the ITL slope dropped from 4.37e-05 to 2.37e-05 ms/token (a 54% reduction), bringing the decode break-even point down to approximately 7k tokens. For gpt-oss-20b, using the skip-SW variant reduced the ITL slope to 71% of the BF16 baseline.

Throughput Under Load

In a heavy load scenario (concurrency 8, ~20k input tokens), FP8 provided the following gains:

  • Llama-3.1-8B: 14.9% higher output throughput and 13.0% faster total runtime.
  • gpt-oss-20b: 4.8% higher output throughput (using the skip-SW variant).

Architecture-Specific Results

  • Blackwell (B200): Using the FlashInfer backend, Llama-3.1-8B saw the ITL slope reduced to 54% of BF16, with a break-even point of roughly 4k tokens. Blackwell does not require two-level accumulation as the precision issues found in Hopper are absent.
  • Hopper (H100) with Large Head Dimensions: For models with head_dim = 256 (e.g., gemma-4-E2B), prefill performance (TTFT) is slower than BF16 (approximately 1.6x slower at long contexts) due to increased register pressure from two-level accumulation.

Accuracy and Validation

Evaluations were conducted using uncalibrated quantization scales (scale = 1.0) to establish a performance lower bound.

Reasoning and Long-Context Tasks

  • Reasoning: On Qwen3-30B-A3B-Thinking-2507 and Qwen3.5-27B, FP8 KV-cache and attention quantization resulted in negligible accuracy loss, typically between 0 and 2 points.
  • Long-Context (MRCR): Llama-3.3-70B-Instruct recovered 97-98% of the baseline AUC@128k. Qwen3.5-27B fully recovered the aggregated AUC@1M metric, demonstrating stability even at extreme context lengths.

When to Use Calibration

While uncalibrated FP8 is often sufficient, some models exhibit systematic degradation. For example, Kimi-K2.5 using the FlashMLA backend showed a consistent downward shift in accuracy across sequence lengths. In such cases, vLLM supports scale calibration via llm-compressor or per-attention-head quantization scales.

Summary of Usage Recommendations

Scenario Recommendation
Decode-heavy, memory-bound workloads Use --kv-cache-dtype fp8
Hybrid-attention models Use --kv-cache-dtype fp8 --kv-cache-dtype-skip-layers sliding_window
Short contexts (< 7k tokens) Stick with BF16 to avoid the small constant FP8 overhead
head_dim = 256 with prefill priority Stick with BF16 or disable two-level accumulation (requires accuracy validation)
Persistent accuracy loss (< 95%) Apply calibration using llm-compressor

Sources