vLLM TurboQuant Study: Accuracy and Performance Analysis

vLLM has conducted a comprehensive study on TurboQuant, a KV-cache quantization method that compresses storage to 3-4 bits and dequantizes to BF16 for attention computation. The study concludes that while TurboQuant can increase KV-cache capacity, it often does so at the cost of throughput, latency, and accuracy, making FP8 the recommended default for most users.

Comparison of Quantization Schemes

The study compared four TurboQuant variants against unquantized BF16 and FP8 baselines across models ranging from 30B to 200B+ parameters (including Llama-3.3-70B-Instruct, Qwen3-30B-A3B, and MiniMax-M2.7).

• FP8 (--kv-cache-dtype fp8): Quantizes both storage and attention computation using hardware-native FP8 Tensor Core operations. It provides 2x KV-cache capacity with negligible accuracy loss and matches or exceeds BF16 performance.
• TurboQuant k8v4: Uses 8-bit keys and 4-bit values.
• TurboQuant 4bit-nc: Uses 4-bit keys and values with norm correction.
• TurboQuant k3v4-nc: Uses 3-bit keys and 4-bit values with norm correction.
• TurboQuant 3bit-nc: Uses 3-bit keys and values with norm correction.

Accuracy Impact

Accuracy degradation varies significantly by the aggressiveness of the quantization variant, particularly in long-context and reasoning tasks.

Long-context Retrieval

Using the openai/mrcr task, the study found that higher-bit variants (k8v4 and 4bit-nc) preserve retrieval performance well. However, aggressive variants (k3v4-nc and 3bit-nc) show meaningful degradation. On Qwen3-30B-A3B-Instruct-2507, these aggressive variants saw a ~30% relative degradation from BF16, with errors accumulating at sequence lengths between 128k and 256k.

Reasoning Performance

On decode-heavy reasoning benchmarks (AIME25, GPQA:Diamond, MATH500, and LiveCodeBench-v6), aggressive TurboQuant variants (k3v4-nc and 3bit-nc) caused drastic accuracy drops of up to 20 points on Qwen3-30B-A3B-Thinking-2507. Even on the 200B+ parameter MiniMax-M2.7 model, these variants showed significant degradation on AIME25 and LiveCodeBench-v6, proving that model size does not fully mitigate the accuracy loss of low-bit quantization.

Performance and Serving Metrics

TurboQuant introduces a dequantization overhead because it must convert low-bit storage back to BF16 before computing attention, unlike FP8 which computes in FP8.

Latency and Throughput

• FP8: Exhibits negligible or no latency overhead and matches BF16 throughput across all tested models.
• TurboQuant: All variants add measurable latency. On Llama-3.3-70B, overhead ranges from 10% to 68%, and this overhead increases with batch size. Throughput is strictly lower than BF16, ranging from 66% to 80% of BF16 throughput depending on the variant.

Serving Speed (TPOT and TTFT)

• Time Per Output Token (TPOT): FP8 tracks or outperforms BF16. TurboQuant variants add substantial per-token overhead that grows with load; at burst on Llama-70B, TQ variants are 1.5x to 2.5x slower than BF16.
• Time To First Token (TTFT): In memory-constrained scenarios (e.g., Llama-3.3-70B on 4xH100), BF16 TTFT can explode due to memory saturation. TurboQuant variants significantly reduce TTFT (staying under 3.5s vs 17s for BF16) by allowing more concurrent requests. However, FP8 provides the best result, with the lowest TTFT (~1.3s).

Final Recommendations

Based on the evaluation, vLLM provides the following guidance for selecting a KV-cache dtype:

• Use FP8 (--kv-cache-dtype fp8): This is the best default. It offers 2x capacity, no throughput cost, and negligible accuracy loss.
• Use TurboQuant 4bit-nc: Consider this for memory-constrained deployments where the need for higher capacity (up to 3.4x) and improved burst TTFT outweighs the cost of moderate accuracy loss and reduced throughput.
• Avoid TurboQuant k8v4: It provides only modest savings (2.4x) over FP8 without the performance benefits.
• Avoid TurboQuant k3v4-nc and 3bit-nc: These variants are unsuitable for production due to drastic accuracy drops and high performance degradation.
• Use BF16: Stick with the unquantized baseline if GPU memory is not a bottleneck.