0xSero/turboquant

What it solves

TurboQuant addresses the high memory overhead of the Key-Value (KV) cache during LLM inference, which often limits the maximum context length and the number of concurrent requests a GPU can handle. It provides a way to compress the KV cache to free up VRAM without significantly sacrificing model quality.

How it works

TurboQuant uses a combination of techniques to compress KV cache entries:

• Random Orthogonal Rotation: Spreads information across dimensions to make quantization more efficient.
• Lloyd-Max Scalar Quantization: Applies optimal scalar quantization on rotated values.
• QJL Projection: Handles residual sign bits for each dimension.
• Group Quantization: Compresses values using per-group scales and zeros (supporting 2-bit or 4-bit).
• Bit-packing: Packs multiple values into a single byte to maximize storage efficiency.

It is integrated with vLLM and includes fused Triton kernels for decode attention, ensuring that the estimated inner product remains unbiased.

Who it’s for

Developers and researchers working with Large Language Models (LLMs) who need to extend context windows or increase throughput on hardware with limited VRAM (e.g., RTX 3090/5090 GPUs).

Highlights

• Significant VRAM Savings: Can achieve up to 4.4x compression on pure dense transformers, freeing up substantial GPU memory (e.g., 30 GB on a 4-GPU setup for Qwen3.5-27B).
• Context Extension: Increases maximum token capacity (e.g., from ~457k to ~914k tokens).
• Near-Lossless Key Compression: 3-bit and 4-bit key compression maintain near-perfect cosine similarity.
• vLLM Integration: Includes a monkey-patch adapter for vLLM 0.18.0.
• Theoretical Validation: Includes tests validating the paper's MSE distortion bounds and unbiasedness claims.