Hugging Face KV Cache Quantization

Hugging Face has introduced KV Cache Quantization in the Transformers library, enabling large language models (LLMs) to generate longer sequences of text by reducing the memory footprint of the key-value (KV) cache. This feature allows users to expand context length on consumer GPUs by trading off a small amount of generation speed for significant memory efficiency.

The Role of KV Cache in Autoregressive Generation

Key-value (KV) cache is essential for optimizing autoregressive models, which predict text token by token. To predict a new token, the model requires information from all previous tokens. Without a cache, the model would need to recompute matrix multiplications for every previous token at every step. The KV cache acts as a memory bank, storing key-value pairs from self-attention layers for previously processed tokens, allowing the model to retrieve them instead of recomputing them, which significantly accelerates text generation.

However, the KV cache becomes a memory bottleneck as context length or batch size increases. For a 7B Llama-2 model with a context length of 10,000 tokens, the KV cache requires approximately 5GB of memory in float16 precision, which is nearly one-third of the memory needed for the model parameters themselves.

Technical Implementation of KV Cache Quantization

Inspired by the KIVI paper, Hugging Face's implementation utilizes affine quantization to compress the KV cache into lower precision formats.

Quantization Method

The implementation performs per-token quantization for both keys and values. To mitigate the potential slowdown caused by quantizing and de-quantizing at every generation step, Hugging Face employs a fixed-size residual cache. This residual cache stores the most recent keys and values in their original precision; once the cache reaches maximum capacity, the stored values are quantized and the cache is cleared. A residual length of 128 is used as the baseline to preserve accuracy.

Supported Backends and Precision

The feature currently supports the following backends and precisions:

  • Quanto: Supports int2 and int4 precisions.
  • HQQ: Supports int2, int4, and int8 precisions.

Performance and Quality Trade-offs

Quantizing the KV cache involves a trade-off between memory savings, generation speed, and model quality.

Model Quality and Accuracy

Tests using the Llama2-7b-chat model on the PG-19 dataset show that int4 cache precision performs almost identically to fp16 precision. However, quality degrades when using int2. On the LongBench benchmark, int4 precision via the Quanto backend is comparable to or slightly outperforms fp16 across several datasets, including TREC, SAMSum, and TriviaQA.

Memory Efficiency vs. Latency

Quantizing the cache to int4 provides approximately 2.5x memory savings. While this reduces memory pressure, it can lead to a decrease in generation speed, particularly as batch sizes increase. Furthermore, combining KV cache quantization with weight quantization can lead to a threefold decrease in generation speed.

Context Length Capacity

When combined with Flash Attention on an 80GB A100 GPU, KV cache quantization allows the model to support up to 128k tokens, compared to a maximum of 40k tokens using half-precision cache.

Integration with Transformers

KV cache quantization is device-agnostic and works on CPU, GPU, and MPS (Apple Silicon). To implement it in 🤗 Transformers, users must install the quanto library and specify the cache_implementation="quantized" argument along with a cache_config dictionary during the generate call.

# Example usage
out = model.generate(
    **inputs, 
    do_sample=False, 
    max_new_tokens=20, 
    cache_implementation="quantized", 
    cache_config={"backend": "quanto", "nbits": 4}
)

Sources