NVIDIA/kvpress

LLM KV cache compression made easy

What it solves

Deploying long-context Large Language Models (LLMs) is expensive because the key-value (KV) cache grows linearly with the number of tokens. For instance, processing 1 million tokens with Llama 3.1-70B in float16 can require up to 330GB of memory. KVPress reduces these memory requirements and increases decoding speed by compressing the KV cache.

How it works

KVPress implements a variety of "presses"—compression methods that prune or merge KV pairs during the prefilling phase or periodically during the decoding phase. It integrates with the Hugging Face transformers library, providing a custom KVPressTextGenerationPipeline to handle tokenization and chat templates.

Compression is achieved through several strategies:

  • Scorer-based pruning: Using importance scores (e.g., random, inverse norm, or attention weights) to evict the least important KV pairs.
  • Structural compression: Maintaining specific cache patterns, such as pyramid-like sizes across layers or keeping only initial and recent tokens (StreamingLLM).
  • Dimensionality reduction: Compressing the dimensions of keys based on channel attention scores.
  • Advanced techniques: Using CUR decomposition, context reconstruction (KVzip), or merging evicted tokens into surviving neighbors.

Who it’s for

Researchers and developers working on long-context LLM efficiency who need a standardized framework to implement, benchmark, and deploy KV cache compression methods.

Highlights

  • Extensive Library: Supports a wide array of training-free compression methods including SnapKV, StreamingLLM, and KVzip.
  • Flexible Pipeline: Offers a custom transformers pipeline for easy integration and evaluation.
  • Dual-Phase Compression: Supports compression during both the prefilling phase and the decoding phase (via DecodingPress).
  • Quantization Support: Compatible with QuantizedCache for further memory reduction.
  • Benchmarking Tools: Includes a CLI and notebooks to measure accuracy, peak memory usage, and total time gain.

Related

  • Dispatch
    NVIDIA KVPress: Toolkit for KV Cache Compression in Long-Context LLMs

    NVIDIA has released KVPress, a Python toolkit that implements state-of-the-art KV cache compression techniques to reduce the memory footprint and increase decoding speed of long-context Large Language Models.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Project
    vllm-project/llm-compressor

    LLM Compressor is a Python library that quantizes and prunes large language models into the `compressed‑tensors` format, enabling memory‑efficient deployment with vLLM. It supports many low‑precision formats (NVFP4, FP8, INT4, etc.), several PTQ/GPTQ algorithms, DDP and disk‑offloading for huge models, and ships pre‑quantized checkpoints for popular LLMs.

  • Project
    quantumaikr/quant.cpp

    A high-performance C library for LLM inference that uses KV cache compression to enable long-context document understanding on consumer hardware, reducing the need for chunk-based RAG.

  • Dispatch
    Optimizing LLM Inference: A Deep Dive into KVBoost

    KVBoost introduces chunk-level KV cache reuse and AWQ layer streaming to drastically reduce TTFT and enable the deployment of large models on consumer hardware.