NVIDIA/kvpress

LLM KV cache compression made easy

KVPress – KV‑Cache Compression for Long‑Context LLMs

What it is

  • A Python library that implements a wide range of presses – algorithms that prune or compress the key‑value (KV) cache of transformer‑based language models during the prefilling phase (and optionally during decoding). Reducing the KV cache cuts memory use and can speed up inference for very long contexts (e.g., 1 M tokens).

Why it matters

  • In modern LLMs the KV cache grows linearly with the number of processed tokens. Running a 70 B model on a 1 M‑token document can need >300 GB of GPU memory. KVPress lets researchers and developers experiment with many state‑of‑the‑art compression methods without rewriting model code.

Key features

  • Plug‑and‑play pipeline – KVPressTextGenerationPipeline registers as a Hugging Face pipeline (kv-press-text-generation).
  • Large catalog of presses – >30 ready‑to‑use methods, from simple random pruning to sophisticated attention‑based and matrix‑decomposition techniques (e.g., KnormPress, SnapKVPress, KVzipPress, CURPress).
  • Prefill and decoding compression – DecodingPress and related wrappers allow periodic compression while tokens are being generated.
  • Composable – Wrapper presses (AdaKVPress, ComposedPress, ChunkPress, etc.) let you stack or combine strategies.
  • Training‑free – All listed presses work without extra model fine‑tuning.
  • Quantized cache support – Works with Hugging Face’s QuantizedCache (e.g., 4‑bit via optimum-quanto).
  • Multi‑GPU ready – Compatible with accelerate device maps.
  • Benchmarks & leaderboard – Evaluation CLI and notebooks, plus a public HF Space leaderboard on the RULER 4k‑token benchmark.

Installation

pip install kvpress               # basic install
# or, for development with optional deps (evaluation, flash‑attention)
uv sync --extra eval --extra flash-attn

Basic usage example

from transformers import pipeline
from kvpress import ExpectedAttentionPress

pipe = pipeline(
    "kv-press-text-generation",
    model="Qwen/Qwen3-8B",
    device_map="auto",
    dtype="auto",
)

context = "A very long text you want to compress once and for all"
question = "\nA question about the compressed context"
press = ExpectedAttentionPress(compression_ratio=0.5)

answer = pipe(context, question=question, press=press)["answer"]
print(answer)

The press compresses the KV cache of the context tokens before the model attends to the question, enabling you to compare different compression ratios.

Evaluation

  • A CLI lets you run each press on long‑context datasets and report accuracy, memory reduction, and speed‑up.
  • The repository includes notebooks (speed_and_memory.ipynb, wikipedia_demo.ipynb) for interactive profiling.
  • Results for the RULER 4k‑token benchmark are displayed on the Hugging Face leaderboard linked in the README.

Documentation & resources

  • Colab notebook demo, HF Space demo, and a blog post introduce the library.
  • Each press links to its source file and the corresponding arXiv paper for deeper reading.
  • A “new press” notebook guides contributors on adding custom compression methods.

License

  • Apache 2.0 (see badge in README).

Cite

@article{devoto2025expectedattention,
  title={Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution},
  author={Devoto, Alessio and Jeblick, Maximilian and J{\'e}gou, Simon},
  journal={arXiv preprint arXiv:2510.00636},
  year={2025},
  url={https://arxiv.org/abs/2510.00636}
}

Where to get it

Related

  • Dispatch
    NVIDIA KVPress: Toolkit for KV Cache Compression in Long-Context LLMs

    NVIDIA has released KVPress, a Python toolkit that implements state-of-the-art KV cache compression techniques to reduce the memory footprint and increase decoding speed of long-context Large Language Models.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Dispatch
    Optimizing LLM Inference: A Deep Dive into KVBoost

    KVBoost introduces chunk-level KV cache reuse and AWQ layer streaming to drastically reduce TTFT and enable the deployment of large models on consumer hardware.

  • Dispatch
    Hugging Face KV Cache Quantization

    Hugging Face has introduced KV Cache Quantization in the Transformers library to reduce memory usage for long-context text generation with minimal impact on model quality.

  • Project
    0xSero/turboquant

    A KV cache compression library for LLM inference that uses orthogonal rotation and optimal quantization to reduce VRAM usage and extend context windows.