Zefan-Cai/R-KV

What it solves

Reasoning models (like DeepSeek-R1) generate extremely long chain-of-thought (CoT) traces, which bloat the key-value (KV) cache and consume massive amounts of GPU memory. Traditional cache compression methods designed for long prompts often fail during long generations, significantly dropping in accuracy because they prune critical reasoning steps.

How it works

R-KV is a redundancy-aware KV-cache compression method that operates on-the-fly during the decoding process. It maintains two fixed-size buffers: a staging buffer for new tokens and a budgeted cache for retained tokens. It selects which tokens to keep based on a joint score combining two factors:

1. Importance Scoring: Uses attention weights from a small window of recent "observation tokens" to identify tokens critical for future predictions.
2. Redundancy Estimation: Uses cosine similarity between key vectors to identify and prune near-duplicate tokens, keeping only the most recent versions of similar semantics.

Tokens are retained based on a trade-off parameter (λ) that balances importance and redundancy, allowing the model to maintain high accuracy with a fraction of the memory.

Who it’s for

Developers and researchers working with reasoning LLMs who need to reduce memory overhead and increase throughput during long-form generation without sacrificing reasoning quality.

Highlights

• Extreme Memory Savings: Reduces KV cache memory usage by up to 90% while maintaining near-full accuracy.
• Zero-Shot Integration: Training-free and plug-and-play; it can be dropped into inference engines without fine-tuning.
• High Throughput: Significant gains in tokens-per-second (up to 6.6x) by allowing larger batch sizes in GPU memory.
• Production-Ready Ports: Includes performance-hardened integrations for vLLM and SGLang, featuring CUDA-graph decode and async scheduling.
• Accuracy Boost: In some cases, removing redundant "noise" tokens actually improves reasoning accuracy beyond the full-cache baseline.