FutureMLS-Lab/OSCAR

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

What it solves

OSCAR addresses the high memory overhead of the KV (Key-Value) cache in Large Language Models, which often limits context length and throughput. It enables 2-bit quantization of the KV cache (INT2) while maintaining accuracy levels close to the original 16-bit (BF16) precision, particularly for complex reasoning and coding tasks where other 2-bit methods typically fail.

How it works

OSCAR uses an offline calibration process to analyze how a model actually uses its attention mechanism. It captures activations on a small dataset to estimate "attention-aware" covariance structures for Keys and Values. From these, it derives per-layer orthogonal rotations and clipping thresholds that align the quantization process with the directions the model most needs for accuracy.

To further protect quality, it employs a mixed-precision strategy: a small "sink" of initial tokens and a recent window of tokens are kept in BF16, while the bulk of the cache is stored in INT2. This is implemented via fused mixed-precision Flash-Attention kernels for high performance.

Who it’s for

  • LLM Deployers: Those looking to reduce KV cache memory by ~8x to support much longer contexts or larger batch sizes on limited hardware.
  • Local LLM Users: Users running models on consumer hardware (e.g., MacBooks) who want to run large models (like Qwen3-32B) with long context windows.
  • ML Researchers: Developers working on KV cache quantization and efficient inference engines.

Highlights

  • Extreme Compression: Reduces KV cache memory by approximately 8x compared to BF16.
  • Performance Gains: Enables up to 7x higher throughput at large batch sizes and up to 3x faster batch-size-1 decoding.
  • Framework Integration: Built into SGLang and llama.cpp (via specific forks).
  • High Fidelity: Maintains near-BF16 quality on reasoning benchmarks (GPQA, HumanEval) where naive 2-bit quantization usually collapses.
  • Rotation Zoo: Provides pre-computed calibrated rotations on Hugging Face so users can skip the calibration phase.

Related

  • Dispatch
    Hugging Face KV Cache Quantization

    Hugging Face has introduced KV Cache Quantization in the Transformers library to reduce memory usage for long-context text generation with minimal impact on model quality.

  • Project
    Zefan-Cai/R-KV

    R-KV is a redundancy-aware KV-cache compression method for reasoning models that reduces memory usage by up to 90% and boosts throughput without losing accuracy.

  • Project
    quantumaikr/quant.cpp

    A high-performance C library for LLM inference that uses KV cache compression to enable long-context document understanding on consumer hardware, reducing the need for chunk-based RAG.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Dispatch
    Optimizing LLMs in Production: Precision, Attention, and Architecture

    Hugging Face outlines key techniques for efficient LLM deployment, focusing on lower precision quantization, Flash Attention for memory efficiency, and architectural optimizations like RoPE, ALiBi, MQA, and GQA.