WeianMao/triattention

TriAttention — Efficient long reasoning with trigonometric KV cache compression. Enables OpenClaw local deployment on memory-constrained GPUs.

What it solves

TriAttention addresses the high memory overhead and slow throughput of the KV (Key-Value) cache in long-reasoning LLMs. It reduces the memory footprint of the KV cache by up to 10.7x and increases throughput by up to 2.5x on long reasoning tasks without sacrificing accuracy.

How it works

Instead of selecting representative queries to score keys, TriAttention uses trigonometric frequency-domain compression. It identifies that pre-RoPE Q/K vectors in long reasoning models concentrate around fixed centers that determine distance preferences via a trigonometric series. By scoring keys using these centers and norms, it enables accurate KV cache compression without the the overhead of existing attention-based methods.

Who it’s for

Developers and researchers working with long-context LLMs, specifically those deploying models like Qwen3 and DeepSeek-R1, who need to reduce VRAM usage and increase inference speed on hardware ranging from consumer GPUs (like the RTX 4090) to NVIDIA DGX systems and Apple Silicon Macs.

Highlights

  • Significant Efficiency Gains: Achieves 10.7x KV memory reduction and 2.5x throughput boost on AIME25 benchmarks.
  • Broad Integration: Officially integrated into NVIDIA TensorRT-LLM and supports vLLM and SGLang backends.
  • Hardware Versatility: Compatible with Apple Silicon (via MLX) and AMD GPUs (via community ggml port).
  • Reasoning-Focused: Maintains full attention accuracy on complex reasoning datasets like AIME24, AIME25, and MATH-500.

Related

  • Project
    Zefan-Cai/R-KV

    R-KV is a redundancy-aware KV-cache compression method for reasoning models that reduces memory usage by up to 90% and boosts throughput without losing accuracy.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Dispatch
    NVIDIA KVPress: Toolkit for KV Cache Compression in Long-Context LLMs

    NVIDIA has released KVPress, a Python toolkit that implements state-of-the-art KV cache compression techniques to reduce the memory footprint and increase decoding speed of long-context Large Language Models.

  • Project
    NVIDIA/kvpress

    A framework for implementing and benchmarking KV cache compression methods to reduce memory usage and increase inference speed for long-context LLMs.

  • Project
    LMCache/LMCache

    A KV cache management layer for LLM inference that reduces TTFT and improves throughput by making KV caches persistent and reusable across requests and engines.