rednote-machine-learning/RedKnot

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

What it solves

RedKnot addresses the high computational and memory costs associated with serving Large Language Models (LLMs) over long contexts. It specifically targets the reduction of Time to First Token (TTFT) and arithmetic compute overhead without significantly sacrificing model quality.

How it works

RedKnot is a model-aware execution framework that combines three primary mechanisms:

  1. Head Decomposition and Aggregation: It classifies attention heads based on their long-context behavior. Local heads are prepared offline for reuse, while global, retrieval, or recovery heads remain online, and their contributions are merged back into the model.
  2. Sparse FFN and MoE Execution: It uses token-level importance to determine which rows enter Feed-Forward Network (FFN) work and employs adaptive expert Top-K selection for Mixture-of-Experts (MoE) models, assigning more experts only when necessary.
  3. SegPagedAttention: This organizes KV pages and visibility per head and segment, allowing different types of heads (global, local, retrieval) to consume different context scopes without requiring a uniform cache layout.

Who it’s for

It is designed for developers and researchers deploying long-context LLMs (such as DeepSeek V4, Mistral, Qwen, and Llama) who need to optimize inference speed and compute efficiency on high-end GPU hardware (e.g., NVIDIA H200/B300).

Highlights

  • Significant Performance Gains: Targets a 2–5× speedup in hot-state TTFT and 70–90% savings in arithmetic compute.
  • Minimal Quality Loss: Aims for quality regression within 1 percentage point.
  • Broad Model Support: Includes adapters and benchmarks for DeepSeek V4 Flash, Mistral, Qwen3, Qwen3.5 MoE, and Llama 3.3.
  • SGLang Integration: Built on top of SGLang and currently migrating core code to vLLM.

Related

  • Project
    BlinkDL/RWKV-LM

    RWKV is a parallelizable RNN architecture that delivers Transformer-level LLM performance with linear-time complexity and constant-space memory usage, eliminating the need for a KV-cache.

  • Project
    AstraNetLab/CacheRoute

    A lightweight LLM scheduling framework that reduces prefill overhead by reusing KV caches for frequently accessed external knowledge across LLM systems.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Dispatch
    Fast & Efficient LLM Inference with vLLM Course Launch

    vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.