rednote-machine-learning/RedKnot

Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

What it solves

RedKnot addresses the high computational cost and latency associated with serving Large Language Models (LLMs) over long contexts. It specifically targets the Time to First Token (TTFT) and arithmetic compute requirements, reducing redundant work during the prefill phase of long-context requests.

How it works

RedKnot is a model-aware execution framework built on SGLang that employs three primary mechanisms to optimize long-context serving:

  • Head Decomposition and Aggregation: It classifies attention heads into reusable local heads (prepared offline) and online global/retrieval heads. Their contributions are merged back into the model without altering the external interface, supporting various attention mechanisms like MLA, MHA, GQA, and sliding-window attention.
  • Sparse FFN and MoE Execution: It uses token-level importance to determine which rows enter the Feed-Forward Network (FFN) and employs adaptive expert Top-K selection for Mixture-of-Experts (MoE) models, reducing the number of experts activated per token.
  • SegPagedAttention: This organizes KV pages and visibility per head and segment, allowing different types of heads (global, local, retrieval) to consume different context scopes without requiring a uniform cache layout.

Who it’s for

It is designed for developers and researchers working on high-performance LLM serving infrastructure, particularly those deploying models like DeepSeek-V4-Flash, Mistral, Qwen, and Llama 3.3 in long-context RAG (Retrieval-Augmented Generation) scenarios.

Highlights

  • Performance Gains: Targets a 2–5× speedup in hot-state TTFT and 70–90% savings in arithmetic compute.
  • LMM Support: Built on SGLang and supports a wide range of architectures including DeepSeek, Mistral, Qwen, and Llama.
  • Hardware Compatibility: Optimized for NVIDIA H200/B300 GPUs and includes ongoing adaptation for Huawei Ascend NPUs.
  • Reproducible Benchmarks: Provides packaged reproduction paths for DeepSeek-V4-Flash across various context lengths (64K to 440K).

Related

  • Project
    WeianMao/triattention

    A KV-cache compression method for long-reasoning LLMs that uses trigonometric frequency-domain compression to reduce memory usage by 10.7x and boost throughput by 2.5x without accuracy loss.

  • Dispatch
    Optimizing LLM Inference: A Deep Dive into KVBoost

    KVBoost introduces chunk-level KV cache reuse and AWQ layer streaming to drastically reduce TTFT and enable the deployment of large models on consumer hardware.

  • Project
    Zefan-Cai/KVCache-Factory

    A unified playground for implementing and evaluating KV cache compression, retrieval, merging, and quantization methods to optimize long-context LLM inference memory usage.

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.

  • Project
    0xSero/turboquant

    A KV cache compression library for LLM inference that uses orthogonal rotation and optimal quantization to reduce VRAM usage and extend context windows.