microsoft/vattention

Dynamic Memory Management for Serving LLMs without PagedAttention

What it solves

vAttention addresses the memory management challenges of the KV-cache in LLM serving systems. Specifically, it solves the problem of needing to rewrite custom attention kernels to support dynamic memory allocation (as required by PagedAttention) and improves performance for prefill-bound workloads.

How it works

vAttention decouples the allocation of virtual memory from physical memory using CUDA virtual memory APIs. This allows the system to allocate physical memory on demand while keeping the KV-cache contiguous in virtual memory. Because the memory appears contiguous, it can be used by unmodified attention kernels without requiring the complex user-space demand paging implemented in other systems.

Who it’s for

It is designed for developers of LLM serving systems and researchers focusing on inference optimization and memory management on NVIDIA GPUs.

Highlights

  • Kernel Compatibility: Supports dynamic memory allocation for unmodified attention kernels.
  • Virtual Memory Decoupling: Uses CUDA virtual memory APIs to separate virtual and physical memory allocation.
  • Performance: Offers performance improvements over PagedAttention, particularly for prefill-bound workloads.
  • Integration: Integrated with Sarathi-Serve and supports backends like FlashAttention and FlashInfer.
  • Custom Driver Support: Includes a modified NVIDIA UVM driver to enable smaller page sizes (64KB to 256KB) beyond the default 2MB.

Related

  • Project
    ovg-project/kvcached

    kvcached is an open‑source Python library that adds virtual‑memory‑style, elastic KV‑cache management to LLM serving engines (vLLM, SGLang). It lets multiple models share a GPU, allocating cache memory only when needed, which improves utilization, reduces latency‑to‑first‑token, and cuts serving costs. Features include dynamic allocation, a CLI for memory limits, prefix caching, router/sleep mode, and support for pipeline‑parallel models. Install via pip or Docker, enable with environment variables, and integrate with existing SGLang/vLLM setups. Licensed under Apache 2.0 and used by major tech companies.

  • Project
    ModelEngine-Group/unified-cache-management

    UCM (Unified Cache Manager) is an open‑source library that provides a shared KV‑cache layer for LLM inference engines. It lets multiple requests reuse attention cache, spills cache to external storage to extend accelerator memory, and offers pluggable engine adapters and storage back‑ends. Ideal for chat‑bot servers, multimodal pipelines, and any workload that repeatedly accesses the same context.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.

  • Project
    flagos-ai/FlagAttention

    FlagAttention is a Triton‑based library offering fast, memory‑efficient attention kernels (FlashAttention, Piecewise, paged, sparse, quantised, recurrent, etc.) for training and inference of large models across many accelerators. Stable dense operators are ready for production; alpha variants provide experimental features like KV‑cache paging, block‑sparse decoding, and INT8 quantisation.

  • Dispatch
    vLLM DeepSeek V4 Support: Efficient Long-context Attention

    vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.