microsoft/vattention

Dynamic Memory Management for Serving LLMs without PagedAttention

What it solves

vAttention addresses the memory management challenges of the KV-cache in LLM serving systems. Specifically, it solves the problem of needing to rewrite custom attention kernels to support dynamic memory allocation (as required by PagedAttention) and improves performance for prefill-bound workloads.

How it works

vAttention decouples the allocation of virtual memory from physical memory using CUDA virtual memory APIs. This allows the system to allocate physical memory on demand while keeping the KV-cache contiguous in virtual memory. Because the memory appears contiguous, it can be used by unmodified attention kernels without requiring the complex user-space demand paging implemented in other systems.

Who it’s for

It is designed for developers of LLM serving systems and researchers focusing on inference optimization and memory management on NVIDIA GPUs.

Highlights

  • Kernel Compatibility: Supports dynamic memory allocation for unmodified attention kernels.
  • Virtual Memory Decoupling: Uses CUDA virtual memory APIs to separate virtual and physical memory allocation.
  • Performance: Offers performance improvements over PagedAttention, particularly for prefill-bound workloads.
  • Integration: Integrated with Sarathi-Serve and supports backends like FlashAttention and FlashInfer.
  • Custom Driver Support: Includes a modified NVIDIA UVM driver to enable smaller page sizes (64KB to 256KB) beyond the default 2MB.

Related

  • Project
    ovg-project/kvcached

    A KV cache library that brings virtual memory abstraction to LLM serving, enabling elastic and demand-driven GPU memory sharing across multiple models.

  • Project
    ModelEngine-Group/unified-cache-management

    A cache management framework for LLMs that reduces inference latency by persisting the KV cache and using sparse attention retrieval to minimize redundant computations.

  • Project
    vllm-project/vllm

    A high-throughput library for LLM inference and serving that uses PagedAttention to optimize memory management and increase efficiency.

  • Project
    flagos-ai/FlagAttention

    A collection of memory-efficient attention operators implemented in Triton, allowing for easier customization of attention mechanisms compared to CUDA-based FlashAttention.

  • Dispatch
    vLLM DeepSeek V4 Support: Efficient Long-context Attention

    vLLM now supports DeepSeek V4-Pro and V4-Flash, implementing a new attention mechanism that enables context lengths up to one million tokens with significant KV cache memory savings.