meta-pytorch/attention-gym

Helpful tools and examples for working with flex-attention

What it solves

Attention Gym provides a centralized collection of kernels, guides, and examples for implementing novel attention mechanisms. It simplifies the process of creating and using custom attention masks and score modifications, particularly for those using PyTorch's FlexAttention API, while also providing efficient implementations of sparse and linear attention.

How it works

The library operates through three primary workflows:

  1. FlexAttention Integration: It provides mask_mod and score_mod functions that can be passed directly into PyTorch's FlexAttention APIs.
  2. Specialized Kernels: It includes APIs and kernels for sparse attention patterns, as well as efficient implementations for GDN and KDA (chunk, recurrent, and decode paths).
  3. Practical Examples: A dedicated examples directory provides benchmarks and real-world workloads, covering topics like CUDA Graphs, compilation, profiling, and variable sparse attention.

Who it’s for

It is designed for researchers and developers working with attention mechanisms in deep learning, specifically those looking to experiment with new attention variants or optimize performance using sparse or linear attention kernels.

Highlights

  • Support for FlexAttention masks and score modifications.
  • Efficient kernels for sparse attention and linear attention (GDN and KDA).
  • Utility functions for creating and combining attention masks.
  • Comprehensive examples covering paged, ring, and variable sparse attention.

Related

  • Project
    SandAI-org/MagiAttention

    A distributed attention mechanism designed for ultra-long context and heterogeneous mask training, providing linear scalability and high performance for large-scale AI models.

  • Project
    Dao-AILab/flash-attention

    A fast and memory-efficient implementation of the attention mechanism for Transformers, reducing memory IO to accelerate LLM training and inference.

  • Project
    lucidrains/x-transformers

    A concise, feature-rich Transformer library that implements a wide range of experimental architectural variants and optimizations from research papers.

  • Project
    kyegomez/zeta

    A modular PyTorch framework providing reusable, high-performance building blocks for assembling state-of-the-art AI models, including attention mechanisms, quantization, and multi-modal architectures.

  • Project
    Wenyueh/MinivLLM

    A custom vLLM-style inference engine implementation that benchmarks Flash Attention and Paged Attention to optimize LLM prefilling and decoding.