meta-pytorch/attention-gym

Helpful tools and examples for working with flex-attention

What it solves

Attention Gym provides a playground and a library of building blocks for implementing and experimenting with novel attention mechanisms. It simplifies the process of creating custom attention masks and score modifications, and provides efficient implementations for sparse and linear attention variants that would otherwise be complex to build from scratch.

How it works

The project operates through three primary workflows:

  1. FlexAttention Integration: It provides mask_mod and score_mod functions that can be passed directly into PyTorch's FlexAttention APIs to customize how attention is calculated.
  2. Specialized Kernels: It includes dedicated APIs and kernels for sparse attention (via gather_attn) and linear attention (including GDN and KDA paths).
  3. Practical Examples: It offers a suite of benchmarks and real-world workloads covering paged attention, ring attention, and CUDA Graphs to demonstrate how to apply these building blocks in production-like scenarios.

Who it’s for

It is designed for AI researchers and developers who are building new attention variants or looking to optimize attention performance using PyTorch 2.5+.

Highlights

  • Integration with PyTorch FlexAttention for easy mask and score modifications.
  • Support for sparse attention patterns and efficient linear attention kernels.
  • Specialized primitives for GDN and KDA (chunk, recurrent, and decode paths).
  • Comprehensive examples covering variable sparse attention, compilation, and profiling.

Related

  • Project
    lucidrains/perceiver-pytorch

    A PyTorch library that implements the Perceiver and Perceiver IO architectures (and a small experimental variant). It provides ready‑to‑use `Perceiver`, `PerceiverIO`, and `PerceiverLM` classes, supports Fourier positional encoding, and is installable via `pip install perceiver-pytorch`. The repo is a genuine AI/ML project focused on a frontier transformer model for handling large, multimodal inputs.

  • Project
    Shmuma/ptan

    PTAN (PyTorch AgentNet) is a small open‑source library that supplies reusable RL building blocks—experience buffers, training loops, and Ignite integration—for PyTorch‑based agents working with OpenAI‑Gym environments.

  • Project
    ixaxaar/pytorch-dnc

    A PyTorch library implementing Differentiable Neural Computers (DNC), Sparse DNC (SDNC) and Sparse Access Memory (SAM) with install instructions, API usage, debugging support, and example training tasks (copy, addition, argmax).

  • Project
    meta-pytorch/torchtune

    torchtune is a PyTorch‑based library that provides ready‑to‑run recipes and configs for fine‑tuning, RL‑HF, distillation, quantization‑aware training, and inference of many modern LLMs (Llama, Gemma, Qwen, Phi, etc.). It abstracts distributed training, memory‑saving tricks, and device support behind a simple CLI (`tune`) and YAML files, integrating with Hugging Face, torch‑ao, FSDP2, and logging tools. Though development stopped in 2025, it remains a practical, open‑source toolkit for LLM experimentation.

  • Project
    webis-de/small-text

    A Python library that provides ready‑to‑use active‑learning loops for text classification, supporting scikit‑learn, PyTorch, and Hugging Face transformers, with many query strategies, initialization methods, and stopping criteria.