Dao-AILab/flash-attention

Fast and memory-efficient exact attention

What it solves

FlashAttention optimizes the attention mechanism in transformer models to be faster and more memory-efficient. It addresses the high memory and computational overhead associated with the standard attention operation, specifically by reducing the number of times data is moved between different types of GPU memory (IO-awareness).

How it works

It implements a specialized version of scaled dot product attention that minimizes memory access. The project provides multiple versions (FlashAttention, FlashAttention-2, 3, and 4) optimized for different GPU architectures, including NVIDIA (Ampere, Ada, Hopper, Blackwell) and AMD (ROCm). It supports advanced features like causal masking, sliding window local attention, and multi-query/grouped-query attention (MQA/GQA).

Who it’s for

It is designed for machine learning engineers and researchers who are training or deploying large language models (LLMs) and need to maximize GPU throughput and reduce memory consumption during the attention phase.

Highlights

  • Multi-GPU Architecture Support: Optimized kernels for NVIDIA Hopper (H100) and Blackwell (B200) GPUs, as well as AMD ROCm-compatible GPUs.
  • Memory Efficiency: Reduces IO overhead to speed up both forward and backward passes.
  • Inference Optimization: Includes specialized functions for incremental decoding with KV cache support.
  • Flexible Attention Patterns: Supports causal masking, ALiBi slopes, and sliding window attention.

Related

  • Project
    flagos-ai/FlagAttention

    FlagAttention is a Triton‑based library offering fast, memory‑efficient attention kernels (FlashAttention, Piecewise, paged, sparse, quantised, recurrent, etc.) for training and inference of large models across many accelerators. Stable dense operators are ready for production; alpha variants provide experimental features like KV‑cache paging, block‑sparse decoding, and INT8 quantisation.

  • Project
    HKUSTDial/flash-sparse-attention

    A high-performance trainable sparse attention implementation that combines Flash Attention's memory efficiency with sparse computation for handling extremely long sequences in Transformer models.

  • Project
    lucidrains/x-transformers

    x‑transformers is a PyTorch library for building configurable transformer models (encoder‑decoder, decoder‑only, encoder‑only, vision) with many research‑grade features (flash attention, memory tokens, alternative normalizations, sparse/top‑k attention, talking‑heads, GLU variants, etc.). It’s aimed at researchers and practitioners who want a concise, extensible implementation to prototype new ideas.

  • Dispatch
    Optimizing LLMs in Production: Precision, Attention, and Architecture

    Hugging Face outlines key techniques for efficient LLM deployment, focusing on lower precision quantization, Flash Attention for memory efficiency, and architectural optimizations like RoPE, ALiBi, MQA, and GQA.

  • Project
    vllm-project/vllm

    vLLM is an open‑source library for high‑throughput, low‑latency serving of large language models. It introduces PagedAttention for efficient KV‑cache handling, supports many quantisation formats, integrates fast attention kernels, and offers an OpenAI‑compatible API that runs on GPUs, CPUs and other accelerators.