Dao-AILab/flash-attention

Fast and memory-efficient exact attention

What it solves

FlashAttention is designed to make the attention mechanism in Transformers faster and more memory-efficient. It addresses the quadratic scaling of memory and time relative to sequence length, which typically slows down the processing of long sequences in LLMs.

How it works

It implements an IO-aware exact attention algorithm that reduces the number of memory reads and writes between the GPU's high-bandwidth memory and its on-chip SRAM. The project provides multiple versions (FlashAttention-2, 3, and 4) with increasing optimizations for different GPU architectures, including NVIDIA Ampere, Ada, Hopper, and Blackwell, as well as AMD ROCm support via Triton and Composable Kernel backends.

Who it’s for

It is primarily for AI researchers and engineers building or training large language models who need to optimize the attention layer for higher throughput and lower memory usage on modern GPUs.

Highlights

  • Multi-GPU Architecture Support: Optimized kernels for NVIDIA (including H100/B200) and AMD (MI200/MI300) hardware.
  • Inference Optimizations: Includes flash_attn_with_kvcache for efficient incremental decoding and in-place KV cache updates.
  • Advanced Attention Variants: Supports causal masking, sliding window local attention, and ALiBi (attention with linear bias).
  • Flexible Head Configurations: Supports Multi-Query Attention (MQA) and Grouped-Query Attention (GQA).

Related

  • Project
    flagos-ai/FlagAttention

    A collection of memory-efficient attention operators implemented in Triton, allowing for easier customization of attention mechanisms compared to CUDA-based FlashAttention.

  • Project
    HKUSTDial/flash-sparse-attention

    A high-performance trainable sparse attention implementation that combines Flash Attention's memory efficiency with sparse computation for handling extremely long sequences in Transformers.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.

  • Project
    xlite-dev/ffpa-attn

    A fast and memory-efficient exact attention implementation for large head dimensions (up to 1024), providing significant speedups over PyTorch SDPA through SRAM and register optimizations.

  • Dispatch
    Optimizing LLMs in Production: Precision, Attention, and Architecture

    Hugging Face outlines key techniques for efficient LLM deployment, focusing on lower precision quantization, Flash Attention for memory efficiency, and architectural optimizations like RoPE, ALiBi, MQA, and GQA.