HKUSTDial/flash-sparse-attention

Trainable fast and memory-efficient sparse attention

What it solves

Flash-Sparse-Attention (FSA) addresses the computational and memory bottlenecks associated with processing extremely long sequences in Transformer models. It combines the memory efficiency of Flash Attention with sparse computation to reduce the overhead of the attention mechanism.

How it works

FSA implements high-performance trainable sparse attention kernels. It utilizes a CuTe backend for optimal performance and provides support for various attention patterns including causal, local window, and gated attention. To optimize workload balancing and precision, it employs techniques like Split-KV (for forward/decode) and Split-QO (for backward passes), as well as fused quantization for low-precision computation on hardware lacking native FP8 support.

Who it’s for

Researchers and developers building large-scale Transformer models that require efficient handling of long-context windows and high-performance training and inference (decode) workloads.

Highlights

  • Versatile Attention Support: Supports dense, sparse, and gated attention, as well as Grouped Query Attention (GQA) and Multi Query Attention (MQA).
  • Memory & Compute Efficiency: Integrates Flash Attention's memory efficiency with sparse computation and Paged Attention.
  • Flexible Configuration: Offers per-head arbitrary window sizes and local ranges via Flex Local Window Attention.
  • Hardware Optimization: Includes fused quantization and workload balancing (Split-KV/Split-QO) to maximize GPU/XPU/NPU/PPU performance.

Related

  • Project
    Relaxed-System-Lab/Flash-Sparse-Attention

    Flash‑Sparse‑Attention (FSA) is a Triton‑based, drop‑in PyTorch module that implements an optimized version of Native Sparse Attention. By reordering loops and splitting the work into three specialized kernels, it cuts memory traffic and removes atomic adds, giving 2‑3× speed‑ups on long‑sequence LLM training and inference on modern NVIDIA GPUs. It works with PyTorch ≥ 2.4, transformers ≥ 4.45, and supports fp16/bf16 on Ampere/Hopper hardware.

  • Project
    fla-org/flash-linear-attention

    A hardware-efficient library of subquadratic kernels and layers for linear attention, state space models, and hybrid LLM architectures across multiple hardware backends.

  • Project
    flagos-ai/FlagAttention

    FlagAttention is a Triton‑based library offering fast, memory‑efficient attention kernels (FlashAttention, Piecewise, paged, sparse, quantised, recurrent, etc.) for training and inference of large models across many accelerators. Stable dense operators are ready for production; alpha variants provide experimental features like KV‑cache paging, block‑sparse decoding, and INT8 quantisation.

  • Project
    Dao-AILab/flash-attention

    A fast and memory-efficient implementation of the attention mechanism for transformer models, reducing GPU memory IO to increase speed and efficiency.

  • Dispatch
    Hugging Face Reads: Long-range Transformers

    Hugging Face analyzes four key architectures—Longformer, Compressive Transformer, Linformer, and Performer—designed to reduce the quadratic memory and time complexity of standard Transformer self-attention to linear complexity.