HKUSTDial/flash-sparse-attention

Trainable fast and memory-efficient sparse attention

What it solves

Flash-Sparse-Attention (FSA) addresses the computational and memory bottlenecks associated with processing extremely long sequences in Transformer models. It combines the memory efficiency of Flash Attention with sparse computation to reduce the overhead of the attention mechanism during training and inference.

How it works

FSA implements a high-performance, trainable sparse attention mechanism. It supports various attention patterns including causal, local window, and gated attention, while utilizing techniques like Split-KV and Split-QO for workload balancing. It also features fused quantization for low-precision computation on hardware lacking native FP8 support and supports Paged Attention for efficient memory management during decoding.

Who it’s for

Researchers and developers building large-scale Transformer models that require efficient handling of long-context windows and high-performance training and decoding on GPUs, XPUs, NPUs, or PPUs.

Highlights

  • Comprehensive Attention Support: Supports dense, sparse, and gated attention with forward and backward passes.
  • Flexible Configuration: Offers per-head arbitrary window sizes, Grouped Query Attention (GQA), and Multi Query Attention (MQA).
  • Workload Balancing: Uses Split-KV (for forward/decode) and Split-QO (for backward) to optimize performance.
  • Hardware Compatibility: Fused quantization allows low-precision computation on older hardware.
  • Decoding Optimizations: Includes Top-k gather KV-cache decode and Paged Attention.

Related

  • Project
    Dao-AILab/flash-attention

    A fast and memory-efficient implementation of the attention mechanism for Transformers, reducing memory IO to accelerate LLM training and inference.

  • Project
    flagos-ai/FlagAttention

    A collection of memory-efficient attention operators implemented in Triton, allowing for easier customization of attention mechanisms compared to CUDA-based FlashAttention.

  • Project
    deepseek-ai/FlashMLA

    A library of optimized attention kernels for Multi-head Latent Attention (MLA), powering DeepSeek-V3 and V3.2 with support for sparse and dense attention patterns.

  • Project
    fla-org/flash-linear-attention

    A library of hardware-efficient, training-ready implementations for linear attention, state space models, and hybrid LLM architectures across multiple hardware platforms.

  • Dispatch
    Hugging Face Reads: Long-range Transformers

    Hugging Face analyzes four key architectures—Longformer, Compressive Transformer, Linformer, and Performer—designed to reduce the quadratic memory and time complexity of standard Transformer self-attention to linear complexity.