fla-org/flash-linear-attention

What it solves

Flash Linear Attention (fla) provides a unified, hardware-efficient library of building blocks for modern sequence models. It addresses the computational inefficiency of standard attention mechanisms by implementing subquadratic kernels and layers for linear attention, state space models (SSMs), and hybrid architectures, allowing for faster training and inference across different hardware platforms.

How it works

The library implements a wide variety of "token mixing" layers and fused modules that can replace standard multi-head attention. It provides platform-agnostic implementations verified on NVIDIA, AMD, and Intel hardware. The project includes specialized kernels for various architectures like Mamba, RetNet, and RWKV, and supports advanced features like context parallel for distributed training across the sequence dimension.

Who it’s for

AI researchers and engineers building large language models (LLMs) or sequence models who want to move beyond standard Transformer attention to more efficient, linear-time alternatives.

Highlights

• Broad Model Support: Implements a vast array of architectures including Mamba-2/3, RWKV-6/7, RetNet, GLA, and Gated DeltaNet.
• Hardware Agnostic: Verified support for NVIDIA (CUDA), AMD (ROCm), and Intel (XPU), as well as NPU and CPU.
• Training Ready: Includes fused linear and cross-entropy layers to reduce memory usage and supports context parallel for distributed training.
• Extensible Architecture: Provides a flexible framework for training hybrid models and replacing standard attention layers with efficient alternatives.