fla-org/flash-linear-attention

What it solves

Flash Linear Attention (fla) provides hardware-efficient implementations of modern sequence models that aim to replace or augment standard Transformer attention. It addresses the computational bottlenecks of traditional softmax attention by providing optimized kernels for linear attention, state space models (SSMs), and hybrid architectures, enabling more efficient training and inference across various hardware platforms.

How it works

The project provides a library of "token mixing" layers and fused modules that can be integrated into LLM architectures. It implements a wide variety of subquadratic kernels—such as Mamba, RetNet, RWKV, and Gated Linear Attention (GLA)—which are platform-agnostic and verified to work on NVIDIA, AMD, and Intel hardware. These implementations are designed to be training-ready and often include fused layers (e.g., fused linear and cross-entropy layers) to reduce memory usage during training.

Who it’s for

AI researchers and machine learning engineers who are developing or deploying sequence models beyond the standard Transformer, specifically those working with linear attention, SSMs, or hybrid LLM architectures.

Highlights

• Broad Model Support: Implements a vast array of state-of-the-art linear attention and SSM models including Mamba2/3, RWKV6/7, RetNet, and GLA.
• Hardware Agnostic: Verified implementations for NVIDIA, AMD, and Intel hardware.
• Training Optimized: Includes fused modules and support for context parallel training to enable efficient distributed training across the sequence dimension.
• Transformers Compatibility: Provides model implementations compatible with the Hugging Face Transformers library.