QwenLM/FlashQLA

What it solves

FlashQLA addresses the performance bottlenecks in linear attention kernels, specifically for Gated Delta Rule (GDN) Chunked Prefill. It provides a high-performance alternative to existing Triton kernels, reducing latency during both the forward and backward passes of model training and inference.

How it works

Built on TileLang, the library optimizes linear attention through several technical approaches:

• Operator Fusion: It uses fused warp-specialized kernels to overlap data movement with Tensor Core and CUDA Core computations, avoiding the overhead of decomposing tasks into independent kernels.
• Algebraic Reformulation: The forward and backward flows are reformulated to reduce the overhead of Tensor Cores, CUDA Cores, and Special Function Units (SFU) without losing numerical precision.
• Automatic Context Parallelism: It leverages the exponential decay property of the GDN gate to automatically enable intra-card context parallelism (CP) under Tensor Parallelism (TP), long-sequence, and small-head-count settings to maximize GPU SM utilization.

Who it’s for

This library is designed for developers and researchers working with linear attention models, particularly those focusing on pretraining large models or deploying agentic inference on edge devices using NVIDIA Hopper and Blackwell GPUs.

Highlights

• Significant Speedups: Achieves 2-3× forward speedup and 2× backward speedup over FLA Triton kernels.
• Hardware Optimized: Specifically tuned for NVIDIA SM90, SM100, SM103, and SM120/121 architectures.
• Integration: Serves as a plug-and-play acceleration backend for the flash-linear-attention (FLA) API.
• Flexible API: Provides both high-level APIs for ease of use and low-level APIs for separate forward and backward control.