inclusionAI/cuLA

CUDA kernels for linear attention variants, written in CuTe DSL and CUTLASS C++.

What it solves

cuLA provides high-performance CUDA kernels for linear attention variants, which are designed to replace standard quadratic attention in LLMs to enable more efficient processing of long-context workloads. It specifically targets the latest NVIDIA GPUs (Blackwell and Hopper) to maximize hardware acceleration.

How it works

The library is written using the CuTe DSL and CUTLASS C++, providing hand-tuned implementations of linear attention algorithms such as GLA, KDA (Kimi Delta Attention), GDN, and Lightning Attention. It is designed to be a drop-in replacement for the flash-linear-attention (FLA) library, sharing the same interface so that users can switch to cuLA kernels for better performance.

Who it’s for

AI researchers and engineers working with long-context LLMs who need maximum throughput and efficiency on NVIDIA Hopper (SM90) and Blackwell (SM10X) GPUs.

Highlights

  • Hardware Optimized: Specifically tuned for NVIDIA Blackwell (SM10X) and Hopper (SM90) architectures.
  • Significant Speedups: Demonstrates substantial performance gains over FLA baselines, including up to 7.56x for FlashKDA prefill on Hopper.
  • Broad Algorithm Support: Implements variants like Lightning Attention and KDA with support for both prefill and decode phases.
  • FLA Compatible: Designed for seamless integration with the flash-linear-attention ecosystem.

Related

  • Project
    MoonshotAI/FlashKDA

    A high-performance CUDA kernel implementation of Kimi Delta Attention (KDA) built on CUTLASS for NVIDIA SM90+ GPUs, serving as an optimized backend for the flash-linear-attention library.

  • Project
    fla-org/flash-linear-attention

    A hardware-efficient library of subquadratic kernels and layers for linear attention, state space models, and hybrid LLM architectures across multiple hardware backends.

  • Project
    Tencent/hpc-ops

    A production-grade operator library for LLM inference that provides highly optimized CUDA kernels for attention, MoE, and sampling to reduce serving latency.

  • Project
    facebookresearch/xformers

    A toolbox of optimized, customizable building blocks for Transformer models that accelerates research by providing memory-efficient and high-performance CUDA kernels.

  • Project
    QwenLM/FlashQLA

    FlashQLA is a Python‑package kernel library that speeds up the Gated‑Delta‑Rule linear‑attention used in Qwen models. Built with TileLang, it offers 2‑3× faster forward and ~2× faster backward passes on modern NVIDIA GPUs, automatic intra‑card parallelism, and a drop‑in API compatible with flash‑linear‑attention.