MoonshotAI/FlashKDA

FlashKDA: high-performance Kimi Delta Attention kernels

What it solves

FlashKDA provides high-performance CUDA kernels for Kimi Delta Attention (KDA), a linear attention mechanism. It addresses the need for efficient, hardware-accelerated computation of these kernels to improve the speed and memory efficiency of models using this specific attention variant.

How it works

The project implements KDA kernels built on the CUTLASS library, specifically targeting NVIDIA SM90 (H100) and above architectures. It integrates as a backend for the flash-linear-attention (FLA) library, allowing users to replace the standard Triton-based implementation with these optimized C++ kernels for better performance.

Who it’s for

It is designed for AI researchers and engineers working with linear attention models and those utilizing the flash-linear-attention framework to optimize their model inference and training performance on modern NVIDIA GPUs.

Highlights

  • High-performance kernels built on CUTLASS for SM90+ GPUs.
  • Seamless integration with the flash-linear-attention library via auto-dispatch.
  • Support for variable-length batching using cumulative sequence lengths (cu_seqlens).
  • Support for both stateless and stateful (initial/final state) recurrent computations.

Related

  • Project
    inclusionAI/cuLA

    A library of high-performance CUDA kernels for linear attention variants like KDA and Lightning Attention, optimized specifically for NVIDIA Blackwell and Hopper GPUs.

  • Dispatch
    Kimi Linear: An Expressive, Efficient Attention Architecture

    Kimi Linear is a hybrid linear attention architecture that outperforms full attention in short-context, long-context, and RL scaling regimes while reducing KV cache usage by up to 75%.

  • Project
    NVIDIA/cutlass

    A collection of CUDA C++ and Python DSL abstractions for implementing high-performance matrix-matrix multiplication (GEMM) and the related linear algebra operations on NVIDIA GPUs.

  • Dispatch
    Kimi Delta Attention: From Linear Attention to DPLR Transitions

    Kimi Delta Attention (KDA) evolves linear attention by introducing per-channel forgetting and a delta-rule update, transforming the state transition into a diagonal-plus-low-rank (DPLR) operation for efficient recurrent and chunkwise execution.

  • Project
    Tencent/hpc-ops

    A production-grade operator library for LLM inference that provides highly optimized CUDA kernels for attention, MoE, and sampling to reduce serving latency.