SHI-Labs/NATTEN

Fast Multi-dimensional Sparse Attention

What it solves

NATTEN addresses the high computational cost of standard self-attention, which typically scales quadratically with the number of tokens. It provides a way to introduce locality and sparsity into the attention mechanism, making it more efficient for multi-dimensional data like images (2D) and volumes (3D).

How it works

NATTEN implements Neighborhood Attention (NA), a sliding window self-attention mechanism. Similar to convolutions, it allows users to define a kernel_size, stride, and dilation to control the attention span in each dimension. It also supports causal masking to prevent tokens from attending to future tokens.

To achieve high performance, the project provides Fused Multi-Headed Attention (FMHA) and Fused Neighborhood Attention (FNA) kernels optimized for NVIDIA GPUs, including native kernels for Hopper (SM90) and Blackwell (SM100, SM103) architectures.

Who it’s for

It is designed for researchers and developers building Transformers for multi-dimensional feature maps, particularly those working with computer vision or 3D data processing.

Highlights

  • Multi-dimensional support: Native support for 2D and 3D token layouts.
  • High-performance kernels: Optimized fused kernels for all NVIDIA architectures from Maxwell (SM50) to Blackwell (SM100/103).
  • Flexible configuration: Control over kernel size, stride, and dilation per dimension.
  • Causal masking: Ability to toggle causal attention for multi-dimensional sparse attention.

Related

  • Project
    deepseek-ai/FlashMLA

    FlashMLA is DeepSeek’s CUDA library of dense and token‑level sparse attention kernels (including FP8 KV‑cache support) optimized for NVIDIA SM90/SM100 GPUs. It powers DeepSeek‑V3/V3.2/V4 models, delivering up to 660 TFLOPs on H800 GPUs and offering a fused kernel that combines Q‑norm, RoPE, attention and FP8 casting. Installation is via a standard pip build, and usage follows simple Python APIs (`flash_mla_with_kvcache`, `flash_mla_sparse_fwd`, etc.).

  • Project
    inclusionAI/cuLA

    A library of high-performance CUDA kernels for linear attention variants like KDA and Lightning Attention, optimized for NVIDIA Blackwell and Hopper GPUs.

  • Project
    fla-org/flash-linear-attention

    A library of hardware-efficient building blocks and optimized kernels for linear attention, state space models, and hybrid LLM architectures.

  • Project
    HKUSTDial/flash-sparse-attention

    A high-performance trainable sparse attention implementation that combines Flash Attention's memory efficiency with sparse computation for handling extremely long sequences in Transformers.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a Python library that supplies ultra‑fast, hardware‑aware CUDA kernels for LLM inference—covering attention, GEMM, MoE, sampling, and related ops. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM), supports low‑precision FP8/FP4, works with all modern NVIDIA GPUs, and is already used in major serving frameworks like vLLM and TensorRT‑LLM.