vipshop/ffpa-attn

Kernel Library for Large Headdim Attention (64~1024, BF16/FP8/FP4), 1.5x~15x↑ vs PyTorch SDPA.

What it solves

FFPA (Fast and Memory-Efficient Exact Attention) addresses the performance bottlenecks of standard attention mechanisms when working with large head dimensions (D > 256, up to 1024). Standard implementations often suffer from memory and register pressure at these scales, leading to significant slowdowns. FFPA provides highly optimized kernels that deliver substantial speedups over PyTorch's native Scaled Dot Product Attention (SDPA).

How it works

FFPA uses two primary techniques to optimize attention:

  • Split-D: This tiling strategy breaks the head dimension (D) axis to keep SRAM usage constant, effectively solving the memory bottleneck.
  • TiledMMA: This approach optimizes register usage by splitting the N-dimension of the matrix multiplication, preventing register spilling and allowing the kernels to handle much larger head dimensions than traditional implementations.

The library supports multiple backends, including CUDA, Triton, and CuTe-DSL, and can be integrated into existing PyTorch workflows with a single line of code.

Who it’s for

This project is for AI researchers and engineers working on large-scale model training and inference who need to optimize attention layers, particularly for models with large head dimensions or those requiring high-throughput FP8/FP4 precision.

Highlights

  • High Performance: Achieves 1.5x to 15x speedups over standard PyTorch SDPA.
  • Large Headdim Support: Extends support for head dimensions up to 1024 without precision loss.
  • Quantization Support: Includes experimental support for FP8 and FP4 attention kernels for faster inference.
  • Flexible Integration: Provides a simple API to monkey-patch torch.nn.functional.scaled_dot_product_attention for immediate performance gains.
  • Broad Compatibility: Supports various attention types including Self-Attention, GQA/MQA, Cross-Attention, and Causal/Masked attention.

Related

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    ellipticmarketing/modelrelay

    An OpenAI-compatible local router that benchmarks and automatically forwards requests to the best available free coding models across multiple providers.

  • Project
    wtdcode/vllm-backport

    A specialized fork of the vLLM inference engine optimized for running frontier models on older NVIDIA Ampere and Ada GPUs using custom quantization and memory management.

  • Project
    z-lab/paroquant

    ParoQuant is an INT4 quantization framework for LLMs that uses learned pairwise rotations to reduce weight outliers and maintain high accuracy during efficient inference.