flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

What it solves

FlashInfer addresses the performance bottlenecks in LLM inference by providing highly optimized GPU kernels. It specifically targets the inefficiencies in attention mechanisms, matrix multiplications (GEMM), and Mixture of Experts (MoE) operations, ensuring high throughput and low latency across a wide range of NVIDIA GPU architectures from Turing to Blackwell.

How it works

It acts as a library and kernel generator that provides unified APIs for critical AI operations. It automatically selects the most efficient backend implementation (such as FlashAttention-2/3, cuDNN, CUTLASS, or TensorRT-LLM) based on the user's hardware and specific workload. It supports advanced memory management like paged and ragged KV-caches and low-precision compute (FP8 and FP4) to further increase speed.

Who it’s for

This project is for developers building high-performance LLM serving frameworks (like vLLM or SGLang) and researchers who need state-of-the-art GPU acceleration for inference tasks.

Highlights

  • Broad Architecture Support: Optimized for GPUs from SM75 (Turing) through the latest Blackwell (SM100+) architectures.
  • Advanced Attention: Native support for DeepSeek's Multi-Latent Attention (MLA), Cascade Attention, and sparse attention patterns.
  • Optimized MoE: Fused kernels for Mixture of Experts with support for multiple routing methods and quantized weights.
  • Low-Precision Compute: Support for FP8 and FP4 quantization for GEMM and MoE operations.
  • Production Ready: Compatible with CUDAGraph and torch.compile for minimal serving latency.

Related

  • Project
    NVIDIA/TensorRT-LLM

    An open-source library from NVIDIA that optimizes inference for LLMs and visual generation models using specialized kernels and an efficient runtime to maximize throughput and latency on NVIDIA GPUs.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance tensor core kernel library for NVIDIA GPUs that provides optimized GEMM, MoE, and MQA kernels to accelerate LLM training and inference.

  • Project
    pytorch/FBGEMM

    A repository of highly-optimized kernels for deep learning, providing low-precision matrix multiplication and convolution libraries for server-side inference and generative AI.

  • Project
    vllm-project/vllm

    A high-throughput library for LLM inference and serving that uses PagedAttention to optimize memory management and increase efficiency.

  • Project
    Dao-AILab/flash-attention

    A fast and memory-efficient implementation of the attention mechanism for Transformers, reducing memory IO to accelerate LLM training and inference.