flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

FlashInfer – High‑Performance GPU Kernels for LLM Inference

What it is

  • A C++/CUDA library (with a Python package) that ships a collection of highly‑optimized GPU kernels for the core operations needed when serving large language models (LLMs). It focuses on attention, matrix‑multiply (GEMM), mixture‑of‑experts (MoE) routing, sampling, and a few supporting ops.

Why it matters

  • LLM inference is often bottlenecked by the speed of attention and GEMM kernels, especially when serving many concurrent requests (prefill + decode) or when using low‑precision formats (FP8/FP4). FlashInfer bundles multiple backend implementations (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) and automatically picks the fastest one for the GPU you have, delivering "state‑of‑the‑art" throughput on everything from Turing GPUs (RTX 20) up to the newest Blackwell chips.

Key capabilities

Area What FlashInfer provides
Attention • Paged/ragged KV‑cache for dynamic batching
• Separate kernels for pre‑fill, decode, and mixed (POD) workloads
• Support for DeepSeek MLA, cascade (hierarchical) attention, block‑sparse patterns, and custom JIT‑able variants
GEMM / Linear BF16, FP8, FP4 matrix multiplication; grouped GEMM for LoRA/MoE; Blackwell‑specific NVFP4/MXFP4 kernels
Mixture‑of‑Experts Fused MoE kernels with several routing algorithms (DeepSeek‑V3, Llama‑4, top‑k) and quantized expert weights
Sampling / Decoding Sorting‑free top‑k / top‑p / min‑p sampling; speculative decoding support
Communication Custom AllReduce, multi‑node NVLink (MNNVL) and NVSHMEM integration for distributed inference
Other ops RoPE (including LLaMA 3.1), fused RMSNorm/LayerNorm, fused SiLU/GELU activations

Hardware coverage

  • Works on any NVIDIA GPU from Turing (SM 7.5) onward. Specific kernels are tuned for Ampere, Ada Lovelace, Hopper, and the newest Blackwell architectures, with low‑precision (FP8/FP4) paths only on the latter.

How you use it

  1. Install the Python wheel: pip install flashinfer-python (optional extra wheels give pre‑compiled kernels or JIT‑cache shims).
  2. Call the high‑level API, e.g. flashinfer.single_decode_with_kv_cache(q, k, v) for a single‑decode attention step, or use the richer batch APIs for prefill.
  3. The library integrates with PyTorch’s torch.compile and CUDA Graphs, so you can embed it in low‑latency serving stacks.

Ecosystem adoption

  • Integrated into major open‑source inference servers such as vLLM, TensorRT‑LLM, HuggingFace Text Generation Inference, SGLang, MLC‑LLM, LightLLM, Lorax, and ScaleLLM.

Documentation & community

  • Full docs at https://docs.flashinfer.ai, a Slack community, and a GitHub discussion forum. CI badges show continuous testing across CUDA versions (12.9, 13.0, 13.4).

When to pick FlashInfer

  • You need the fastest possible attention/GEMM kernels on NVIDIA GPUs, especially if you serve mixed prefill‑decode workloads, want low‑precision (FP8/FP4) inference, or run on newer Blackwell hardware. It is a drop‑in replacement for the attention kernels used by existing PyTorch‑based servers.

All details above are taken directly from the repository’s README.

Related

  • Project
    deepseek-ai/FlashMLA

    FlashMLA is DeepSeek’s open‑source library of ultra‑fast attention kernels (sparse and dense) for LLM inference. It provides CUDA and Ascend implementations that work with FP8/FP4‑quantized KV caches, a fused kernel that merges RoPE and casting, and APIs compatible with the popular flash‑attention package. The kernels achieve >1 000 TFlops on modern GPUs and ~95 % of peak on Ascend NPUs, dramatically speeding up inference of DeepSeek‑V4.1 and similar models.

  • Project
    omnimind-ai/OmniInfer

    OmniInfer is a cross‑platform inference engine that lets you run LLMs and vision‑language models locally via a simple CLI and OpenAI‑compatible HTTP server. It supports many backends (llama.cpp, MLX, MNN, etc.), works on desktop, Android and iOS, and provides hardware‑aware optimisation and a terminal UI for chatting.

  • Project
    QwenLM/FlashQLA

    FlashQLA is a Python‑package kernel library that speeds up the Gated‑Delta‑Rule linear‑attention used in Qwen models. Built with TileLang, it offers 2‑3× faster forward and ~2× faster backward passes on modern NVIDIA GPUs, automatic intra‑card parallelism, and a drop‑in API compatible with flash‑linear‑attention.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    pegainfer-project/pegainfer

    A production-ready LLM inference server written in Rust and CUDA that removes the Python runtime to achieve lower latency and reduced memory overhead.