NVIDIA/cudnn-frontend

cuDNN Frontend is NVIDIA's modern, open-source entry point to the cuDNN library and a growing collection of high-performance open-source kernels.

NVIDIA cuDNN Frontend (cuDNN‑FE)

What it is – A thin, header‑only C++ library plus Python bindings that wrap NVIDIA’s cuDNN Graph API. It gives developers a higher‑level, easier‑to‑use way to build and run high‑performance deep‑learning primitives (convolutions, GEMM, attention, normalization, etc.) on modern NVIDIA GPUs (Hopper H100/H200, Blackwell B200/GB200/GB300). The package also ships a growing set of open‑source kernels (Flash‑Attention / SDPA, fused GEMM‑+‑activation, block‑sparse attention, etc.) that can be JIT‑compiled and selected at runtime.

Key features

  • Unified Graph API – Create a cudnn_frontend::graph::Graph object once and reuse it for many executions.
  • Header‑only C++ – No library to link; just #include <cudnn_frontend.h>.
  • Python / PyTorch integration – pip install nvidia‑cudnn‑frontend provides cudnn Python module with pybind11 bindings and ready‑made torch.compile custom ops.
  • Auto‑tuning & plan ranking – The frontend builds multiple engine candidates (including the new OSS “FROST” kernels) and picks the fastest.
  • Open‑source kernels – Flash‑Attention (SDPA), grouped GEMM for Mixture‑of‑Experts, fused RMSNorm+SiLU, block‑sparse and native sparse attention, etc., all written in CuTe DSL and shipped under Apache‑2.0.
  • Support for FP16, BF16, FP8, MXFP8 – Enables the newest low‑precision formats on Hopper/Blackwell.

Typical use cases

  • Researchers building custom transformer models that need the absolute fastest attention kernels on H100/Blackwell.
  • Engineers integrating cuDNN‑accelerated layers into PyTorch pipelines without writing CUDA kernels.
  • Teams training Mixture‑of‑Experts models who want fused grouped‑GEMM + activation kernels to reduce memory traffic.
  • Anyone who wants to inspect, modify, or contribute cutting‑edge GPU kernels (the OSS kernels are JIT‑compiled and can be customized).

Installation

  • Python: pip install nvidia-cudnn-frontend (requires Python 3.10+, NVIDIA driver, CUDA Toolkit, cuDNN ≥ 8.5.0).
  • C++: Just add the include/ directory to the compiler’s include path and compile against the CUDA and cuDNN libraries.
  • From source: pip install -v git+https://github.com/NVIDIA/cudnn-frontend.git for the Python bindings, or use CMake to build the C++ samples (cmake … && cmake --build .).

Ecosystem & integration

  • Works with PyTorch (custom ops under python/cudnn/experimental/ops).
  • Compatible with CUDA Graphs and can be used inside torch.compile pipelines.
  • Provides sample notebooks and C++ examples for convolution, matmul, SDPA/Flash‑Attention, RMSNorm, etc.
  • Documentation hosted at NVIDIA’s developer site and a blog with deep‑dive articles.

Maturity

  • Actively maintained by NVIDIA; versioned releases are published on PyPI.
  • The core frontend API is stable, while new OSS kernels are added incrementally (e.g., FROST GEMM, Flex Attention, HSTU, DSA).
  • Logging and environment‑report utilities help with debugging and reproducibility.

License

  • Primarily Apache License 2.0; a few files are under the MIT License (clearly marked). All third‑party attributions are listed in THIRD_PARTY_LICENSES.txt.

Related

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Project
    uxlfoundation/oneDNN

    A cross-platform library of optimized deep learning operations that serves as an acceleration layer for frameworks like PyTorch and TensorFlow across various CPUs and GPUs.

  • Project
    NVIDIA/TransformerEngine

    NVIDIA Transformer Engine is an open‑source library that speeds up Transformer model training on NVIDIA GPUs by providing fused kernels and automatic mixed‑precision APIs (FP8, MXFP8, NVFP4, BF16, FP16) for PyTorch and JAX. It handles scaling‑factor management, quantization metadata, and distributed communication, letting users adopt low‑precision training without changing their code. Integrated with major LLM frameworks (Hugging Face, Lightning, Megatron‑LM, etc.), it targets large‑scale language, vision, and MoE models on Hopper/Blackwell/Rubin hardware.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance CUDA kernel library for LLMs that optimizes GEMM and MoE operations, featuring runtime JIT compilation and extreme throughput on NVIDIA SM90/SM100 GPUs.

  • Project
    NVlabs/tiny-cuda-nn

    A small, self-contained CUDA framework for training and querying neural networks, featuring a lightning-fast fully fused MLP and multiresolution hash encoding.