NVIDIA/cudnn-frontend

NVIDIA cuDNN Frontend (cuDNN‑FE)

What it is – A thin, open‑source wrapper around NVIDIA’s cuDNN library that lets developers build and run high‑performance deep‑learning graphs (convolutions, matrix‑multiplies, attention, normalisation, etc.) from C++ or Python. It exposes the newer cuDNN Graph API through a header‑only C++ interface and a Python package (with PyTorch bindings), and ships a growing set of open‑source kernels written in NVIDIA’s CuTe DSL.

Why it matters – cuDNN is the low‑level GPU library that powers most NVIDIA‑accelerated AI frameworks. The frontend removes a lot of the boiler‑plate required to describe complex sub‑graphs, adds automatic autotuning, and makes it easy to plug in custom kernels (e.g., Flash‑Attention, fused GEMM + SwiGLU, block‑sparse attention). This gives researchers and engineers a way to get backend‑level performance without writing CUDA kernels from scratch.

---

Key Features (as listed in the README)

• Unified Graph API – Create reusable cudnn_frontend::graph::Graph objects that describe arbitrary sub‑graphs (convolution, GEMM, SDPA, RMSNorm, etc.).
• Header‑only C++ – Just include <cudnn_frontend.h>; no library linking needed.
• Python bindings – Install via pip install nvidia-cudnn-frontend; bindings built with pybind11 and include native PyTorch integration.
• Open‑source kernels – A catalog of CuTe‑DSL kernels such as:
  • Flash‑Attention / Scaled‑Dot‑Product Attention (SDPA)
  • FROST GEMM engine (JIT‑compiled Blackwell GEMM with fused epilogues)
  • Grouped GEMM for Mixture‑of‑Experts (MoE) with fused activations (SwiGLU, GLU, sReLU, etc.)
  • Block‑Sparse and Native Sparse attention kernels
  • Fused RMSNorm + SiLU, linear‑attention kernels (GDN, KDA, etc.)
• Autotuning & engine ranking – The frontend builds a list of candidate plans (including the open‑source engines) and selects the fastest one at runtime.
• Debug logging – Environment variables (CUDNN_FRONTEND_LOG_INFO, CUDNN_FRONTEND_LOG_FILE) give tensor‑level traces useful for debugging.
• Support for newest GPUs – Targets Hopper (H100/H200) and Blackwell (B200/GB200/GB300) architectures, covering FP16, BF16, FP8, and MXFP8 precisions.

---

Typical Use Cases

Scenario	How cuDNN‑FE helps
Research prototypes – trying a new attention variant or MoE layout	Write a short CuTe DSL kernel or use one of the shipped kernels; plug it into a graph with a few lines of C++/Python.
Performance‑critical training – LLMs, diffusion models, vision transformers	Use the built‑in Flash‑Attention and fused GEMM kernels to reduce memory traffic and achieve state‑of‑the‑art TFLOPs on H100/Blackwell.
Framework integration – extending PyTorch or TensorFlow with custom ops	Import cudnn from Python, build a graph, and expose it as a torch.autograd.Function via the provided experimental ops.
Production inference – low‑latency serving on NVIDIA GPUs	Create persistent graph objects that can be reused across requests; the frontend’s autotuner picks the optimal plan once and reuses it.
Kernel development & education – learning how high‑performance kernels are built	The open‑source CuTe DSL kernels are fully visible, JIT‑compiled, and can be modified or re‑compiled locally.

---

Installation & Getting Started

Platform	Steps
Python	pip install nvidia-cudnn-frontend (requires Python ≥ 3.9, NVIDIA driver, CUDA Toolkit, cuDNN ≥ 8.5.0).
C++	Header‑only – add include/ to the compiler’s include path and #include <cudnn_frontend.h>.
Build from source	```bash
git clone https://github.com/NVIDIA/cudnn-frontend.git
pip install -v git+https://github.com/NVIDIA/cudnn-frontend.git # Python bindings
mkdir build && cd build
cmake -DCUDNN_PATH=/path/to/cudnn -DCUDAToolkit_ROOT=/path/to/cuda ../
cmake --build . -j$(nproc)

| **Samples** | C++ samples in `samples/cpp`; Python notebooks in `samples/python`. |
| **Environment debugging** | `python -m cudnn.collect_env` collects version/GPU info for bug reports. |

---
### Ecosystem & Integration
- **PyTorch** – The package ships `torch.compile`‑compatible custom ops (`sdpa` etc.) and can be used from regular PyTorch code via the `cudnn` Python module.
- **CuTe DSL** – NVIDIA’s domain‑specific language for writing high‑performance kernels; the repo includes many ready‑made kernels.
- **NVIDIA software stack** – Works with the standard CUDA Toolkit, cuDNN library, and the newer cuDNN Graph API.
- **Open‑source kernels** – Contribute new kernels or modify existing ones; they are opt‑in via `CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1`.

---
### Maturity & Activity
- Actively maintained by NVIDIA (release notes, blog posts, YouTube talks linked in the README).
- Versioned on PyPI (`nvidia-cudnn-frontend`) with regular releases.
- Open‑source kernel contributions are listed in `ACKNOWLEDGEMENTS.md` and the repo includes a full test suite for the kernels.
- The core API (header‑only C++ and Python bindings) is stable; new kernels are added as “OSS engines” and can be toggled on/off.

---
### License
- Primarily **Apache License 2.0** (see `LICENSE.txt`).
- Some files are under the **MIT License**; each source file declares its SPDX identifier.  Third‑party attributions are listed in `THIRD_PARTY_LICENSES.txt`.

---
### TL;DR
`cudnn-frontend` is a genuine, production‑grade library that gives developers a clean, high‑level way to harness NVIDIA’s cuDNN Graph API and a suite of open‑source, high‑performance kernels for the latest GPU architectures.  It is ideal for anyone building or optimizing deep‑learning workloads (LLMs, MoE, attention‑heavy models) on Hopper or Blackwell GPUs, and it integrates smoothly with both C++ projects and Python/PyTorch codebases.