xlite-dev/LeetCUDA

An Open-source book with Modern CUDA learning Notes for Beginners - BF16/FP8/FP4, HGEMM, CuTe, Flash-Attention, etc.

What it solves

LeetCUDA provides a comprehensive, structured learning path for mastering modern CUDA programming, specifically targeting the high-performance requirements of AI and LLM kernels. It bridges the gap between basic CUDA knowledge and the implementation of state-of-the-art operators like FlashAttention and high-performance GEMM (General Matrix Multiply).

How it works

The project is organized as an open-source book and a corresponding repository of over 200 CUDA kernels. It guides users through a tiered difficulty system (Easy to Hard++), moving from simple element-wise operations to complex Tensor Core implementations using MMA PTX instructions. Each topic follows a workflow of implementing a custom CUDA kernel, creating PyTorch Python bindings, and running tests.

Who it’s for

It is designed for developers and researchers who want to learn how to write high-performance GPU kernels for deep learning, ranging from beginners to those looking to optimize advanced LLM components.

Highlights

  • Extensive Kernel Library: Includes over 200 kernels covering element-wise ops, reductions, and advanced matrix multiplications.
  • Tensor Core Mastery: Detailed implementations of HGEMM and FlashAttention-2 using pure MMA PTX, achieving performance close to cuBLAS.
  • Agentic Integration: Provides a leetcuda-cpp-kernel skill that can be integrated with coding agents like GitHub Copilot and Claude Code.
  • Comprehensive Documentation: A 600+ page open-source book with 40+ chapters of detailed explanations.

Related

  • Project
    gau-nernst/learn-cuda

    A collection of CUDA kernel implementations and educational resources for PyTorch users to learn GPU programming and hardware optimization.

  • Dispatch
    Mastering CUDA: Navigating the Landscape of GPU Programming Resources

    An exploration of the best resources for learning CUDA, from foundational books to modern kernel source code and high-level Python frameworks.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance CUDA kernel library for LLMs that optimizes GEMM and MoE operations, featuring runtime JIT compilation and extreme throughput on NVIDIA SM90/SM100 GPUs.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Project
    CVCUDA/CV-CUDA

    CV‑CUDA is NVIDIA’s open‑source GPU‑accelerated computer‑vision library. It provides fast, scalable image/video operators (resize, filter, color conversion, etc.) that run entirely on the GPU and integrate with C/C++ and Python AI frameworks. Install via pre‑built wheels (`cvcuda-cu12` or `cvcuda-cu13`) for Linux, decode images directly to GPU with `nvimgcodec`, and chain CV‑CUDA operators before feeding data to deep‑learning models. The library targets modern NVIDIA GPUs (Turing and newer) and is used in production systems such as Microsoft Bing visual search and Tencent Cloud video processing.