xlite-dev/LeetCUDA

Modern CUDA Learn Notes with PyTorch for Beginners, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.

What it solves

LeetCUDA provides a comprehensive, structured learning path for beginners to master modern CUDA programming, specifically targeting the optimization of kernels for AI workloads. It bridges the gap between basic CUDA knowledge and high-performance implementations of critical AI operations like matrix multiplication (GEMM) and attention mechanisms.

How it works

The project organizes learning through a progression of over 200 CUDA kernels, ranging from "Easy" to "Hard++" difficulty. Each topic follows a consistent workflow: implementing a custom CUDA kernel, creating PyTorch Python bindings, and running tests. It specifically emphasizes the use of Tensor Cores (via WMMA, MMA, and CuTe) and supports various data types including FP32, FP16, BF16, and FP8. The repository includes high-performance implementations of HGEMM and FlashAttention-2, demonstrating advanced techniques like block swizzling, multi-staging, and fine-grained tiling.

Who it’s for

It is designed for beginners and developers who want to learn how to write high-performance CUDA kernels for deep learning, as well as those preparing for technical interviews in the AI infrastructure space.

Highlights

  • Extensive Kernel Library: Over 200 kernels covering element-wise operations, reductions, and complex AI operators.
  • High-Performance Benchmarks: Includes HGEMM implementations that achieve 98-100% of cuBLAS performance.
  • Advanced Attention Implementations: Features FlashAttention-2 using pure MMA PTX instructions with optimizations like Split-Q and shared QKV SMEM.
  • Modern Hardware Support: Build scripts target Ada Lovelace (sm_89), Hopper (sm_90a), and Blackwell (sm_120a) architectures.
  • PyTorch Integration: All kernels are designed to be used with PyTorch bindings for easy testing and integration.

Related

  • Project
    facebookresearch/xformers

    A toolbox of optimized, customizable building blocks for Transformer models that accelerates research by providing memory-efficient and high-performance CUDA kernels.

  • Dispatch
    Mastering CUDA: Navigating the Landscape of GPU Programming Resources

    An exploration of the best resources for learning CUDA, from foundational books to modern kernel source code and high-level Python frameworks.

  • Project
    deepseek-ai/DeepGEMM

    A high-performance tensor core kernel library for NVIDIA GPUs that provides optimized GEMM, MoE, and MQA kernels to accelerate LLM training and inference.

  • Project
    rwth-i6/returnn

    An extensible training framework for universal recurrent neural networks optimized for fast multi-GPU training and decoding.

  • Project
    Tencent/hpc-ops

    A production-grade operator library for LLM inference that provides highly optimized CUDA kernels for attention, MoE, and sampling to reduce serving latency.