xlite-dev/LeetCUDA

What it solves

LeetCUDA provides a comprehensive, structured learning path for beginners to master modern CUDA programming, specifically tailored for high-performance AI workloads. It bridges the gap between basic CUDA knowledge and the implementation of complex, production-grade kernels like FlashAttention and HGEMM, using PyTorch for integration and testing.

How it works

The project organizes CUDA learning into a tiered progression of over 200 kernels, ranging from "Easy" (element-wise operations) to "Hard++" (advanced Tensor Core implementations). It utilizes PyTorch Python bindings to allow users to implement a custom CUDA kernel, bind it to Python, and run tests to verify correctness and performance. The repository includes specialized implementations of HGEMM and FlashAttention-2 using pure MMA PTX instructions, demonstrating advanced techniques like multi-staging, block swizzling, and fine-grained tiling.

Who it’s for

It is designed for developers and AI researchers who want to learn CUDA programming from the ground up, as well as those preparing for technical interviews in the GPU kernel optimization space.

Highlights

• Extensive Kernel Library: Over 200 CUDA kernels categorized by difficulty level.
• High-Performance Implementations: Includes HGEMM achieving 98-100% of cuBLAS performance and a FlashAttention-2 implementation using pure MMA PTX.
• Modern Hardware Support: Build scripts target multiple NVIDIA architectures, including Ada Lovelace (sm_89), Hopper (sm_90a), and Blackwell (sm_120a).
• Advanced Techniques: Demonstrates the use of Tensor Cores, TF32/F16/BF16/F8 precision, and complex memory management like shared memory swizzling and prefetching.