gau-nernst/learn-cuda

What it solves

This repository provides a structured learning path for writing high-performance CUDA kernels, specifically tailored for those using PyTorch. It bridges the gap between high-level PyTorch code and low-level GPU hardware acceleration by providing practical examples of common AI-related operations.

How it works

The project consists of a series of progressive examples (01 through 09) that implement fundamental GPU operations. These range from simple vector addition to complex operations like Flash Attention and matrix multiplication optimized for specific GPU architectures (SM80, SM100, SM120). It also includes guidance on using profiling tools like Nsight Compute, the PyTorch profiler, and Perfetto to optimize kernel performance.

Who it’s for

Developers and researchers who want to learn how to write custom CUDA kernels to optimize PyTorch models, and those interested in the underlying hardware architecture of NVIDIA GPUs.

Highlights

• Architecture-specific optimizations: Examples covering matrix multiplication for different SM versions and CDNA3.
• AI-focused primitives: Implementations of Softmax, Flash Attention, and row/block-scaled matrix multiplication.
• Hardware deep-dives: Detailed explanations of GPU memory hierarchy (Global, L2, Shared, and Registers) and SIMT execution.
• Profiling toolkit: Practical instructions for using ncu and compute-sanitizer for debugging and performance tuning.