mlc-ai/modern-gpu-programming-for-mlsys

What it solves

This project provides a structured educational resource for learning how to program modern GPU kernels, specifically targeting the Blackwell-class GPU architecture. It bridges the gap between understanding hardware mechanics and writing high-performance, state-of-the-art kernels for machine learning systems.

How it works

The book uses a progression-based approach: it first explains the GPU hardware (memory hierarchy, Tensor Cores, and asynchronous engines) and then introduces TIRx (Tensor IR next), a Python domain-specific language. Readers use TIRx to write kernels at the intermediate representation level, building up from basic matrix multiplication to complex implementations like Flash Attention 4.

Who it’s for

This resource is designed for developers and researchers interested in low-level GPU programming for machine learning. Note that running the provided code examples requires access to Blackwell-class hardware (such as a B200 GPU).

Highlights

• Comprehensive guide to modern GPU hardware, including TMA, Tensor Memory, and warpgroups.
• Hands-on introduction to TIRx for writing GPU kernels in Python.
• Step-by-step development of high-performance kernels, including tiled GEMM and Flash Attention 4.
• Practical appendices on benchmarking, profiling, and debugging asynchronous kernels.