datawhalechina/llm-algo-leetcode

What it solves

This project addresses the gap between understanding LLM theory and actually running, measuring, and optimizing real models. It provides a structured, hands-on path that takes learners from PyTorch implementations of Transformers and training loops down to low-level GPU kernels in Triton and CUDA. It helps people understand not just how models compute, train, and serve, but also how to diagnose and improve memory, communication, throughput, and latency by running reproducible benchmarks and profiling real hardware.

How it works

The tutorial is organized as a series of runnable Jupyter notebooks divided into five parts (Part 00 through Part 05). The recommended path starts with Part 02 (PyTorch algorithm practice) to build implementation intuition, then backfills prerequisites from Part 00/01 as needed, and finally moves to Part 03 (Triton kernels) and Part 04 (CUDA and system optimization). Part 05 is planned as integrated projects for architecture decisions. Alternatively, learners can enter through five main learning routes — architecture evolution, post-training optimization, inference optimization, performance optimization, and operator optimization — and use cross-cutting topics (quantization, parallelism, performance analysis, etc.) to fill specific gaps. The notebooks include runnable code, exercise sections, answer cells, and basic validation, and the project emphasizes evidence-based experimentation with real GPU runs, inference backends, and profiling tools.

Who it's for

This is built for LLM algorithm learners who want to understand training, fine-tuning, inference, and model compression through PyTorch; performance and systems learners who want to study memory, profiling, communication, Triton, CUDA, and GPU optimization; and project practitioners who want to make reproducible engineering judgments using benchmarks, real GPU runs, and inference backends.

Highlights

• Notebook-first design: Every lesson revolves around runnable code, exercises, answer cells, and basic validation, making it easy to modify and experiment.
• Goal-oriented entry: You can start from Part 02 for implementation practice or jump into any of the five main learning routes and topic-based navigation.
• Algorithm-to-system progression: Follows the PyTorch → Triton → CUDA path, connecting model implementation with kernel optimization, memory management, communication, and system performance.
• Evidence-based projects: Uses benchmarks, profiling, real GPU runs, and inference backends to compare throughput, latency, memory, quality, and cost.
• Comprehensive coverage: Parts 00–04 are complete (with continuous refinement), covering everything from Python/PyTorch basics to raw CUDA; Part 05 is under construction for integrated projects and architecture decisions.