gpu-mode/lectures

Material for gpu-mode lectures

This repo hosts the supporting files for the GPU MODE lecture series, a community-run set of talks on GPU computing. The project itself is just a folder of materials: each lecture has its own subfolder or external link with slides, Jupyter notebooks, code, or PDFs.

What it covers: Practical and technical GPU topics, with a strong focus on CUDA and PyTorch integration. The stated companion text is the book Programming Massively Parallel Processors (PMPP). Topics range from beginner CUDA basics and profiling, to advanced subjects like Flash Attention, quantization, Triton internals, collective communication (NCCL), and even GPU kernels for ARM CPUs.

Who it's for: People who already have some background in Python or PyTorch and want to go deeper into GPU programming. The lectures are recorded on a YouTube channel, and there's a Discord community (links in the README) for discussion.

Format: There is no main codebase or library here — the repository is an organized collection of links and files, one entry per lecture. Each entry typically lists a speaker (with their social or GitHub link) and either a local file path (e.g., lecture_003/ for a notebook) or an external link to slides on Google Drive, Dropbox, or Colab. Some lectures have hands-on notebooks that you can run directly in Google Colab.

A practical note on organization: Lecture numbers are not sequential — the list jumps from 43 to 57, then 67, then 78, etc. — so the numbering reflects the actual talk order, not a complete curriculum. A few entries have incomplete metadata (e.g., Lecture 43 has no speaker name, only slides).

In short: If you want to learn GPU programming from real-world practitioners, this repo is a well-indexed syllabus with all the original teaching materials (slides, notebooks, code) in one place.

Related

  • Project
    kvcache-ai/Mooncake

    Mooncake is a high-performance infrastructure project that provides a disaggregated, KV cache-centric architecture to improve the efficiency, scalability, and latency of large-scale LLM inference and training.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Project
    mizorewww/laya-mlx

    Laya‑MLX is a Python library that runs ConvAI’s Laya LLMs locally on Apple‑Silicon via the MLX runtime. It provides ultra‑low‑latency *typed‑decision* inference (choice, score, yes/no) without PyTorch or cloud services, includes a router for multilingual/English models, optional compilation, confidence gating, and CLI tools such as a snake‑game demo.

  • Project
    google/gemma.cpp

    A lightweight, standalone C++ inference engine for Google's Gemma models, designed for research and experimentation with a minimalist, CPU-optimized implementation.