wafer-ai/gpu-perf-engineering-resources

A curated resource list for learning AI performance engineering, from GPU fundamentals to production inference.

What it solves

This project provides a structured, curated roadmap for engineers who need to understand and optimize GPU performance for AI workloads. It bridges the gap between high-level model architecture and the low-level hardware execution required to run large-scale inference efficiently.

How it works

The repository acts as a comprehensive reference library. It is organized into a logical progression, starting with fundamental GPU programming concepts and moving through kernel optimization, inference engine design, and distributed system architecture. It relies on primary sources, including research papers, official hardware documentation, and high-performance implementation repositories.

Who it’s for

This is intended for software engineers, performance engineers, and researchers working on AI infrastructure. It is particularly useful for those building or optimizing inference engines, writing custom GPU kernels, or managing large-scale distributed model serving.

Highlights

  • A "Start here" guide that establishes a mental model for inference, from request handling to hardware execution.
  • Deep dives into kernel optimization, covering matrix multiplication, tensor cores, and attention mechanisms.
  • Extensive resources on modern inference engines, including continuous batching, KV cache management, and speculative decoding.
  • Coverage of distributed inference techniques, such as Mixture-of-Experts (MoE) serving and prefill-decode disaggregation.
  • Technical references for specific hardware architectures, including NVIDIA, AMD, Google TPU, and AWS Trainium.

Related

  • Project
    CalvinXKY/InfraTech

    InfraTech is a collection of educational resources and practical code exercises designed to help developers master AI infrastructure, including model training, inference optimization, and distributed systems.

  • Project
    zml/zml

    ZML is an open‑source inference framework written in Zig that compiles AI models (including LLMs) to run at native speed on NVIDIA, AMD, Intel, TPU, and AWS Trainium hardware. It uses Bazel/MLIR for builds, offers a unified model‑loading VFS, and provides ready‑to‑run examples (MNIST, LLMs) with simple platform flags.

  • Project
    algorithmicsuperintelligence/optillm

    OptiLLM is an OpenAI‑compatible inference proxy that applies 20+ reasoning‑boost techniques (best‑of‑N, chain‑of‑thought, MCTS, Mixture‑of‑Agents, etc.) at request time, giving 2‑10× higher accuracy on math, coding and logic tasks without any model training. It works with any OpenAI‑compatible provider via LiteLLM, offers a plugin system (memory, privacy, URL fetch, code execution, web search), and can be installed via pip or Docker.

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.