datawhalechina/zero-to-sglang

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

zero‑to‑sglang – A hands‑on tutorial for building an LLM inference engine

What it is – An open‑source, bilingual (English & Chinese) course that walks you through the theory and code behind modern LLM inference. It starts with the fundamentals (KV‑cache, pre‑fill vs. decode, compute‑ vs. memory‑bound bottlenecks), then guides you to implement a mini‑sglang inference engine from scratch, and finally shows how the real SGLang project implements advanced tricks such as RadixAttention, paged KV‑cache, continuous batching, hierarchical caching, quantisation, and distributed attention.

Who it’s for – Anyone with basic Python and PyTorch knowledge who wants to understand how LLMs are served efficiently. No GPU is required for the first part; later chapters can be run on CPU but benefit from a GPU for performance experiments.

Key learning outcomes

  • Grasp core inference concepts (KV‑cache, pre‑fill/decoding, roof‑line model).
  • Write a functional mini‑inference engine (forward pass, generation, HTTP serving, scheduler).
  • Read and comprehend the actual SGLang source code, including cutting‑edge optimisations.
  • Gain practical skills in profiling, trace analysis, and contributing a PR to an active open‑source project.

Course structure

Part Content Status
0 – Prep Coding ethics, first SGLang server deployment ✅
I – Foundations LLM basics, inference theory, GPU intro, KV‑cache, benchmarking ✅
II – Build mini‑sglang Architecture overview, request flow, forward pass, KV‑cache optimisation, HTTP & concurrency, RadixAttention, etc. ✅ (most chapters) – some later chapters still in progress
III – Advanced techniques Attention back‑ends, quantisation, hierarchical caching, DP/EP/PP scaling, speculative decoding ✅ for hierarchical caching; others in progress
IV – Contribute Deploy cookbook, profiling, full PR workflow Planned

How to get started

# clone the repo
git clone https://github.com/datawhalechina/zero-to-sglang.git
cd zero-to-sglang
# follow the chapter‑by‑chapter instructions in `course-material/eng/` (or the Chinese version)

The repository contains the markdown course files, a community/ folder for learner contributions, and a VitePress site configuration for the online version.

Why it matters – As LLMs become larger, inference cost dominates. Understanding and building efficient inference engines is a scarce skill. This tutorial bridges the gap between high‑level concepts and production‑grade code, and it directly connects learners to the real SGLang project (maintained by RadixArk), enabling them to contribute back.

License – Creative Commons Attribution‑NonCommercial‑ShareAlike 4.0.

Related

  • Project
    jmaczan/tiny-vllm

    tiny-vllm is a C++ and CUDA implementation of a high-performance LLM inference engine, built from scratch and accompanied by a step-by-step course. It focuses on serving Llama 3.2 1B Instruct with features like continuous batching, PagedAttention, and CUDA-optimized kernels.

  • Dispatch
    Building a High-Performance LLM Inference Engine from Scratch with tiny-vLLM

    An exploration of tiny-vLLM, a C++ and CUDA-based educational project that teaches the internals of LLM inference, from Safetensors loading to PagedAttention.

  • Dispatch
    Fast & Efficient LLM Inference with vLLM Course Launch

    vLLM, in collaboration with DeepLearning.AI and Red Hat, has launched a free intermediate course titled Fast & Efficient LLM Inference with vLLM to teach the full AI deployment lifecycle.

  • Project
    sophgo/LLM-TPU

    LLM-TPU is an open‑source toolkit that compiles HuggingFace LLM/VLM checkpoints into SOPHGO TPU‑specific bmodels and provides ready‑to‑run Python/C++ demos, supporting quantization, KV‑cache, dynamic shapes, and multi‑chip parallelism.

  • Project
    datawhalechina/llms-from-scratch-cn

    A Chinese‑language, hands‑on tutorial that walks users from basic PyTorch concepts to a full GPT‑style implementation, then extends the lessons to modern architectures (GLM‑4, Llama 3, RWKV, MiniCPM). Includes notebooks, training scripts, and exercises, all under Apache 2.0 / CC‑BY‑NC‑SA licensing.