datawhalechina/diy-llm

Diy‑LLM – A Hands‑On Chinese Course for Building Large Language Models

What it is – An open‑source, university‑level curriculum (based on Stanford CS336) that walks learners through every stage of creating, training, and evaluating a large language model (LLM). The repository bundles:

• Structured lecture notes (Chinese + English) covering theory, system‑level optimizations, scaling laws, multimodal models, and alignment.
• Six graded programming assignments that implement core components such as a tokenizer, a minimal Transformer, FlashAttention, distributed training, data preprocessing, and RL‑based alignment.
• Sample code, scripts, and utilities (PyTorch‑based) that can be run on a single GPU or scaled to multi‑node clusters.
• Links to PDFs, an online VitePress site, and references to Chinese‑domestic models (Qwen, DeepSeek, etc.).

Why it matters – While many LLM tutorials stop at “run a pretrained model”, Diy‑LLM lets you build a model from the ground up, giving you the same engineering experience as large‑scale AI labs. It is tailored for Chinese‑speaking students, providing localized explanations, resource‑friendly tips, and examples that work with the domestic open‑source ecosystem.

Key components

Component	What you learn / get
Tokenizer chapter	BPE algorithm, Unicode normalization, a hand‑written tokenizer trainer.
Model architecture	RoPE, RMSNorm, SwiGLU, AdamW, pre‑norm/post‑norm, learning‑rate schedules.
System optimizations	Mixed‑precision, gradient accumulation, FLOPs/VRAM estimation, FlashAttention, Triton kernels.
GPU programming	CUDA basics, Tensor Cores, shared memory, an intro to Triton.
Distributed training	Data‑, model‑, pipeline‑parallelism, ZeRO‑1/2/3, FSDP, All‑Reduce.
Scaling laws	Chinchilla law, how to predict compute‑optimal model size.
Inference	KV‑cache, speculative decoding, quantization (GPTQ/AWQ), PagedAttention, continuous batching.
Data engineering	Filtering, MinHash deduplication, PII scrubbing, curriculum learning.
Evaluation	MMLU, HumanEval, HELM, CEval, AlpacaEval, Arena ranking via lm‑evaluation‑harness and evalscope.
Alignment	Supervised fine‑tuning, GRPO, rule‑based verifiers, RL‑based reward models.
Multimodal	CLIP, LLaVA, Qwen‑VL, Chameleon pipelines.

How to get started

# Clone the repo
git clone https://github.com/datawhalechina/diy-llm.git
cd diy-llm

# Follow the learning path
# 1️⃣ Read the docs in docs/zh/ (or docs/en/)
# 2️⃣ Pick an assignment, e.g. assignment1-basics, and run its training script
#    (install the required Python packages listed in the assignment’s README)
# 3️⃣ Progress through the chapters, scaling up to distributed training and RL‑HF.

The README notes that CPU‑only work is possible for theory and small‑scale debugging, but full‑model training requires a GPU (cloud instances are recommended).

Who should use it

• Undergraduate or graduate students wanting a deep, implementation‑first understanding of LLMs.
• Engineers preparing to join AI R&D teams who need end‑to‑end experience (data prep → training → inference → alignment).
• Researchers looking for a Chinese‑language teaching resource that mirrors Stanford’s CS336 syllabus.

License – Creative Commons Attribution‑NonCommercial‑ShareAlike 4.0 (CC‑BY‑NC‑SA 4.0).

Contributing – The project welcomes documentation improvements, bug fixes, and new chapters via standard fork‑branch‑PR workflow.

---

Diy‑LLM aims to turn “reading about LLMs” into “building your own LLM”, providing a complete, locally runnable curriculum for Chinese‑speaking learners.