raiyanyahya/how-to-train-your-gpt

What it solves

Most machine learning tutorials are either too shallow (just call a pre-built model) or too academic (dense papers and math with no worked examples). This project fills the gap for Python developers who want to truly understand how modern large language models like GPT, LLaMA, and Mistral work internally — without needing a math or ML background.

How it works

You follow 12 chapters in order, each with the same four-step structure: a simple analogy, a worked example with real numbers, fully annotated code (every line says WHAT it does and WHY), and a diagram. Alongside the chapters, 28 standalone explainers dig deeper into specific techniques (RoPE, attention, RMSNorm, SwiGLU, KV cache, sampling, etc.) and two narrative walkthroughs trace a single sentence through the entire model.

You build the model yourself, line by line: a BPE tokenizer, embeddings, RoPE positional encoding, multi-head attention, transformer blocks, a complete 151M-parameter GPT model, a training pipeline (AdamW, cosine warmup, mixed precision, gradient accumulation), and an inference engine (KV cache, temperature, top-k/p, beam search). The default config is tiny (17M params, runs in minutes on CPU), and a larger GPT-2-scale config (151M) is available with a GPU.

Who it's for

• Python developers who are curious about how ChatGPT actually works and want zero ML prerequisites
• Students who want to deeply understand Transformers and can commit to reading ~3,500 lines of commented code
• Engineers evaluating LLM architectures who want to understand tradeoffs like RoPE vs learned embeddings or RMSNorm vs LayerNorm
• Anyone who got lost at "attention" in other tutorials and needs a party analogy plus a numeric example

Basic Python (variables, functions, classes, pip install) is the only prerequisite. No calculus, linear algebra, or PyTorch experience required — topics are taught as you go.

Highlights

• 12 chapters, 7,500+ lines, 28 standalone topic explainers, and 2 narrative walkthroughs
• Every line of code commented with WHAT and WHY — no black boxes
• Implements the latest publicly documented decoder-only architecture: RoPE, RMSNorm, SwiGLU, pre-norm, AdamW, BPE, weight tying, mixed precision
• Each chapter has a companion Jupyter notebook you can run live — watch the model learn in minutes on a small dataset
• A Colab notebook is provided for one-click cloud training
• Includes a fine-tuning guide (LoRA, QLoRA, data preparation) and troubleshooting resources like loss-curve reading and an FAQ
• The project is learning-focused, not a production library — it teaches you to understand, not just to use