skyzh/tiny-llm

What it solves

Most LLM courses treat models as black boxes or focus on training. tiny-llm is a practical, hands-on course for systems engineers who want to understand how LLM inference actually works under the hood — from loading a model to generating text. It shows the full path: turning tokens into logits, managing memory, and serving requests. The course is designed so you can read the entire implementation and connect the equations to real-world concerns like memory traffic, kernel occupancy, KV-cache growth, batching, and request scheduling.

How it works

The course is built around MLX (Apple's machine learning framework) and runs on Apple silicon. It uses MLX arrays and the MLX extension runtime, but deliberately avoids high-level neural-network layers. In each chapter, you implement the operator yourself — in Python, C++, or Metal — rather than calling the optimized built-in version. MLX remains the reference: it checks your correctness and gives you a performance baseline.

The learning path is four weeks long:

• Week 1: From Matmul to Text. You build a Qwen3 model from scratch using basic array operations, covering attention, RoPE, GQA, RMSNorm, MLP, sampling, and the autoregressive loop.
• Week 2: Faster single-request. You add optimizations like KV-cache, quantized weights, SIMD matmul, fused primitives, and tiled prefill.
• Week 3: Mini vLLM. You implement continuous batching, chunked prefill, paged KV cache, and FlashAttention that reads pages directly.
• Week 4: Coding agent. You build a validated agent that can inspect a workspace, make approved edits, run commands, checkpoint and resume, and evaluate outcomes — all through a bounded loop.

The companion book is at skyzh.github.io/tiny-llm, and the repo includes a reference solution (tiny_llm_ref) plus tests.

Who it's for

Systems engineers who already understand programming and low-level performance but want to grasp how LLM inference works end to end. It's a great fit if you have an Apple Silicon Mac and want to learn without needing expensive GPU servers. The course assumes you can read code and are comfortable with concepts like matrices, memory, and concurrency — but it doesn't assume deep ML background.

Highlights

• End-to-end understanding: You build the entire inference path, not just pieces, and connect math to real hardware behavior.
• Runs locally on Apple silicon: No GPU cluster needed; one machine with shared memory and Metal access.
• Uses a real modern model (Qwen3-4B) with grouped-query attention, QK normalization, BF16 activations, and 4-bit weights.
• Learn by doing: You implement operators yourself in Python, C++, or Metal, with MLX as the oracle.
• Four-week structured path that escalates from basic matrix ops to a mini-vLLM and a coding agent.
• Reference solution and tests included for self-checking.
• Community support via Discord.