datawhalechina/llms-from-scratch-cn

仅需Python基础，从0构建大语言模型；从0逐步构建GLM4\Llama3\RWKV6，深入理解大模型原理

📚 What is LLMs From Scratch (CN)?

A community‑driven tutorial series (in Chinese) that walks you through building a large language model from the ground up using only PyTorch notebooks. The repo re‑hosts the material from the original rasbt/LLMs‑from‑scratch project, adds Chinese translations, and expands the curriculum to cover several modern architectures such as GLM‑4, Llama 3 and the RWKV family.

🎯 Goal & Audience

Goal – Teach developers and researchers how a ChatGPT‑style model works, how to implement the core components (tokeniser, attention, transformer blocks, training loops) and how to train a small but functional model on their own data.
Audience – Anyone with basic Python/PyTorch knowledge who wants a hands‑on, code‑first path into LLM internals. The material is written in Chinese and assumes a modest programming background.

📂 How the repo is organised

Section	What you’ll find	Typical files
1️⃣ 基础知识	Theory plus short notebooks for a quick start.	`Codes/ch02/.ipynb`, `Codes/ch03/.ipynb`
2️⃣ 模型实现	Step‑by‑step notebooks that implement attention, multi‑head attention, a GPT‑style model, and training scripts.	`Codes/ch04/gpt.py`, `Codes/ch05/train.py`, `Codes/ch05/generate.py`
3️⃣ 进阶模型讨论	Architecture‑specific notebooks that load and inspect weights of ChatGLM‑3, Llama‑3, RWKV‑V2‑V6, MiniCPM, etc.	`Model_Architecture_Discussions//.ipynb`
4️⃣ 附录	PyTorch refresher, distributed‑training example, extra exercises.	`Codes/appendix‑A/*.ipynb`, `Codes/appendix‑D/appendix‑D.ipynb`

✨ Key Highlights

Complete code for a minimal GPT‑like model, from tokenisation to pre‑training and generation.
Hands‑on notebooks that can be run locally (or in Colab) without needing a massive compute cluster.
Multi‑model coverage – the same pedagogical flow is reused to explore GLM, Llama, RWKV and MiniCPM architectures.
Exercise solutions are provided for each chapter, making self‑study straightforward.
Open‑source licensing – Apache 2.0 for the code, CC‑BY‑NC‑SA 4.0 for the tutorial content.

🚀 Getting started (quick steps)

Clone the repo

git clone https://github.com/datawhalechina/llms-from-scratch-cn.git
cd llms-from-scratch-cn

Create a Python environment (recommended Python 3.9+, PyTorch 2.x).

pip install -r requirements.txt   # if a requirements file exists, otherwise install torch, tqdm, numpy, etc.

Run the introductory notebook to see a minimal GPT implementation:
```
jupyter notebook Codes/ch04/01_main-chapter-code/ch04.ipynb
```
Follow the chapter order (Chapter 2 → Chapter 5) to progress from data loading → attention → full model → pre‑training.
When ready, explore the Model_Architecture_Discussions notebooks to see how the same concepts map onto larger public models.

📜 License

Code – Apache 2.0 (permissive, allows reuse in other projects).
Tutorial text & notebooks – Creative Commons BY‑NC‑SA 4.0 (non‑commercial sharing with attribution).

🙋‍♀️ Contributing & Community

Open issues for bugs or feature requests.
Discussions are used for learning‑group coordination.
Contributions are welcomed via pull‑requests; see the Datawhale contribution guide linked in the README.

TL;DR – LLMs From Scratch (CN) is a practical, Chinese‑language learning kit that lets you write, train, and experiment with small‑scale LLMs yourself, while also offering deep‑dive notebooks on popular modern architectures. Perfect for self‑learners who want to demystify the internals of ChatGPT‑style models without wading through abstract papers alone.