rasbt/LLMs-from-scratch

What this project is

This repository contains the official source code that accompanies the book Build a Large Language Model (From Scratch) by Sebastian Raschka (Manning, 2024, ISBN 978-1633437166). It provides step-by-step code for developing, pretraining, and finetuning a GPT-like large language model. Instead of relying on existing LLM libraries, the code builds the model from the ground up, using PyTorch as the only deep-learning framework.

What you'll learn and build

The chapters mirror the book and take you through the full process of creating a small but functional LLM:

1. Understanding Large Language Models (no code)
2. Working with Text Data – preparing text for training (tokenization and data loading)
3. Coding Attention Mechanisms (including multi-head attention)
4. Implementing a GPT Model from Scratch
5. Pretraining on Unlabeled Data – training the model on raw text
6. Finetuning for Text Classification – adapting the model for tasks like spam detection
7. Finetuning to Follow Instructions – teaching the model to respond to user prompts

Appendices add extra material: an intro to PyTorch, reference reading, exercise solutions, training-loop enhancements, and parameter-efficient finetuning with LoRA.

The README notes that the training and development approach used here mirrors the approach behind large-scale foundation models like those used in ChatGPT, but scaled down for educational purposes. The repository also includes code for loading weights of larger pretrained models for finetuning.

How the repository is organized

Each chapter has a dedicated folder containing the main chapter code as Jupyter notebooks, summary scripts (e.g., gpt.py, gpt_train.py), and exercise solutions. The "Table of Contents" in the README maps each chapter to its main notebooks and supplementary files.

Prerequisites

• Strong Python foundation – listed as the most important prerequisite.
• Familiarity with deep neural networks – useful, since LLMs are built on them, but not strictly required.
• PyTorch basics – useful but not required; Appendix A provides a concise PyTorch introduction.

Hardware requirements

The code in the main chapters is designed to run on conventional laptops within a reasonable timeframe and does not require specialized hardware. It will automatically use a GPU if one is available.

Extras included

• Exercises in every chapter, with solutions in Appendix C.
• Bonus material in the repository, including optional topics like a Byte Pair Encoding tokenizer built from scratch, different attention variants (grouped-query attention, multi-head latent attention, etc.), Mixture-of-Experts, alternative model architectures (Llama 3.2, Qwen3, Gemma 3, etc.), and user-interface demos.
• Companion video course – a 17-hour, 15-minute course that codes through each chapter.
• Sequel book – Build A Reasoning Model (From Scratch), a standalone follow-up that starts from a pretrained model and implements reasoning techniques (inference-time scaling, reinforcement learning, distillation). The sequel has its own GitHub repository.
• Free quiz PDF – a 170-page self-test booklet with ~30 quiz questions per chapter.

Setup and instructions

The README recommends reading the setup folder for guidance on installing Python and required packages. It also notes that if you download the code bundle from the publisher's website instead of the GitHub repo, you may want to check the official GitHub repository for the latest updates.

Limitations on contributions

Because this repository mirrors a printed book, the author notes that he cannot accept contributions that extend the main chapter code, since that would create deviations from the physical book. Feedback is welcomed via the Manning forum and GitHub Discussions.

Citation

The book can be cited in Chicago style:

Raschka, Sebastian. Build A Large Language Model (From Scratch). Manning, 2024. ISBN: 978-1633437166.