rasbt/reasoning-from-scratch

What this project is

rasbt/reasoning-from-scratch is the official code repository for the book Build a Reasoning Model (From Scratch) by Sebastian Raschka (Manning, 2025, ISBN 9781633434677). It contains all the code that accompanies the book, organized chapter by chapter.

What it does

The project teaches you, hands-on, how a reasoning large language model (LLM) works. Instead of just reading theory, you start with a pre-trained base LLM (the open-source Qwen3 model) and add reasoning capabilities to it yourself, step by step in code. The approach mirrors how large-scale reasoning models like DeepSeek R1 and GPT-5 Thinking are built.

The book covers three main techniques for improving LLM reasoning:

• Inference-time scaling — giving the model more time/computation to "think" before answering (e.g., chain-of-thought prompting, self-consistency, best-of-N sampling, self-refinement).
• Reinforcement learning — training the model to reason better through reward-based learning (specifically a method called GRPO).
• Distillation — taking a larger, more capable "teacher" model and training a smaller "student" model to mimic its reasoning, for efficiency.

What's inside

• 8 chapters + 7 appendices of code, mostly as Jupyter notebooks (.ipynb files). Each chapter folder contains the main code and exercise solutions.
• Bonus materials in several folders, including:
  • Scripts for evaluating on the MATH-500 benchmark
  • GRPO training scripts (basic and advanced)
  • Distillation data generation and training scripts
  • Downloadable training checkpoints
  • A chat interface (Appendix G)
  • MMLU evaluation methods, LLM leaderboards, and LLM-as-a-judge (Appendix F)
• A troubleshooting guide for common problems.

Hardware requirements

The main chapter code is designed to run on consumer hardware (no specialized servers needed) within a reasonable time. Chapters 2–4 work well on CPUs and GPUs. Chapters 5 and 6 recommend a GPU if you want to replicate the book's results. The code automatically uses a GPU if one is available.

Who it's for

This is for people who want to understand how reasoning LLMs actually work by building one themselves, rather than just using them as a black box. It's educational — the model you build is small but functional. If you want to first understand how a basic LLM is built from scratch, the author points to his previous book, Build a Large Language Model (From Scratch).

Important notes

• The repository mirrors a printed book, so the author cannot accept contributions that would change the main chapter code (to keep it consistent with the book). Feedback and questions are welcome via the Manning forum or GitHub Discussions.
• The README includes a citation (Chicago-style and BibTeX) for referencing the book in research.

Quick start

You can download the repository as a ZIP or clone it:

git clone --depth 1 https://github.com/rasbt/reasoning-from-scratch.git

Chapter 2 of the book provides setup tips for Python and coding environments.