karpathy/nanochat

What it solves

nanochat provides a minimal, hackable, and cost-effective harness for training Large Language Models (LLMs) from scratch. It removes the cognitive and financial complexity of LLM development, allowing users to handle the entire pipeline—tokenization, pretraining, finetuning, evaluation, and inference—on a single GPU node for under $100.

How it works

The project is built around a single "complexity dial" called --depth (the number of transformer layers). When a user sets the depth, the system automatically calculates all other optimal hyperparameters, including transformer width, number of heads, learning rate, and weight decays. It uses a custom Linear layer to manage precision explicitly (e.g., bfloat16 or float32) rather than relying on automatic casting, ensuring better control over VRAM and performance.

Who it’s for

• Researchers looking to experiment with scaling laws and micro-model performance.
• Developers who want a readable, "strong baseline" codebase to understand the end-to-end LLM lifecycle.
• Hobbyists wanting to train a GPT-2 grade model on a budget.

Highlights

• End-to-End Pipeline: Covers everything from BPE tokenization and pretraining to SFT, RL, and a CLI for chatting.
• Compute Optimal: Automatically tunes hyperparameters based on the model depth to ensure optimal performance.
• High Efficiency: Capable of training a GPT-2 grade model in ~1.5 hours on an 8XH100 GPU node.
• Minimalist Design: Avoids complex configuration objects or factories to remain easily forkable and readable.