KellerJordan/modded-nanogpt

NanoGPT (124M) in 90 seconds

What it solves

It aims to find the fastest possible algorithm and system configuration to train a language model to a specific performance target (3.28 cross-entropy loss on the FineWeb validation set) using 8 NVIDIA H100 GPUs.

How it works

The project operates as a "speedrun," iteratively applying and testing a vast array of architectural, optimization, and systems-level techniques to reduce training time. It evolved from Andrej Karpathy's llm.c and NanoGPT, moving from a 45-minute baseline to under 3 minutes. Key methods include:

  • Architecture Changes: Implementing Rotary embeddings, QK-Norm, ReLU², and skip connections from embeddings to every block.
  • Optimizer Innovations: Utilizing the Muon and NorMuon optimizers, and the ANVIL optimizer with twin-rail momentum.
  • Precision & Hardware: Using FP8 for the MLP forward/backward passes, QKV projections, and the head, alongside CUDA-graph capture and Flash Attention 3.
  • Data & Training Logic: Aligning batch starts with End-of-Sentence (EoS) tokens, implementing multi-token prediction, and using a batch size schedule.

Who it’s for

Researchers and engineers interested in extreme LLM training efficiency, hardware acceleration, and the frontier of optimizer and architecture design.

Highlights

  • Reduced training time from 45 minutes to under 3 minutes on 8xH100 GPUs.
  • Reduced token requirement from 10B to under 330M tokens to reach the target loss.
  • Implements a wide variety of cutting-edge techniques like Polar Express, sparse attention gates, and hashed bigram/trigram embedding tables.

Related

  • Project
    qlabs-eng/slowrun

    NanoGPT Slowrun is a benchmark that evaluates language‑model training when the data is fixed (100 M tokens) but compute is effectively unlimited. Four tracks (15 min, 1 h, 2 h, unlimited) let participants experiment with heavy regularisation, large models, and exotic optimisers. The repo provides data‑prep scripts, training code for each track, and a world‑record table that logs every improvement (e.g., swiglu, U‑Net, meta‑gradients, ensembles). To try it, clone the repo, install the Python requirements, run `prepare_data.py`, then launch training with `torchrun` on an 8‑GPU H100 node. Submit a PR with a lower validation loss to claim a spot on the leaderboard.

  • Project
    cfregly/ai-performance-engineering

    A comprehensive collection of code and resources for optimizing AI systems, focusing on GPU tuning, distributed training, and high-throughput inference scaling.

  • Project
    colinlikescode/NanoGPT-Speedrun-Winner

    An optimized GPT-2 training implementation that uses memory and weight averaging techniques to set a new speed record for the modded-nanogpt challenge.

  • Project
    ServiceNow/Fast-LLM

    A high-performance library for training large language models that uses 3D parallelism and optimized kernels to reduce training time and cost.

  • Dispatch
    Training Language Models with Megatron-LM

    Hugging Face provides a guide on using NVIDIA's Megatron-LM framework to efficiently pre-train large language models on GPUs, including integration with the Transformers library.