colinlikescode/NanoGPT-Speedrun-Winner

What it solves

This project implements a highly optimized training pipeline for GPT-2 (124M) designed to achieve a specific validation loss (3.28) in the shortest time possible on a single 8xH100 GPU node.

How it works

The project improves upon previous speed records by implementing three primary technical optimizations:

1. Tail EMA of readout weights: Averaging the model over the final stage of training to increase accuracy, allowing the total training steps to be reduced from 1300 to 1288.
2. Lean ReLU²: Reducing memory traffic by storing only the squared MLP activation and reconstructing the input during the backward pass.
3. MLP Residual Optimization: Recovering gradients from a small identity instead of re-reading large tensors to save memory bandwidth.

Who it’s for

Researchers and engineers focused on LLM training efficiency, GPU kernel optimization, and those participating in the modded-nanogpt speedrun challenge.

Highlights

• Achieves a training time of 74.19–74.6s, surpassing previous records.
• Uses a paired referee system to ensure timing accuracy across 8xH100 GPUs.
• Leverages PyTorch 2.10, Flash-Attention 3, and Triton 3.6 for maximum performance.