qlabs-eng/slowrun

NanoGPT Slowrun – a benchmark for infinite‑compute, fixed‑data language‑model training

What it is – Slowrun is a community‑driven benchmark that measures how low a language‑model can push validation loss when the data budget is fixed (100 M tokens from the FineWeb dataset) but the compute budget is effectively unlimited. The idea is the opposite of the popular “speed‑run” contests that optimise for wall‑clock time on a fixed hardware budget. By removing the time constraint, researchers can experiment with heavy regularisation, large models, expensive optimisers, and algorithmic tricks that only pay off when you can spend many hours or days on the same data.

Why it matters – Most modern LLM research focuses on scaling data and compute together. Slowrun isolates the algorithmic side of the problem: given a modest, realistic data set, which training tricks actually improve generalisation when you can afford to train for a long time? The leaderboard records a series of incremental improvements (new activations, U‑Net style architectures, stochastic weight averaging, meta‑gradients, etc.) and provides reproducible scripts for each.

How it works

• Data: 100 M tokens from the publicly‑available FineWeb corpus.
• Tracks: Four compute‑time caps are defined so participants can compete at different resource levels:
  1. Limited – 1 hour on a single 8×H100 node (≈100× the compute of the original NanoChat 1‑epoch baseline).
  2. Tiny – 15 minutes on the same hardware.
  3. Two‑hour – 2 hours on the same hardware.
  4. Unlimited – No hard wall‑clock limit; only minimal hardware restrictions.
• Baseline: A 2.7 B‑parameter transformer trained with the Muon optimiser, dropout 0.1, and a very high weight‑decay (1.6). On the limited track this baseline reaches a validation loss of 3.402 in ~47 min.
• Leaderboard: Contributors submit pull‑requests that improve the validation loss for a given track. The README logs every world‑record entry, the script used, and a short description of the change (e.g., “add exclusive self‑attention (XSA)”, “use stochastic weight averaging”, “first‑order meta‑gradient on MLP matrices”).

Getting started

# 1. Clone the repo
git clone https://github.com/qlabs-eng/slowrun.git && cd slowrun

# 2. Install dependencies (Python 3, PyTorch, HuggingFace, wandb, etc.)
pip install -r requirements.txt

# 3. Download and prepare the 100 M token FineWeb subset
python prepare_data.py

# 4. Launch training (example: limited‑compute track)
#    Requires an 8‑GPU node; torchrun will spawn one process per GPU.
HF_TOKEN=… WANDB_API_KEY=… torchrun --standalone --nproc_per_node=8 train.py

• Replace train.py with the script from the track you want (tiny/train.py, two_hour/train.py, or unlimited/train.py).
• Set your Hugging Face token (HF_TOKEN) for dataset access and a Weights & Biases API key (WANDB_API_KEY) if you want experiment logging.

Key take‑aways from the leaderboard

Track	Best val‑loss (as of the README)	Notable tricks that got us there
Limited (1 h)	3.183	meta‑gradient on MLP matrices, exclusive self‑attention, MuonEq‑R, weighted checkpoint averaging
Tiny (15 min)	3.295	fp8 mixed‑precision, 2× recurrence, XSA, SWA, EMA
Two‑hour	3.139	Interleaved Head Attention, document‑level shuffling, context‑window scheduling
Unlimited	2.987	large ensembles, snapshot ensembles, gradient‑based model selection, extensive hyper‑parameter tuning

Who’s using it – The project is backed by members of the Q‑Labs engineering team and has attracted attention from Andrej Karpathy (who retweeted the announcement). Contributors are listed with their X handles; many are active AI researchers experimenting with novel optimiser or architecture ideas.

How to contribute – Fork the repo, modify the appropriate train.py (or add a new script), achieve a lower validation loss within the track’s time limit, and open a pull‑request. The maintainers will verify the run (using the provided HF token and wandb logs) and, if successful, add your entry to the world‑record table.

---

TL;DR – Slowrun is a living benchmark that lets anyone with access to a multi‑GPU H100 node (or a cloud equivalent) experiment with heavyweight training tricks in a data‑constrained regime. By submitting a PR that beats the current best validation loss for a track, you earn a spot on the leaderboard and help the community understand which algorithmic ideas truly scale when compute is the only limiting factor.