qlabs-eng/noise

What it solves

Backpropagation is the standard way to train neural networks, but it requires a backward pass through the model. Noise provides a controlled benchmark to test alternative learning algorithms that never run a backward pass, using only forward passes to measure loss and update weights. It addresses the question of whether more general, search-based credit assignment can match or beat backprop.

How it works

Noise fixes the model, data, and scoring: a 37.7M-parameter transformer, a 30M-token training set derived from FineWeb, and cross-entropy on a validation set. There are three race tracks: (1) lowest validation loss after one epoch, (2) fastest time to reach a validation loss of 3.5, and (3) the same as track 1 but with the architecture free to modify. All entries must follow strict rules: no backpropagation, no autograd (forward or reverse mode), and no hand-written backward passes. Learning signals must come only from evaluating the model forward, and credit can only be assigned through at most one nonlinear step. Submissions are made via pull requests, and the leaderboard shows current records.

Who it’s for

Researchers and engineers interested in non-backpropagation learning algorithms, such as evolutionary strategies or other forward-only methods. It is also for anyone who wants to participate in a structured competition to improve such algorithms and see how they compare against established baselines.

Highlights

• Fixed transformer model and dataset for fair, apples-to-apples comparison across submissions.
• Three tracks: validation loss, speedrun to a target loss, and an open-architecture variant.
• Current record for track 1 and 2 is a validation loss of 3.409 achieved in 1 hour 43 minutes using a method called "Boost → Cut".
• Baseline backprop results are provided for reference: Adam with learning rate cooldown reaches 3.419, and ANVIL II on block matrices plus Adam elsewhere reaches 3.340. The goal is to beat these by a large margin.