mdlARC: Achieving 44% on ARC-AGI-1 with High Sample Efficiency

Overview

A small transformer model trained from scratch at test time has achieved a 44% score on the ARC-AGI-1 benchmark with a total lifetime compute cost of 67 cents. This result demonstrates that high sample efficiency and abstract reasoning capabilities can be achieved using a simple autoregressive transformer architecture without relying on massive pre-training, synthetic data, or complex recursive loops.

Technical Implementation

The model employs a test-time training (TTT) approach where a small transformer is trained from scratch on both the training set and evaluation set puzzles (with test labels hidden) for approximately 1.5 hours on a single NVIDIA RTX 5090.

Architecture and Training

• Model Structure: The model uses a modern transformer architecture with 8 layers, replacing GELU with SwiGLU and LayerNorm with RMSNorm.
• Representations: The system uses 3D RoPE (Rotary Positional Embeddings) and a separate learned additive embedding for each puzzle to enable cross-task learning.
• Training Objective: The model is trained supervisedly, meaning the loss function only includes output tokens rather than both input and output tokens. This change improved performance from 40% to 44%.
• Optimization: The model utilizes the NorMuon optimizer, which solved convergence issues observed with vanilla Muon, and a WSD (warmup, hold, linear decay) learning rate schedule.
• Data Augmentation: Input test data is augmented with color and dihedral permutations. The two most common outputs produced from these augmented inputs are submitted.

Performance and Ablations

Experimental results indicate that representation is the primary driver of performance. Removing either the 3D RoPE or the per-task embeddings caused scores to drop steeply to approximately 25%.

Ablation	Score
Full Model	44%
No 3D RoPE	~24%
No Per-Task Embeddings	~24%
Training on Inputs	~39%
ARC-1 + ConceptARC only	~40%
1D RoPE instead of 3D	~24%
CompressARC style (unsupervised, per-task)	~18%

Analysis of ARC-AGI Benchmarking

The author argues that current LLM-based approaches to ARC-AGI are often misleading due to a reliance on synthetic data and massive offline pre-training, which may lead to "benchmaxxing" rather than the development of general abstract reasoning.

The Role of Synthetic Data

Synthetic data is criticized for lowering the bar of fluid intelligence required to solve puzzles. The author proposes banning offline pre-training and requiring models to train from scratch after submission to ensure no synthetic data is used and to make comparisons fair across different model types.

Transductive Reasoning and TTT

There is a significant debate regarding the use of evaluation puzzle inputs during training. The author defends this as "transductive reasoning," arguing that in a meta-learning benchmark, the AI should be allowed to learn from the available context of the evaluation puzzles, provided the final test labels remain hidden.

Comparison with Other Approaches

Recursion vs. Simple Transformers

While recent high-performing models like TRM and HRM emphasize recursive loops and deep supervision, the mdlARC results suggest that similar performance can be reached without recursion. The author posits that the benefit of recursion may be primarily in increasing compute without increasing memory movement.

LLM Limitations

Observations of LLM progress on ARC-AGI suggest that performance gains are largely driven by post-training on synthetic data. Evidence for this includes the fact that base models often remain at single-digit scores on ARC-2, and that progress on ARC-2 often mirrors the amount of synthetic data available rather than an increase in general reasoning capabilities.

Community Insights and Counterpoints

Discussion among researchers and practitioners highlighted several key points regarding the validity and utility of this approach:

"The '67 cents' part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results."

"If the exam was adjusted so that you can only look at one question at a time, you won't get 44% anymore."

Critics also pointed out that training on all test tasks simultaneously may be an unrealistic representation of real-world problem solving, although the author notes that this is consistent with how the benchmark organizers compare models like TRM.