ARC-AGI-3: Solving the Benchmark With No Instructions

The Core Challenge of ARC-AGI-3

ARC-AGI-3 transforms the previous static grid puzzles of the ARC benchmark into an interactive and agentic environment. Unlike its predecessors, the model is not given a set of rules or a goal; it must discover the objective and the mechanics of the world through raw frames and interaction. The primary difficulty lies in the interplay between exploration (discovering the rules) and exploitation (solving the level), while maintaining extreme action efficiency.

Action Efficiency vs. Brute Force

While early preview competitions were won using brute-force methods—specifically searching for actions that resulted in frame changes—the full ARC-AGI-3 benchmark is designed to resist such approaches.

The Failure of Brute Force

• Action Space: The action space is enormous, featuring over 4,000 possible actions (including a 64x64 mouse-click grid), making random search computationally intractable.
• Efficiency Scoring: The benchmark uses a scoring system based on the ratio of human baseline actions to AI actions. If an agent is significantly less efficient than a human, its score drops toward zero, even if it eventually solves the level.
• Hardened Environments: Newer games include timer bars that move even when an action is valid but does not change the game state, neutralizing simple "frame-change" detection strategies.

The Role of LLMs and High-Level Abstractions

Tufa Labs leverages Large Language Models (LLMs) not as direct action predictors, but as reasoning engines within a specialized harness.

Induction and Transduction

• Transductive Methods: Directly predicting actions from input frames as context. This approach generally fails to generalize well.
• Inductive Methods: Using chain-of-thought reasoning in English to create a rationale for the game's mechanics. This allows the agent to identify objects and dynamics, which can then be cross-applied to future levels.

The "Abstraction Mountain"

Humans solve ARC-AGI-3 by leveraging deep priors (e.g., recognizing a "maze" or a "player"). LLMs possess "fractured, entangled representations" of these concepts from their pre-training on the internet. While these representations are not as clean as formal symbolic logic, they allow LLMs to skip levels of abstraction that a pure reinforcement learning (RL) model would have to learn from scratch.

Language as a Shortcut

Language serves as a critical bootstrap for intelligence in this benchmark. Tufa Labs found that representing game states in language (e.g., using characters like 'B' for blue) helps the model lean on its pre-training priors, whereas using raw numbers or stripped-down representations significantly degrades performance.

Agency and Planning

ARC-AGI-3 tests "agency," defined as the ability to acquire goals, plan, and realize them in a dynamic environment.

Two Types of Planning

1. Path Planning: Once the rules are understood, the agent must plan a path to the goal. This is handled by the LLM writing and executing Python code (e.g., using breadth-first search) to find the optimal path.
2. Goal Acquisition: The agent must figure out how to figure out the rules. This involves balancing exploration and exploitation, a process the Tufa team describes as "simulated planning" where the transformer pretends to plan by iterating through hypotheses.

Goal Loops and Failure Modes

Agents often fall into "wrong-goal loops," where they lock onto a false hypothesis (e.g., believing the goal is to reduce an energy bar to zero) and cannot escape that logic, even when it fails to produce a win.

Engineering the Solution: Harnesses and Requirements

Because frontier models score poorly (under 1%) without guidance, Tufa Labs uses a "harness" to provide general thinking patterns.

Requirements-Based Engineering

To manage the increasing complexity of the codebase—which is often written by coding agents—the team employs requirements-based engineering. They formally write and review requirements and tests, then hand them to coding agents for implementation. This prevents "understanding debt," where the human developers lose sight of how their own system functions.

Reward Shaping

To improve the agent, the team uses reward shaping based on:

• Level transitions.
• ARC-AGI scores (efficiency).
• Whether generated code executes successfully.
• The length of reasoning steps to optimize token usage.

AGI and the "Bitter Lesson"

The Tufa team discusses the tension between the "Bitter Lesson" (the idea that general methods like scaling and compute always win over hand-crafted heuristics) and the need for specialized harnesses.

• The Bet: The team believes the winning solution for ARC-AGI-3 will not be a purely "bitter lessened" solution. They argue that the current state of models requires a level of basic design and structural guidance to handle the abstraction and efficiency requirements of the benchmark.
• The AGI Question: Solving ARC-AGI-3 does not prove AGI, but failing it suggests a system is not yet AGI. The team notes that even humans struggle to score 100% due to the inherent exploration required in novel games.