OpenAI GPT-6 Astra Performance on ARC-AGI-3 Benchmark

OpenAI's GPT-6 Astra has achieved state-of-the-art results on the ARC-AGI-3 benchmark, demonstrating a significant leap in agentic intelligence. Using a specialized Provider Adapter harness, the model reached a score of 99.9% on the Semi-Private test set, while surpassing the median human baseline in action efficiency on 96% of the levels it completed.

ARC-AGI-3: Measuring Agentic Intelligence

ARC-AGI-3 is a benchmark designed to measure the "residual gap" between current AI and Artificial General Intelligence (AGI). Unlike previous iterations, ARC-AGI-3 focuses on agentic capabilities in novel, abstract, turn-based environments where agents must explore and solve puzzles without explicit instructions.

The benchmark evaluates four primary components of intelligence:

• Exploration: Actively interacting with the environment to gather information.
• Modeling: Converting raw observations into generalizable models to predict future states.
• Goal-setting: Identifying target states using only sparse rewards.
• Planning and Execution: Mapping a path to the goal and course-correcting based on new data.

GPT-6 Astra Performance and Cost

GPT-6 Astra demonstrated high performance across different reasoning effort levels and harness configurations. The results show that higher reasoning effort often leads to lower total costs because the model solves games in fewer actions, reducing the number of required model calls.

Performance by Harness

• Standard Harness: This provider-neutral interface requires the model to manage its own visible notes. In this configuration, Astra (max) scored 62.7% on the Semi-Private set at a cost of $26,098.
• Provider Adapter Harness: This harness preserves opaque reasoning state between requests and uses compaction for long conversations. In this configuration, Astra (high) achieved a score of 99.9% at a cost of $18,817.

Reasoning Effort Comparison

Reasoning effort	Standard harness	Provider Adapter harness
max	62.7%, $26,098	98.6%, $17,332
xhigh	59.3%, $37,317	98.4%, $18,147
high	54.8%, $40,705	99.9%, $18,817
medium	38.6%, $48,090	98.4%, $19,285
low	17.5%, $38,166	98.0%, $21,298
none	35.2%, $49,791	96.7%, $23,457

Key Behavioral Insights

Analysis of Astra's replays revealed three distinct capabilities that contribute to its high performance:

1. Custom Symbolic World Models

Astra develops an on-the-fly algebraic shorthand to track game state and plan actions. Rather than using a full programming language, it creates compact, information-dense notes to record coordinates, rules, and sequences. For example, it may record a state as L8: hub q2 (8↓). Lengths: 14=1… or a plan as extend8 to3; retract10 to2; shorten8 to1.

2. Human-Level Action Efficiency

Astra (max) using the Provider Adapter harness used fewer actions than the median human on 96.0% of levels and required 51.7% fewer actions per level on average. This indicates that once the model understands the environment's mechanics, it executes solutions with efficiency that matches or exceeds human performance.

3. Autonomous Tool Creation

When evaluated in the PRO-LONG harness—which provides a code execution sandbox—Astra created custom software libraries for specific games. In a maze-like game (tu93), Astra developed separate Python scripts for navigation (maze_solver.py), combat rules (combat_solver.py), and patrol modeling (patrol_solver.py), using a synchronization script (sync_state.py) to verify predictions against observations.

Discussion and Limitations

While the results are a major milestone, the ARC Prize team emphasizes that saturating this benchmark does not constitute proof of AGI. The environments are deterministic and closed-ended, which does not reflect the open-ended complexity of the real world.

Community discussion on Hacker News highlighted several critical perspectives:

"Is solving a snake like puzzle game in the least number of moves really what defines intelligence?"

"Was OpenAI able to run ARC-AGI-3 tests previously so that they could build a custom harness for the specific tests in the set?"

Other observers noted the extreme cost of these runs (approximately $360 per puzzle) compared to human labor, though some argued that if price-performance trends continue, AI costs will eventually drop below human minimum wage.

Additionally, external data suggests that while Astra excels at ARC-AGI-3, it still struggles with highly complex mathematical problems. According to reports on the FrontierMath-Erdos benchmark, GPT-6 Astra solved only 5 of 68 problems, with some solutions requiring over $220,000 in compute.