OpenAI GPT-5.6 Sol ARC-AGI-3 Benchmark Performance Optimization

OpenAI has demonstrated that the performance of GPT-5.6 Sol on the ARC-AGI-3 benchmark increased from 13.3% to 38.3% by enabling two specific API settings: retained reasoning and compaction. This finding highlights that benchmark scores often reflect the configuration of the evaluation harness rather than the inherent capabilities of the model alone.

Performance Gains on ARC-AGI-3

GPT-5.6 Sol achieved a score of 38.3% on the ARC-AGI-3 public set when using a harness powered by the Responses API with retained reasoning and compaction enabled. In contrast, the official ARC-AGI-3 harness resulted in a score of 13.3%.

These scores are measured using Relative Human Action Efficiency (RHAE), which compares model performance against a human baseline. For context, OpenAI estimates the average human tester scored 48% on the same tasks. The improvement represents a nearly threefold increase in performance while simultaneously reducing output tokens by 6x.

The Impact of Harness Design on Model Reasoning

The official ARC-AGI-3 harness uses a generic design to make model shortcomings more visible and ensure fair comparisons. However, OpenAI identified two primary limitations in this harness that hindered GPT-5.6 Sol's ability to learn and reason within 2D puzzle games:

Discarded Reasoning

In the official harness, all private reasoning was discarded after each game action. This forced the model to re-interpret the game state from scratch every turn, as it could not access the plans, insights, or thoughts that led to previous moves.

Rolling Truncation

To manage context limits, the official harness employs rolling truncation, discarding the oldest messages once the conversation exceeds 175,000 characters. This causes the model to lose memory of its earliest observations and actions, and can impair performance by keeping the context window consistently full.

Technical Solutions: Retained Reasoning and Compaction

To align the evaluation with how models are deployed in ChatGPT and Codex, OpenAI implemented a harness using the Responses API. This approach introduced two key technical improvements:

Retained Reasoning

By passing the previous response ID via the Responses API, GPT-5.6 Sol automatically retains its private reasoning across tool calls and turns. This allows the model to employ coherent strategies over time and reduces the time spent thinking before each action, as it no longer needs to re-analyze the game from the beginning of each turn.

Compaction

Replacing rolling truncation with compaction allows the model to better preserve learned information across longer runs. Unlike rolling truncation, which simply deletes old data, compaction manages the context more efficiently, allowing the model to maintain a higher score with fewer output tokens.

Recommendations for API Developers

OpenAI recommends that developers seeking to maximize model performance should adopt the settings used in their production environments. Specifically, they advise:

  • Use the Responses API instead of the legacy Chat Completions API.
  • Retain reasoning to allow the model to maintain a logical thread across interactions.
  • Use compaction to manage context windows without losing critical early-turn information.

Sources