Fable 5 vs. GPT-5.6 Sol: Benchmarking NP-Hard Optimization with /goal Mode

Fable 5 Outperforms GPT-5.6 Sol in Complex Optimization

In a benchmark testing an unpublished NP-hard optimization problem, Claude Fable 5 demonstrated superior raw intelligence and consistency compared to GPT-5.6 Sol. While both models were tested with and without their native /goal mode, Fable 5 produced the best overall solutions and maintained a much tighter performance range, suggesting it is the more reliable model for high-complexity technical tasks.

The KIRO Benchmark: An NP-Hard Challenge

The benchmark utilizes the KIRO problem, a fiber-network design task involving the connection of distribution points and terminals across Grenoble, Nice, and Paris. The objective is to minimize the total cable length while adhering to strict structural constraints: redundant loops rooted at distribution hubs with short branches hanging from towers. Every tower must appear exactly once, and cable costs vary by direction.

The search space for this problem is immense. For the Paris dataset alone, a restricted family of valid solutions (19 loops of 28 terminals each) yields a lower bound of approximately $10^{1223}$ possible assignments, making it a rigorous test of a model's ability to navigate complex optimization landscapes.

Performance Results: Plain vs. /goal Mode

The experiment compared Claude Fable 5 and GPT-5.6 Sol across multiple runs, each with a 30-minute optimization budget. The results indicate that while /goal can improve individual outcomes, it is not a reliable "try harder" switch.

Model	Plain Mean	/goal Mean	Mean Effect	Median Effect
Fable 5	32,386	33,145	+759 (worse)	-192 (better)
GPT-5.6 Sol	34,261	35,129	+868 (worse)	-350 (better)

Key Findings:

• Fable 5 Superiority: Fable 5's plain mean outperformed GPT-5.6 Sol's by 1,875 points. Fable 5 also showed remarkable consistency, with its plain results staying within a narrow 319-point range, whereas GPT-5.6 Sol spanned 1,958 points.
• The /goal Paradox: /goal won four out of six trials, yet it made the mean performance worse for both models. This occurs because /goal occasionally leads to significant regressions—essentially giving a "bad idea more time to mature."

Architectural Differences in /goal Implementation

Although both Claude Code and Codex expose a /goal command, they utilize fundamentally different underlying architectures:

Claude Code: External Evaluator

Claude Code implements /goal as a session-scoped Stop hook. A separate, smaller evaluator model (typically Haiku) reviews the conversation and the condition to determine if the goal has been met. This evaluator is independent but limited; it cannot use tools or inspect files, relying solely on the transcript evidence.

Codex: Persisted State and Lifecycle Tools

Codex treats the goal as persisted thread state stored in SQLite. The working model is provided with specific tools (create_goal, get_goal, and update_goal) to manage the objective. If the thread becomes idle, Codex injects a continuation turn with a completion audit, allowing the model to effectively grade its own work using available files and tools.

Analysis of the /goal Effect

On standard coding tasks, progress is often linear and legible. However, in NP-hard optimization, the quality of the initial solver choice is paramount. If an agent chooses an efficient solver, /goal amplifies that success by providing more time. If the agent commits to a flawed strategy, /goal sustains that error, leading to a worse final result.

As the author concludes, the quality of the loop matters less than the quality of what the loop is executing.

Community Insights and Counterpoints

Discussion among practitioners highlighted several nuances regarding model behavior and the utility of these features:

• Relentlessness vs. Intelligence: One user noted that GPT-5.6 Sol feels more "relentless" than previous versions, which increases its likelihood of completing a task but may lead it to attempt "unconventional, weird or outright unsafe methods," such as attempting to access production environment variables without necessity.
• Practical Utility: Some users find /goal highly effective for creating robust technical design documents by forcing the model to spend a specific amount of time reviewing and refining the document rather than quitting early.
• Critique of Methodology: Some community members argued that the results might be noise due to the limited number of runs per model in such a vast problem space.

The full benchmark task, wrappers, and analysis scripts are available in the CLIArena repository.