Claude Fable 5 vs. GPT‑5.6 Sol on KIRO NP‑Hard Optimization: /goal’s Mixed Impact

Claude Fable 5 vs. GPT‑5.6 Sol on KIRO NP‑Hard Optimization: /goal’s Mixed Impact

TL;DR

Claude Fable 5 solved the unpublished KIRO NP‑hard fiber‑network design problem better than GPT‑5.6 Sol, and the native /goal flag helped in a majority of individual trials but increased the overall mean score for both models.

The KIRO Benchmark

The KIRO problem models fiber‑network design for three French cities. A valid solution must:

  • Connect every tower exactly once.
  • Form redundant loops rooted at distribution hubs.
  • Attach short branches to towers on those loops.
  • Minimize total cable length (lower score is better).

Search‑space magnitude

Even a crude lower bound—assigning each of the 532 terminals in Paris to one of 11 hubs—yields 11^532 possibilities. A tighter bound using a restricted family of solutions (19 loops of 28 terminals each) gives:

(532! / 19!) × 11^19 ≈ 10^1223

Thus exhaustive search is infeasible, making the task a genuine stress test for LLM‑driven solvers.

Experimental Setup

Parameter Value
Models Claude Fable 5, Opus 4.8, Sonnet 5; OpenAI GPT‑5.6 Sol, Terra, Luna
Modes Plain (no hint) vs. native /goal
Budget per run 30 minutes of wall‑clock time
Outer‑agent timeout 1,900 seconds
Reasoning Maximum available for each model
Execution environment Harbor 0.1.43, Docker, subscription authentication

All code, prompts, and raw logs are available in the [CLIArena](https://github.com/charles-azam/CLIArena) repository.

Primary Results

The author first ran a single 30‑minute no‑hint pair for each of the six models. The headline comparison focused on the two flagship models, repeating the experiment three times for each.

Clean score distributions

Clean score distributions

Model Run Plain /goal Δ (/goal − Plain)
Fable 5 1 32,197 31,934 ‑263
Fable 5 2 32,516 32,324 ‑192
Fable 5 3 32,446 35,178 +2,732
GPT‑5.6 Sol 1 33,581 39,371 +5,790
GPT‑5.6 Sol 2 35,539 32,703 ‑2,836
GPT‑5.6 Sol 3 33,663 33,313 ‑350

Interpretation

  • /goal produced a lower (better) score in four of six runs, giving it a 66 % win‑rate on individual trials.
  • However, the mean score for each model worsened when /goal was enabled:
    • Fable 5: plain mean = 32,386, goal mean = 33,145 (+759 worse).
    • GPT‑5.6 Sol: plain mean = 34,261, goal mean = 35,129 (+868 worse).
  • The median effect was modestly positive for Fable 5 (‑192) and negative for Sol (‑350), indicating that large regressions in a few runs dominate the average.

Relative strength of the models

  • Fable 5 plain outperformed Sol plain by 1,875 points on average.
  • Fable 5 /goal beat Sol /goal by 1,984 points.
  • Fable 5’s scores were tightly clustered (range ≈ 319 points) whereas Sol’s plain scores spanned 1,958 points, highlighting Fable’s consistency.

How /goal Works Under the Hood

The same command hides two distinct implementations.

Claude Code (Fable 5)

  • Implements /goal as a session‑scoped stop hook.
  • After each turn, a small evaluator model (default Haiku) reads the transcript and decides whether the goal is satisfied.
  • The evaluator cannot access tools or files; it judges only the evidence present in the conversation.
  • Documentation: Anthropic goal docs.

OpenAI Codex (Sol)

  • Persists the goal as thread state in SQLite.
  • The working model receives create_goal, get_goal, and update_goal tools, allowing it to read/write the objective.
  • If the thread becomes idle, Codex injects a continuation turn with the objective and an audit of completion.
  • Source references: thread goal actions, tool spec.

Key difference: Claude delegates the decision to an independent evaluator that sees only the transcript, while Codex lets the working model self‑assess using persisted state and tool access.

Why /goal Can Win Most Runs Yet Harm Averages

On typical coding tasks, progress is incremental and visible; extra turns often fix a failing test. In a hard optimization problem, time amplifies the quality of the chosen search path:

  • If the agent has already selected a promising solver, additional time can deepen the optimum (positive tail).
  • If the agent has locked onto a poor strategy, extra time magnifies the mistake (negative tail).

The KIRO runs illustrate both scenarios:

  • /goal helped when Fable 5’s fast compiled portfolio or Sol’s successful chain repartition continued to improve.
  • /goal hurt when Fable 5 built a slow solver or Sol committed to an exhaustive anchor sweep, causing the score to deteriorate sharply. Thus the median moves slightly in the right direction, but the worst‑case regressions dominate the mean.

Limitations of the Study

  • The benchmark is a single unpublished NP‑hard task; results may not generalize to other domains.
  • Only Fable 5 and Sol have three clean, matched runs; other models were tested once each.
  • Experiments ran sequentially on subscription services, which could introduce drift.
  • Containers exposed eight CPUs despite metadata declaring one, potentially favoring Fable’s parallel portfolios.
  • All outputs were validated by the wrapper, so the scores reflect the entire system (model, CLI, prompt, subscription, harness).

Reproducing the Benchmark

The full repository, including wrappers, analysis scripts, figure generators, and a detailed memo, is in [CLIArena](https://github.com/charles-azam/CLIArena). Raw job directories are omitted due to size, but the memo records every score, city breakdown, elapsed time, strategy, exclusion, and run ID.

Primary commands used:

RUN_ID=article-kiro-YYYYMMDD-clean \
PHASE=nohint-all \
./scripts/run_subscription_article_matrix.sh

uv run python scripts/summarize_subscription_article_results.py $RUN_ID …
uv run python scripts/analyze_subscription_article_results.py $RUN_ID …

Community Reactions (HN Comments)

  • tyleo noted that the chart’s y‑axis is inverted, making higher visual bars correspond to lower (better) scores.
  • theptip suggested that “ultra mode” (parallel investigators, adversarial review) could be a superior alternative to /goal for search‑heavy tasks.
  • Tenoke observed that Claude sometimes forgets long‑term instructions, implying /goal might aid memory in short sessions.
  • sreekanth850 contrasted Anthropic’s speed with OpenAI’s Codex, claiming Codex feels faster and more efficient for large codebases.
  • enraged_camel warned that GPT‑5.6 Sol’s increased “relentlessness” can lead to unsafe actions, requiring tighter supervision.
  • andai argued the results look noisy because each model was evaluated only once per configuration.
  • akoboldfrying suggested formulating KIRO as an integer linear program and solving it with Gurobi for a lower bound.
  • spikk requested a time‑series plot of the best score over the run to better visualize /goal’s effect.

Bottom Line

  • Claude Fable 5 demonstrates superior raw intelligence and consistency on a challenging NP‑hard fiber‑network design problem.
  • The native /goal flag can improve individual runs but degrades average performance because it sometimes prolongs a poor search trajectory.
  • Understanding the underlying implementation (independent evaluator vs. persisted thread state) is crucial when interpreting /goal’s behavior.
  • For hard optimization tasks, a persistence feature like /goal should be used judiciously, and alternative search‑enhancement modes (e.g., ultra parallelism) may yield more reliable gains.

Sources