FrontierHarness Eval: Benchmarking AI Coding Harnesses

The Impact of Harness Selection on AI Coding Performance

FrontierHarness Eval demonstrates that the "harness"—the execution environment, toolset, and orchestration layer surrounding an LLM—is a critical determinant of performance and cost. In a controlled test using the same model (Kimi K3) across 12 different configurations, the pass rate for software engineering tasks varied from 50% to 66.7%, while the median cost per task ranged from $1.05 to $18.34, representing a 17x difference in expenditure for the same underlying intelligence.

Performance and Cost Leaderboard

The evaluation tested 9 harnesses across 12 configurations on identical software engineering tasks. The results highlight a sharp divergence between quality, cost, and speed.

Quality and Pass Rates

Codex emerged as the quality leader with a 66.7% pass rate, followed by Claude Code and DSH Creator, both at 63.3%.

  • Highest Pass Rate: Codex (66.7%)
  • Mid-Range: Pi, DSH PTC, and DSH Standard (60.0%)
  • Lowest Pass Rate: Hermes and OpenCode (50.0%)

Cost Efficiency

Cost varies significantly based on the harness's efficiency in managing turns and tool calls. Exo Harness is the overall cost leader for total task expenditure, while OpenCode shows the lowest median cost per successful task.

  • Lowest Median Cost per Task: Exo Harness ($1.0452)
  • Highest Median Cost per Task: Claude Code ($18.3368)
  • Lowest Median Cost per Successful Task: OpenCode ($0.0615)

Execution Speed

Speed is measured by the median time to complete a successful task. DSH Minimal is the fastest configuration.

  • Speed Leader: DSH Minimal (5m 41s)
  • Slowest: Claude Code (9m 38s)

Technical Methodology

To ensure the results were grounded in the harness's capabilities rather than model variance or environment noise, the evaluation employed a strict control protocol:

  • Model Consistency: Every run used the Kimi K3 model.
  • Environment Isolation: All 360 trials started from a fresh checkpoint restore on Runta, ensuring identical vCPU, memory, disk size, disk contents, and memory state.
  • Cold Start: Every run began with a fresh restore to prevent warm-cache bias.
  • Focus: The benchmark specifically targets software engineering contexts and terminal-based tasks.

Key Technical Insights

The Divergence of Quality and Cost

High pass rates do not linearly correlate with lower costs. For example, Claude Code achieved a high pass rate (63.3%) but incurred the highest cost per task ($18.34), suggesting a more exhaustive or expensive orchestration strategy that succeeds more often but at a higher price point.

Cache Hit Rates vs. Actual Cost

While Codex and Kimi Code showed the highest median cache hit rates (88.0%), the evaluation notes that cache hit rate is not a direct proxy for cost. A long, failing sequence of 300 turns that is cached can still be more expensive than a short sequence that misses the cache.

Community Analysis and Counterpoints

Discussion among technical users on Hacker News highlighted several caveats and potential biases in the benchmark:

  • Model-Harness Coupling: Some users argued that testing only against Kimi K3 might skew results. One user noted that some harnesses are specifically optimized for the quirks of certain models (e.g., Kimi's tendency for tool-call loops), and a harness that excels with Kimi may not perform as well with Claude or GPT.
  • Statistical Significance: Critics pointed out that with only 30 samples per harness, the confidence intervals for accuracy may be too wide to definitively rank the top performers.
  • Median vs. Mean: Some observers noted that using median cost rather than mean cost may understate the actual financial impact, as a few extremely expensive failing runs can significantly pull the mean upward.
  • Harness Bloat: The high performance of minimal harnesses like Pi (60% pass rate) led some to suggest that many complex harnesses may be unnecessarily bloated.

"As models become commodities, the harness will be the next optimizing game."

"Testing it against Kimi is potentially skewing the numbers massively... Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that."

Tested Harness Versions

Harness Version
Codex v0.148.0
DeepSeek Harness (DSH) v0.1.0-rc.8
Claude Code v2.1.237
Pi v0.84.2
Oh My Pi v17.4.0
Kimi Code v0.37.2
Exo Harness v0.1.0
OpenCode v1.18.19
Hermes v0.20.4

Sources

Related