HarnessTax Study Shows Harness Choice Mostly Affects Cost, Not Success, for Coding Agents

TL;DR

Choosing a coding‑agent harness can change the monetary cost of solving a task by up to five times while barely affecting the success rate, and a minimal open‑source harness (Pi) is often as effective as proprietary systems.

Overview of the HarnessTax Evaluation

The HarnessTax project evaluated 21 model–harness pairs covering seven language models (Claude, GPT‑5 series, Kimi K3, etc.) and three coding‑agent harnesses (Claude Code, Codex CLI, and the open‑source Pi). Each pair was run on 30 randomly sampled tasks from two public benchmarks—SWE‑bench Lite and Terminal‑Bench 2.0—with three repetitions per task to capture variability. Costs were measured in token dollars using a September 1 2026 price list, and success was measured with each benchmark’s official evaluator.

The study reports three key findings, each described in a self‑contained section below.

Finding 1: Harness Choice Primarily Impacts Cost, Not Correctness

Conclusion: The same model achieves nearly identical success rates across different harnesses, but token costs can differ by up to 5×.

  • Success rates vary by only ±2 % on SWE‑bench Lite and ±5 % on Terminal‑Bench 2.0 when swapping harnesses for a given model.
  • Cost ratios (geometric means) show Claude Code is about 2.0× more expensive than Pi and 1.6× more expensive than Codex on SWE‑bench Lite, and 1.5× more expensive than Pi on Terminal‑Bench 2.0.
  • Example: Claude Fable 5 solves 97.8 % of attempts in Claude Code, 96.7 % in Codex, and 96.7 % in Pi, yet Claude Code costs $1.33 per attempt versus $0.67 for Pi.

“Paying extra for essentially the same quality because the use of different harnesses is like paying a… Harness Tax 💰” – authors, HarnessTax.

Implication: Users who focus solely on success metrics may unknowingly incur a hidden “harness tax.” Evaluations should always compare cost‑adjusted performance across harnesses.

Finding 2: A Minimal Harness Can Be Competitive

Conclusion: The lightweight, open‑source Pi harness—providing only four tools (read, write, edit, bash)—reaches the Pareto frontier on both benchmarks.

  • Turn counts are similar across harnesses (e.g., Pi and Claude Code average ~15.4 vs. 15.3 turns for Fable 5 on SWE‑bench Lite), yet Claude Code’s per‑turn token usage is roughly double that of Pi.
  • Initial context size is a major cost driver: Claude Code’s first model call includes >10× more characters of instruction and tool schema than Pi, inflating token consumption before any model output.
  • Simpler harnesses reduce overhead without sacrificing success, suggesting that rich, proprietary harness features are not strictly necessary for many tasks.

“The effectiveness of Pi and Codex demonstrates opportunities for open‑source harness research with existing models.” – authors, HarnessTax.

Implication: Developers can achieve cost‑effective coding assistance by building or adopting thin harnesses, especially when security or alignment constraints are handled elsewhere.

Finding 3: Models Often Perform Better Outside Their Native Harness

Conclusion: Provider‑specific harness optimizations do not guarantee the best model‑harness pairing; alternative harnesses achieve the highest success in 9 of 12 cross‑model comparisons.

  • Anthropic’s Claude models performed comparably—or better—in Codex CLI and Pi, despite being tuned for Claude Code.
  • OpenAI’s GPT‑5.6 Sol achieved 83.3 % success in Pi versus 78.9 % in Codex on Terminal‑Bench 2.0, at ≈50 % of the cost.
  • Sonnet 4.6 solved 68.9 % in Codex versus 66.7 % in Claude Code on SWE‑bench Lite, with similar cost.

“A model’s capabilities are compatible, generalizable and can carry over to other harnesses.” – authors, HarnessTax.

Implication: Practitioners should experiment with multiple harnesses for a given model rather than assuming the provider’s default is optimal.

Community Reactions on Hacker News

  • Cost vs. robustness: Several commenters note that while Pi is cheap, Claude Code and Codex may be more robust against malformed tool calls due to richer system prompts.
  • Benchmark limitations: Users point out that the two open‑source benchmarks may favor models that have seen similar data during training, and that real‑world workloads could show different cost‑accuracy trade‑offs.
  • Tool‑call compatibility: One comment emphasizes that using the native tool signatures (e.g., Edit for Claude, apply_patch_call for GPT) can affect performance more than the harness itself.
  • Security considerations: Some argue that extra context in proprietary harnesses often includes safety instructions, which, while costly, may be required for production use.
  • Future directions: The discussion highlights interest in concurrent harness designs, dynamic harness selection, and standardized benchmarking that captures both token cost and robustness.

Practical Takeaways for Developers and Researchers

  1. Measure cost, not just success. When evaluating a coding agent, record token dollars alongside pass/fail metrics.
  2. Start with a thin harness. Pi’s four‑tool design shows that minimalism can be sufficient for many tasks; add complexity only if you need specific safety or tooling features.
  3. Test cross‑harness pairings. Even if a model is marketed with a proprietary harness, run it through alternatives (e.g., Claude models in Codex CLI) to discover cheaper, equally effective configurations.
  4. Watch the initial prompt size. Large system prompts can dominate cost; trimming unnecessary instructions and tool schemas yields immediate savings.
  5. Expect variability across workloads. The reported results apply to SWE‑bench Lite and Terminal‑Bench 2.0; different domains (e.g., scientific code, large‑scale refactoring) may exhibit different cost‑accuracy curves.

Future Research Directions

  • Benchmark expansion: Create a broader suite of tasks, including long‑horizon and multi‑session workflows, to test harnesses beyond the current open‑source benchmarks.
  • Automated harness selection: Develop meta‑agents that dynamically choose the most cost‑effective harness based on task characteristics and real‑time feedback.
  • Security vs. cost trade‑offs: Quantify how safety‑related context (e.g., alignment prompts) impacts both cost and failure modes.
  • Open‑source harness ecosystems: Encourage community contributions to lightweight harnesses, fostering interoperability across providers and models.

Citation

If you use the HarnessTax results in your work, please cite:

@misc{pan2026harnesstax,
  title  = {{HarnessTax: How Much Does Harness Matter for Coding Agents?}},
  author = {Pan, Melissa Z. and Yang, Shuo and Arabzadeh, Negar and Chiang, Wei-Lin and Stoica, Ion and Zaharia, Matei},
  year   = {2026},
  url    = {https://harnesstax.github.io/},
}

Sources

Related