An Empirical Study of Harness Design for Coding Agents

Executive Summary

Coding agent performance is not determined by the underlying Large Language Model (LLM) alone, but by the the interaction between the model and its "harness"—the system of planning, action space, and context management that wraps the model. Research indicates that there is no universal "best" harness; instead, the optimal configuration depends on the model's native capabilities, the specific task type, and the available resource budget.

The Role of Context Management

Context management is most critical when working with tight context-window budgets. Its primary value lies in preventing context-overflow failures, which allows agents to maintain longer execution trajectories without fundamentally changing their behavior.

Key findings regarding context management include:

Budget Sensitivity: The benefit of context management is most pronounced when the context window is small. For example, the gap in success rates between managed and unmanaged contexts can be as high as 35.7 points at a 32k window, dropping to 2.7 points at 128k.
Optimal Strategy: Staging rule-based elision (removing unnecessary parts of the context) before LLM-based summarization provides the highest overall efficiency.
Recoverability: Adding mechanisms to make elided content recoverable does not yield accuracy gains, as models rarely utilize these features.

Planning as a Scaffold vs. Cost Saver

Planning components shift in utility as model strength increases. For weaker models, planning acts as an accuracy scaffold that prevents them from abandoning tasks too early. For stronger models, planning primarily serves as a cost-saving measure by reducing redundant verification steps, with little to no impact on final accuracy.

This trend is mirrored in industry practice; for instance, some newer frontier models have seen the removal of built-in task-tracking tools (like TaskCreate or TodoWrite) because the models' native reasoning capabilities have reduced the need for explicit in-session planning tools.

Action Space and Tooling Proficiency

The choice between predefined structured tools and a raw bash-only interface depends on the model's proficiency with shell commands.

Bash-Capable Models: Models with high bash proficiency operate more effectively and at a lower cost using a bash-only interface. This is especially true for command-line-centric tasks, as it allows the model to combine multiple code modifications into a single tool call.
Weaker Models: Predefined tools improve success rates for models that lack strong bash control, providing a necessary structure that compensates for the model's lack of shell proficiency.

Conclusion: Harness Design as a Conditional Systems Problem

The study concludes that harness design should be treated as a conditional systems problem. Rather than adopting a default set of components, developers should select components based on the target model, the task type, and the resource budget.

Synthesis of Community Insights

Industry practitioners and researchers have highlighted several nuances regarding these findings:

"If a Car A is performing better... then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body... Your harness can adapt to the underlying model's native capabilities, or can make up for its absence."

Some critics noted that the research focused on Nemotron and Mistral models rather than the current frontier of models from providers like Claude, GPT, or DeepSeek. However, others argued that the findings regarding the relationship between context windows and management strategies remain highly applicable to real-world usage patterns, such as checkpointing sessions to keep API costs down.