Quantifying Infrastructure Noise in Agentic Coding Evaluations

Agentic coding benchmarks are significantly influenced by underlying infrastructure configuration, meaning that small differences in leaderboard scores may reflect hardware differences rather than genuine model capability gaps. This research demonstrates that resource headroom can drastically alter evaluation outcomes, necessitating that resource configuration be treated as a primary experimental variable in model testing.

The Impact of Runtime Environment on Agentic Evals

Agentic coding evaluations differ fundamentally from static benchmarks because the runtime environment is an integral component of the problem-solving process. In these tests, models write code, run tests, install dependencies, and iterate over multiple turns within a full environment. Therefore, the resource budget and time limits provided to an agent are not passive constraints but active factors influencing success.

In internal experiments using Terminal-Bench 2.0, the gap in success rates between the most- and least-resourced setups was measured at 6 percentage points (p < 0.01). This finding highlights that infrastructure configuration alone can produce differences that exceed the typical margins used to judge relative model capability.

Resource Allocation and Success Rate Trends

To quantify the effect of the evaluation scaffold, the researchers ran Terminal-Bench 2.0 across six distinct resource configurations, varying from strict enforcement of per-task specifications (1x) to completely uncapped resources. The experiments showed a clear trend: success rates increased monotonically with resource headroom.

Infrastructure Reliability vs. Capability Lift

Strict Enforcement (1x): Under strict enforcement, infrastructure error rates were high, reaching 5.8% of tasks failing due to pod errors unrelated to the model's ability. Tasks that crashed at 1x were often not on a path to a correct solution.
Moderate Headroom (1x to 3x): Increasing resources to 3x headroom significantly dropped infrastructure error rates (from 5.8% to 2.1%, p < 0.001). However, success scores in this range fluctuated within the margins of noise (p=0.40).
Generous Headroom (3x to Uncapped): Above the 3x mark, the trend changed. Success rates climbed faster than infra errors declined. The extra resources enabled agents to utilize approaches requiring generous allocations, such as spawning expensive subprocesses or pulling in large dependencies.
Total Lift: At uncapped resources, the total success rate lift over the 1x configuration was +6 percentage points (p < 0.01).

How Resource Limits Change Evaluation Measurement

Resource configuration affects the evaluation in two distinct ways, depending on the level of headroom provided.

Regime 1: Fixing Reliability (Up to 3x)

Up to approximately 3x the Terminal-Bench specifications, additional resources primarily fix infrastructure reliability problems, specifically transient resource spikes. The sandboxing provider used by the benchmark maintainers implicitly handles this stability without making the evaluation inherently easier.

Regime 2: Changing the Problem Space (Above 3x)

Above the 3x threshold, additional resources actively help the agent solve problems it could not solve previously. This demonstrates that resource limits can change what the evaluation measures: tight limits inadvertently reward highly efficient strategies, while generous limits reward agents that can effectively exploit all available resources.

For example, on the bn-fit-modify task, generous limits allow models to install the standard Python data science stack (pandas, networkx, scikit-learn). Under tight limits, the pod runs out of memory during installation, preventing the agent from even writing solution code.

Generalizing the Noise Across Benchmarks

This pattern of resource-induced variance was replicated across different Anthropic models. The effect was also tested on SWE-bench, a different coding evaluation, by varying total available RAM up to 5x the baseline across 227 problems. The same effect held, though the magnitude was smaller, with scores being only 1.54 percentage points higher at 5x RAM than at 1x. This smaller effect is expected because SWE-bench tasks are less resource-intensive.

Beyond resource allocation, other variables contribute to evaluation noise. Agentic evals are end-to-end system tests, making them susceptible to confounders such as time limits, cluster health, concurrency level, and API latency, which can fluctuate with traffic patterns.

Recommendations for Reproducible Evaluation

To mitigate infrastructure noise and ensure reproducible results, the following practices are recommended:

Ideal Scenario: The most rigorous approach is running the evaluation under identical hardware conditions for both the scaffold and the inference stack.
Practical Implementation: Given how container runtimes enforce resources (via a guaranteed allocation and a hard kill threshold), evaluations should specify both parameters per task, not a single pinned value. Setting the guaranteed allocation equal to the kill threshold leaves zero margin for transient memory spikes.
Calibration Principle: The band between the guaranteed allocation and the hard kill threshold should be calibrated so that scores at the floor and ceiling fall within the noise of each other. For Terminal-Bench 2.0, a 3x ceiling cut infra error rates by roughly two-thirds (5.8% to 2.1%, p < 0.001) while keeping the score lift modest and within noise (p = 0.40).

Conclusion: Interpreting Leaderboard Differences

Benchmark scores are increasingly used for critical decision-making, but the reliance on these numbers must be tempered by the understanding of infrastructure variability. Without standardized setup configurations, it is difficult to distinguish between a genuine capability difference and an infrastructure quirk.

Anthropic recommends that resource configuration for agentic evals be treated as a first-class experimental variable. Until resource methodology is standardized, leaderboard differences below 3 percentage points should be viewed with skepticism, as the documented infrastructure confounders stack on top of existing statistical uncertainty.