Quantifying Infrastructure Noise in Agentic Coding Evals

Anthropic has found that infrastructure configuration alone can produce performance differences in agentic coding benchmarks that exceed the margins typically used to distinguish top-performing models. In internal experiments using Terminal-Bench 2.0, the gap in success rates between the most- and least-resourced setups was 6 percentage points (p < 0.01).

Infrastructure as an Active Component of Agentic Evals

Unlike static benchmarks that score a model's output directly, agentic coding evaluations (such as SWE-bench and Terminal-Bench) provide models with a full runtime environment to write programs, install dependencies, and run tests. This makes the runtime an integral part of the problem-solving process rather than a passive container.

Anthropic observed that when resource specifications (CPU and RAM) are treated as both a floor and a hard ceiling, the lack of headroom for transient spikes leads to high infrastructure error rates. In their Google Kubernetes Engine (GKE) setup, up to 6% of tasks failed due to pod errors unrelated to model capability. This occurs because a momentary memory fluctuation can trigger an Out-of-Memory (OOM) kill in containers where guaranteed allocation and hard limits are identical.

Impact of Resource Headroom on Success Rates

To quantify the effect of the scaffold, Anthropic tested Terminal-Bench 2.0 across six resource configurations, ranging from strict enforcement (1x) to completely uncapped resources. The findings demonstrate a two-stage relationship between resources and scores:

1. Reliability Phase (1x to 3x Headroom)

Increasing resources up to roughly 3x the recommended specs primarily improves infrastructure reliability. Infra error rates dropped monotonically from 5.8% at strict enforcement to 2.1% at 3x headroom (p < 0.001). During this phase, success scores fluctuated within the margins of noise (p = 0.40), suggesting that additional resources primarily fixed spurious crashes rather than making the tasks easier to solve.

2. Capability Phase (3x to Uncapped)

Beyond the 3x mark, success rates began to climb faster than infrastructure errors declined. Between 3x and uncapped resources, infra errors dropped by 1.6 percentage points, but success rates jumped by nearly 4 percentage points. This indicates that generous resources allow agents to employ strategies that are only possible with high allocations, such as:

Pulling in large dependencies.
Spawning expensive subprocesses.
Running memory-intensive test suites.

For example, in the bn-fit-modify task, some models attempt to install a full data science stack (pandas, networkx, scikit-learn). This strategy succeeds under generous limits but fails under tight ones, whereas a leaner strategy (implementing math from scratch) succeeds regardless of the limit.

Cross-Benchmark Validation

Anthropic replicated these findings across different models and benchmarks:

Model Consistency: The direction of the effect was consistent across different Anthropic models, though the magnitude varied.
SWE-bench: A crossover experiment on 227 problems (10 samples each) showed that scores increased monotonically with RAM up to 5x the baseline. However, the effect was smaller (1.54 percentage points higher at 5x than 1x), likely because SWE-bench tasks are generally less resource-intensive than Terminal-Bench.

Other Sources of Variance

Beyond RAM and CPU, other system-level factors can act as confounders in agentic evaluations:

Time Limits: Certain configurations see performance fluctuations based on the time allowed for the agent.
Environmental Noise: Pass rates can fluctuate based on the time of day, potentially due to API latency variations tied to traffic patterns.
Hardware and Network: Cluster health, hardware specs, concurrency levels, and egress bandwidth can all influence the final score.

Recommendations for Evaluation Rigor

To minimize infrastructure noise and prevent the conflation of system quirks with model capability, Anthropic recommends the following:

Specify Dual Parameters: Evals should specify both a guaranteed allocation (floor) and a separate hard kill threshold (ceiling) per task. This prevents spurious OOM kills while maintaining a hard limit to prevent score inflation.
Calibrate the Band: The gap between the floor and ceiling should be calibrated so that scores remain within the noise margin. In Terminal-Bench 2.0, a 3x ceiling was found to be an effective tradeoff, reducing infra errors significantly without substantially inflating scores.
Report Configurations: Resource multipliers and enforcement methodologies should be explicitly reported alongside benchmark results.
Increase Sampling: Running evaluations multiple times across different days can help average out transient noise.

Implications for Benchmark Interpretation

Because a small lead on a leaderboard may reflect a larger VM rather than a superior model, Anthropic suggests that leaderboard differences below 3 percentage points should be viewed with skepticism unless the evaluation configuration is documented and matched. The observed spread across moderate resource configurations in Terminal-Bench is just below 2 percentage points, which stacks on top of existing binomial confidence intervals.