OpenAI Analysis of SWE-bench Pro Coding Evaluation Flaws

OpenAI has retracted its recommendation for the SWE-bench Pro benchmark, concluding that approximately 30% of the tasks within the dataset are broken. This finding follows a rigorous audit of the benchmark's 731-task public split, revealing that many evaluations provide a false signal of model capability by penalizing functionally correct solutions or rewarding incomplete ones.

The Failure Modes of Coding Benchmarks

OpenAI identified four primary categories of flaws that invalidate coding evaluations. These issues often stem from the fact that benchmarks are programmatically sourced from real-world GitHub issues and pull requests, which were originally intended for human collaboration rather than isolated machine testing.

• Overly Strict Tests: Tests that enforce specific implementation details not specified in the prompt, causing functionally correct submissions to fail.
• Underspecified Prompts: Prompts that omit requirements that are enforced by hidden tests but are not reasonably inferable from the provided context.
• Low-Coverage Tests: Tests that fail to sufficiently check the requested feature, allowing incomplete or incorrect fixes to pass.
• Misleading Prompts: Instructions that point models toward incorrect behavior or directly contradict the requirements of the tests.

Audit Methodology: Agents and Humans

To identify these flaws, OpenAI employed a dual-track quality assurance pipeline. This approach highlights the growing utility of AI agents to perform scalable data quality checks that were previously too costly to perform manually.

Automated Agent Review

An initial filter flagged 286 potentially broken tasks. These were then audited by Codex-based investigator agents with access to the task repository and environment. These agents could run tests, inspect files, and investigate common failure modes to distinguish between reasonable ambiguity and true underspecification.

Human Annotation Campaign

In parallel, five experienced software engineers reviewed the flagged tasks. Reviewers formed independent judgments based on the problem statement, test cases, and the "gold patch" (ground-truth reference solution).

Comparative Results

While the agent pipeline and human reviewers overlapped in 74% of cases, human reviewers were more likely to identify tasks as broken. Specifically, humans flagged low-coverage tests as the most common issue for 9.4% of the benchmark, compared to only 4.1% identified by the agent pipeline. This suggests that agent-led reviews are conservative and may undercount overlapping issues.

Industry Perspectives and Counterpoints

Community discussion surrounding these findings reveals a tension between the desire for "clean" benchmarks and the reality of software engineering.

Real-World Noise as a Feature

Some critics argue that underspecified prompts and contradictory instructions are not "bugs" in the benchmark, but rather accurate reflections of the professional environment. As one commenter noted:

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play.

The "Benchmaxx" Problem

Other observers pointed to the prevalence of "reward hacking" and "benchmaxxing"—where labs modify timeouts or hardware configurations to artificially inflate scores—as a broader systemic issue in AI evaluation. This is further complicated by the fact that many benchmarks are "one-shot" and fail to test the long-iteration cycles that characterize actual developer workflows.

Conclusion: The Need for Curated Benchmarks

OpenAI concludes that benchmarks sourced from organic open-source data are inherently prone to noise. To ensure that deployment and safety decisions are grounded in valid data, OpenAI advocates for the development of new benchmarks created by experienced software developers specifically for model evaluation, rather than relying on programmatically harvested data.