OpenAI Analysis of SWE-Bench Pro Coding Evaluation Flaws

OpenAI has identified significant flaws in the SWE-Bench Pro coding benchmark, estimating that approximately 30% of its tasks are broken. These findings lead OpenAI to retract its previous recommendation that model developers adopt SWE-Bench Pro as a replacement for SWE-bench Verified.

Audit Findings and Task Failure Rates

An audit of the SWE-Bench Pro public split (731 tasks) revealed that a substantial portion of the dataset provides an unreliable signal of model capabilities. The analysis identified two different failure rates based on the methodology used:

• Human Annotation: Identified 249 broken tasks (34.1%).
• Data Quality Pipeline: Flagged 200 broken tasks (27.4%).

Primary Categories of Evaluation Flaws

OpenAI categorized the broken tasks into four primary failure modes that invalidate the results of the benchmark:

• Overly Strict Tests: Tests that enforce specific implementation details not requested in the prompt, causing functionally correct solutions to be marked as incorrect.
• Underspecified Prompts: Prompts that omit requirements that are enforced by hidden tests but cannot be reasonably inferred by the model.
• Low-Coverage Tests: Tests that do not sufficiently check the requested feature, allowing incomplete or incorrect fixes to pass.
• Misleading Prompts: Prompts that point models toward incorrect behavior or directly contradict the requirements of the tests.

Methodology for Quality Assurance

To ensure that task failures reflect genuine model limitations rather than benchmark flaws, OpenAI employed a multi-stage quality assurance pipeline:

Automated Filtering

An initial automated filter reviewed model attempts, task metadata, and failure traces to flag 286 potentially problematic tasks.

Human-Supervised Agent Review

Codex-based investigator agents were given access to the task repositories and environments. These agents ran tests, inspected files, and investigated common failure modes to distinguish between reasonable ambiguity and true underspecification. A researcher then reviewed the agent summaries to make a final judgment.

Human Annotation Campaign

Experienced software engineers reviewed the flagged subset. Each task was reviewed by five engineers who formed independent judgments based on the problem statement, test cases, and the gold patch (ground-truth reference solution) before reviewing the pipeline analysis.

Comparison of Agent vs. Human Review

Human reviewers were more likely to identify tasks as broken than the investigator agents. While the agent pipeline and human reviewers overlapped in 74% of cases, humans were more likely to assign multiple labels to a single task, suggesting the agent-plus-reviewer pipeline was conservative. The most notable difference was in "low-coverage tests," which humans identified in 9.4% of the benchmark compared to 4.1% by the agent pipeline.

Implications for Coding Benchmarks

OpenAI notes that benchmarks sourced programmatically from open-source repository history—where issues and pull requests were designed for human collaboration—often fail to produce clean, isolated tasks. Tests in pull requests are frequently written to validate a specific change rather than to define an implementation-agnostic standard.

OpenAI advocates for the development of new benchmarks created by experienced software developers specifically for model evaluation to ensure they are hard to game, easy to trust, and genuinely reflective of model capabilities.