OpenAI Audit Reveals 30% of SWE-bench Pro Tasks are Broken

OpenAI has retracted its recommendation for the SWE-bench Pro coding benchmark after a detailed audit revealed that approximately 30% of its tasks are broken. The analysis found that a significant portion of the dataset fails to provide a meaningful signal of model capabilities because the tasks are either logically flawed, underspecified, or graded by overly restrictive tests.

Audit Findings: 30% of Tasks are Broken

An audit of the 731-task public split of SWE-bench Pro identified that a substantial number of tasks are fundamentally flawed. An automated datapoint analysis pipeline flagged 200 tasks (27.4%) as problematic, while a subsequent human annotation campaign by experienced software engineers identified 249 broken tasks (34.1%).

OpenAI categorized the failure modes into four primary types:

• Overly strict tests: Tests that enforce specific implementation details not requested in the prompt, causing functionally correct solutions to be marked as failures.
• Underspecified prompts: Prompts that omit requirements which are enforced by hidden tests but cannot be reasonably inferred by the model.
• Low-coverage tests: Tests that fail to sufficiently check the requested feature, allowing incomplete or incorrect fixes to pass.
• Misleading prompts: Instructions that point the model toward incorrect behavior or directly contradict the requirements of the tests.

Methodology: Hybrid Agent-Human Review

To ensure that task failures reflect genuine model limitations rather than benchmark flaws, OpenAI employed a two-stage quality assurance pipeline.

Automated Agent Filtering

An initial filter reviewed model attempts, task metadata, and failure traces to flag potentially broken examples. This filter identified 286 potentially problematic tasks.

Human-Supervised Agent Audit

Flagged tasks were then audited using Codex-based investigator agents. These agents had access to the task repository and environment, allowing them to run tests and inspect files to distinguish between reasonable ambiguity and true underspecification. A researcher then reviewed the agent's summaries to make a final judgment.

Human Annotation Campaign

In parallel, five experienced software engineers reviewed each flagged task independently. Reviewers analyzed the problem statement, test cases, and the "gold patch" (ground-truth reference solution) before assigning a label and severity rating.

OpenAI noted that human reviewers were more likely than investigator agents to mark tasks as broken and were more likely to identify multiple overlapping issues per task. Specifically, humans identified low-coverage tests as the most common issue for 9.4% of the benchmark, compared to 4.1% identified by the agent pipeline.

The Challenge of Real-World Data in Benchmarks

OpenAI attributes these flaws to the nature of how SWE-bench tasks are sourced. Because tasks are programmatically derived from the history of feature changes in public and private repositories, they often reflect the messy reality of human collaboration.

Pull requests and issues in open-source repositories are often the result of long discussions between maintainers and contributors. Consequently, the merged code and unit tests may not align perfectly to form a clean, isolated task. Tests included in pull requests are often written to validate a specific change rather than to define an implementation-agnostic standard for solving a problem.

Community Perspectives and Counterpoints

Industry practitioners and researchers have raised several points regarding the validity of these findings and the utility of such benchmarks:

"Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play."

Some argue that dealing with underspecified prompts and contradictory instructions is a core part of a software engineer's job, and therefore, these "flaws" are actually realistic tests of a model's ability to handle real-world ambiguity. Others point out that many current benchmarks are "one-shot" and fail to test the long-iteration cycles that characterize actual software development.

Additional concerns were raised regarding "benchmaxxing"—the practice of labs modifying timeouts or hardware configurations to artificially inflate results on leaderboards like Terminal Bench 2.0, which may further degrade the signal-to-noise ratio in public coding evaluations.

Conclusion

OpenAI concludes that for an evaluation to be useful, it must be hard to game, easy to trust, and genuinely reflective of model capability. Given the high error rate in SWE-bench Pro, OpenAI advises model developers to examine their results carefully and suggests that the community move toward benchmarks specifically designed by experienced developers to test model capabilities, rather than programmatically sourced datasets.