OpenAI Discontinues SWE-bench Verified Evaluation
OpenAI has ceased reporting results for the SWE-bench Verified benchmark, concluding that it no longer accurately measures the real-world software engineering capabilities of frontier AI models. The decision follows an analysis showing that improvements in scores often reflect training data contamination rather than genuine capability gains, compounded by significant flaws in the benchmark's test cases.
Flawed Test Case Design
An audit of 138 problems that were not consistently solved by OpenAI o3 revealed that 59.4% of these tasks contained material issues in test design or problem descriptions. These flaws make tasks nearly impossible to solve regardless of the model's capability.
OpenAI categorized these failures into two primary types:
- Narrow Test Cases (35.5%): Tests that enforce specific implementation details not required by the problem description. For example, in task
pylint-dev__pylint-4551, tests failed because models did not name a new functionget_annotation, even though the function name was not specified in the requirements. - Wide Test Cases (18.8%): Tests that check for functionality not mentioned in the problem description. In task
sympy__sympy-18199, the problem description covered only one of three issues addressed in the original PR, but the tests required fixes for all three to pass.
Training Data Contamination
Because SWE-bench problems are sourced from open-source repositories used in model training, frontier models have been exposed to the solutions (gold patches) during their training phase. OpenAI found that all tested frontier models could reproduce original human-written bug fixes or verbatim problem specifics for certain tasks.
Evidence of contamination was found across multiple model families:
- GPT-5.2: Demonstrated knowledge of release notes and specific parameters (e.g.,
edit_onlyin Django 4.1) not explicitly required by the problem statement. - Claude Opus 4.5: Recalled exact four-line functional changes, specific filenames, and verbatim inline comments from the original diffs.
- Gemini 3 Flash: Produced verbatim details from task descriptions and gold patches, including regex formulas and exact line numbers, using only the task ID as a prompt.
OpenAI noted that models exposed to these problems during training are more likely to succeed because they possess the additional information needed to pass the underspecified or flawed tests.
Implications for AI Evaluation
OpenAI identifies two critical lessons for the design of future AI benchmarks:
- Contamination Risk: Benchmarks sourced from public material are prone to silent score inflation. OpenAI recommends that developers use password-protected datasets and strict canary strings for training data filtering.
- Automated Scoring Complexity: Creating perfect test cases that are agnostic to unimportant implementation details while remaining robust against shortcuts is inherently difficult and requires extensive human review.
Recommended Alternatives
OpenAI is currently developing new, uncontaminated evaluations to track coding capabilities. In the interim, the lab recommends that model developers report results from the public split of SWE-bench Pro. While not perfect, empirical evidence suggests SWE-bench Pro suffers from significantly less contamination, with no tested model able to produce a complete verbatim gold patch.
For high-stakes measurement, OpenAI points to GDPVal, where tasks are privately authored by domain experts and graded holistically by human reviewers to eliminate exposure risk.