OpenAI SWE-bench Verified release: human‑validated benchmark improves software‑engineering evaluation

TL;DR

OpenAI released SWE‑bench Verified, a human‑validated 500‑sample subset of the SWE‑bench software‑engineering benchmark that removes ambiguous issues and unfair unit tests, enabling more reliable evaluation of AI models’ autonomous coding abilities; GPT‑4o solves 33.2% of these samples, more than double its score on the original benchmark.

Background on SWE‑bench

SWE‑bench evaluates large language models (LLMs) on real‑world GitHub issues from 12 open‑source Python repositories. Each test sample provides an issue description and repository code; the model must edit the code so that the associated FAIL_TO_PASS unit tests (which initially fail) pass, while all PASS_TO_PASS tests continue to pass. The benchmark has become a standard for measuring autonomous software‑engineering capability, with top agents achieving 20% on the full set and 43% on the easier SWE‑bench Lite as of August 5 2024.

Why the Original Benchmark Needed Improvement

OpenAI identified three systematic problems that caused SWE‑bench to underestimate model performance:

  1. Over‑specific or unrelated unit tests – some FAIL_TO_PASS tests required behavior not mentioned in the issue description, rejecting correct solutions.
  2. Underspecified issue statements – ambiguous problem descriptions left models without enough information to produce a valid fix.
  3. Environment setup fragility – difficulties reproducing the repository environment caused tests to fail regardless of the solution.

An illustrative case is the scikit-learn__scikit-learn-14520 sample, where the issue mentions a copy argument being ignored, but the required test also demands a specific DeprecationWarning message that never appears in the issue text. Without access to the hidden test, an agent cannot reliably satisfy the requirement.

SWE‑bench Verified: A Human‑Validated Subset

To address these flaws, OpenAI launched a large‑scale annotation campaign:

  • Annotators – 93 professional Python developers screened 1,699 random SWE‑bench samples.
  • Annotation rubric – each sample received four severity labels (0‑3) for problem‑statement underspecification and unit‑test fairness, plus a difficulty estimate (time to solve for an experienced engineer).
  • Consensus process – every sample was labeled by three independent annotators; the highest severity label was retained to ensure conservative filtering.
  • Filtering criteria – any sample with a severity label ≥ 2 on either axis, or flagged with a major issue, was removed.

The resulting SWE‑bench Verified set contains 500 high‑quality samples that are free of the identified problems. Difficulty annotations reveal an “easy” subset of 196 tasks estimated to require under 15 minutes, and a “hard” subset of 45 tasks estimated to take over an hour. The dataset supersedes both the original SWE‑bench and SWE‑bench Lite.

New Evaluation Harness

OpenAI collaborated with the SWE‑bench authors to release a Docker‑based evaluation harness, simplifying environment setup and improving reproducibility for future benchmarking.

Model Performance on SWE‑bench Verified

Using the new dataset, OpenAI evaluated GPT‑4o with several open‑source scaffolds that previously performed well on the original leaderboard:

  • Best scaffold – GPT‑4o achieved 33.2% solve rate on SWE‑bench Verified, more than doubling its 16% score on the original SWE‑bench.
  • Open‑source scaffold improvement – the Agentless scaffold, an open‑source system, also doubled its score from 16% to roughly 33% after the dataset cleaning.

Performance Across Difficulty Levels

Performance gains were observed within each difficulty bucket, not merely because the dataset contains more easy tasks. This indicates that the filtering removed genuinely infeasible samples rather than simply shifting the difficulty distribution.

Implications for AI Preparedness

SWE‑bench Verified serves as a more trustworthy metric within OpenAI’s Preparedness Framework, which tracks model autonomy risk at the Medium level. Accurate benchmarks are crucial for:

  • Detecting genuine capability improvements versus evaluation artefacts.
  • Guiding risk assessments as models approach higher levels of autonomy.
  • Informing the community about the need for rigorous benchmark curation.

OpenAI emphasizes three takeaways:

  1. Deep benchmark understanding – even well‑designed suites can hide systematic biases that misrepresent model abilities.
  2. Ecosystem awareness – external scaffolding advances can dramatically affect scores; continuous re‑evaluation is essential.
  3. Recognize limitations – static, public‑GitHub‑derived datasets may suffer from data contamination and cover only a narrow slice of autonomy risk, so they must be complemented by other evaluations.

Data Availability

All annotations, the verified subset, and the new Docker harness are publicly released. Researchers can download the resources via the links provided in the original post.

Authors & Acknowledgements

The work was conducted by Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry (equal contributions). Acknowledgements include the original SWE‑bench creators, the Preparedness team, and the many annotators who made SWE‑bench Verified possible.

Citation: OpenAI, Introducing SWE‑bench Verified, August 13 2024. URL: https://openai.com/index/introducing-swe-bench-verified

Sources