OpenAI SWE-Lancer Benchmark Release

OpenAI has released SWE-Lancer, a benchmark consisting of over 1,400 freelance software engineering tasks sourced from Upwork. This benchmark allows researchers to evaluate AI models' ability to solve real-world, paid software engineering problems, mapping model performance directly to monetary value to assess the economic impact of AI development.

Technical Composition of SWE-Lancer

SWE-Lancer includes a diverse range of tasks that reflect actual professional software engineering work. The benchmark is divided into two primary categories of work:

Independent Engineering Tasks

These tasks range in complexity and value, from small bug fixes valued at $50 to large-scale feature implementations valued up to $32,000. These tasks are graded using end-to-end tests that have been triple-verified by experienced software engineers to ensure accuracy.

Managerial Tasks

Beyond technical implementation, SWE-Lancer includes managerial tasks where models must choose between different technical implementation proposals. These decisions are assessed against the same choices made by the original hired engineering managers.

Model Performance and Evaluation

OpenAI reports that frontier models currently cannot solve the majority of the tasks within the SWE-Lancer benchmark. This indicates a significant gap between current AI capabilities and the professional software engineering standards required for freelance work.

Research Accessibility and Updates

To support the research community, OpenAI has open-sourced a unified Docker image and a public evaluation split known as SWE-Lancer Diamond.

An update on July 28, 2025, noted that the dataset and results were updated as of July 17, 2025. This update removed the requirement for internet connectivity during execution, which was intended to eliminate a primary source of variability in model performance. The updated dataset and results are available via the OpenAI preparedness GitHub repository.

Economic Impact Mapping

By utilizing tasks with real-world payouts totaling $1 million USD, SWE-Lancer is designed to enable research into the economic value of AI-driven software engineering, providing a monetary metric for model performance improvements.