scaleapi/SWE-bench_Pro-os

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

What it solves

SWE-Bench Pro provides a rigorous benchmark to evaluate how well LLMs and AI agents can handle real-world, long-horizon software engineering tasks. It addresses the need for a validated dataset where AI must not only understand a codebase and a reported issue but also generate a functional patch to resolve the problem.

How it works

The project provides a dataset of software issues and corresponding codebases. AI agents are given a codebase and an issue description and are tasked with producing a code patch. The system evaluates these patches using a reproducible pipeline involving Docker containers (or Modal) to ensure the environment is consistent. The V2 version includes 642 validated tasks across 11 repositories, utilizing the Harbor format for task directories and verifiers.

Who it’s for

It is designed for researchers and developers building AI agents for software engineering, as well as those benchmarking the coding capabilities of large language models.

Highlights

  • Validated Dataset: V2 contains 642 validated tasks and a specific "HARD-51" subset for extreme testing.
  • Reproducible Evaluation: Uses Docker and Modal to provide isolated, consistent environments for testing patches.
  • Integration: Compatible with tools like SWE-agent and the Harbor task framework.
  • Public Leaderboards: Maintains public and private leaderboards to track agent performance.

Related

  • Project
    Kaggle/kagglehub

    A Python library for programmatically downloading, uploading, and loading Kaggle datasets, models, and competition resources into data science frameworks.

  • Project
    socprime/detectflow-main

    A real-time cyberattack detection system that uses AI and Flink-based pipelines to process streaming events and apply Sigma rules in-flight, drastically reducing detection latency.

  • Project
    termio-sh/termio

    A terminal-first agentic development environment that manages and monitors CLI-based AI coding agents across local and remote machines.

  • Project
    awslabs/generative-ai-cdk-constructs

    An open-source extension of the AWS CDK that provides well-architected infrastructure patterns for quickly deploying generative AI solutions on AWS.