SWE-bench/SWE-bench

SWE-bench: Can Language Models Resolve Real-world Github Issues?

What it solves

SWE-bench evaluates how well large language models (LLMs) can resolve real-world software issues. It addresses the gap between simple coding tasks and the complex reality of managing large codebases, where a model must understand a codebase and generate a functional patch to fix a GitHub issue.

How it works

The benchmark provides a set of real-world GitHub issues and their corresponding codebases. A model is given a codebase and an issue description, and it must generate a code patch (a diff) that resolves the problem. The system uses a fully containerized evaluation harness via Docker to ensure that the evaluations are reproducible and isolated.

Who it’s for

AI researchers and software engineering agents developers who are building and testing LLMs for autonomous software engineering tasks.

Highlights

  • Real-world Data: Uses actual GitHub issues to test models on genuine software problems.
  • Containerized Evaluation: Employs Docker for reproducible and isolated testing environments.
  • Diverse Datasets: Includes specialized versions like SWE-bench Verified (human-confirmed solvable problems), Multimodal (visual software domains), and Multilingual.
  • ** uma Cloud Integration**: Supports cloud-based evaluations via sb-cli and Modal.
  • Training Support: Provides pre-processed datasets and tools like SWE-smith for creating training data.

Related

  • Project
    SWE-agent/SWE-agent

    An autonomous AI agent that enables LLMs to fix GitHub issues, find security vulnerabilities, and perform custom coding tasks.

  • Dispatch
    Senior SWE-Bench Open-Source Benchmark for Evaluating LLMs as Senior Engineers

    Senior SWE-Bench is an open-source benchmark that measures how well AI agents perform senior software engineering tasks, highlighting both current capabilities and challenges of LLM-driven development.

  • Project
    augmentcode/augment-swebench-agent

    A software engineering agent that solves real-world GitHub issues by combining Claude 3.7 Sonnet for execution and OpenAI o1 for solution ensembling.

  • Project
    SWE-bench/SWE-smith

    A toolkit for scaling the creation of training data and environments for software engineering agents, enabling the training of LLMs to solve real-world GitHub issues.

  • Project
    datacurve-ai/deep-swe

    DeepSWE is a benchmark for measuring coding agents on 113 long-horizon software engineering tasks from real-world open-source repositories.