SWE-bench/SWE-bench

SWE-bench: Can Language Models Resolve Real-world Github Issues?

What it solves

SWE-bench is designed to evaluate how well large language models (LLMs) can resolve real-world software engineering issues. It moves beyond simple code snippets to test if an AI can handle entire codebases and generate functional patches for actual GitHub issues.

How it works

The benchmark provides a set of real-world GitHub issues paired with their corresponding codebases. A model is given the codebase and the issue description, and it must generate a code patch (a diff) that resolves the problem. The system uses a fully containerized evaluation harness via Docker to execute and verify if the generated patch actually fixes the issue in a reproducible way.

Who it’s for

It is primarily for AI researchers and developers building software engineering agents or LLMs specialized in coding, who need a standardized way to measure their model's ability to solve complex, real-world software bugs.

Highlights

  • Real-World Data: Uses actual GitHub issues and codebases rather than synthetic tasks.
  • Containerized Evaluation: Employs Docker to ensure that evaluations are reproducible and isolated.
  • Diverse Datasets: Includes specialized versions like SWE-bench Verified (human-confirmed solvable problems), Multimodal (visual software domains), and Multilingual.
  • Cloud Integration: Supports cloud-based evaluations through Modal and sb-cli.
  • Training Support: Provides pre-processed datasets and tools for training and inference on existing models.

Related

  • Project
    SWE-agent/SWE-agent

    An autonomous AI agent that enables LLMs to fix GitHub issues, find security vulnerabilities, and perform custom coding tasks.

  • Dispatch
    Senior SWE-Bench Open-Source Benchmark for Evaluating LLMs as Senior Engineers

    Senior SWE-Bench is an open-source benchmark that measures how well AI agents perform senior software engineering tasks, highlighting both current capabilities and challenges of LLM-driven development.

  • Project
    augmentcode/augment-swebench-agent

    A software engineering agent that solves real-world GitHub issues by combining Claude 3.7 Sonnet for execution and OpenAI o1 for solution ensembling.

  • Project
    SWE-bench/SWE-smith

    A toolkit for scaling the creation of training data and environments for software engineering agents, enabling the training of LLMs to solve real-world GitHub issues.

  • Project
    datacurve-ai/deep-swe

    DeepSWE is a benchmark for measuring coding agents on 113 long-horizon software engineering tasks from real-world open-source repositories.