datacurve-ai/deep-swe

Measuring frontier coding agents on original, long-horizon engineering tasks

What it solves

DeepSWE provides a standardized way to measure the performance of frontier coding agents. It addresses the need for benchmarks that use original, long-horizon software engineering tasks from real-world open-source repositories, rather than simple snippets or synthetic tests.

How it works

DeepSWE consists of 113 tasks across five languages (TypeScript, Go, Python, JavaScript, and Rust). Each task is packaged using the Harbor task format, which includes metadata, instructions, and isolated environments. To ensure fair and objective grading, it uses program-based verifiers that check if the agent's observable behavior is correct, regardless of how the code is written. It is typically run using Pier, a framework that provides sandboxed environments and network allowlists to keep tasks isolated while allowing agents to communicate with LLM APIs.

Who it’s for

Developers and researchers building coding agents and LLMs who need to evaluate their ability to handle complex, multi-step software engineering tasks in real-world settings.

Highlights

  • Real-world tasks: Tasks are drawn from active open-source repositories.
  • Multi-language support: Covers TypeScript, Go, Python, JavaScript, and Rust.
  • Multi-modal grading: Uses isolated containers and program-based verifiers to ensure correctness.
  • Sandboxed execution: Integrates with Pier to provide secure, isolated environments for agent execution.

Related

  • Project
    langchain-ai/open-swe

    An open-source framework for building internal organizational coding agents that integrate with Slack, Linear, and GitHub using isolated cloud sandboxes.

  • Project
    ByteDance-Seed/EdgeBench

    EdgeBench is a benchmark of 134 real-world tasks designed to evaluate how autonomous AI agents learn and improve through iterative interaction with environments over long horizons.

  • Project
    SWE-bench/SWE-bench

    A benchmark for evaluating large language models on their ability to resolve real-world GitHub software issues by generating functional code patches.

  • Project
    SWE-agent/SWE-agent

    An autonomous AI agent that enables LLMs to fix GitHub issues, find security vulnerabilities, and perform custom coding tasks.

  • Dispatch
    Senior SWE-Bench Open-Source Benchmark for Evaluating LLMs as Senior Engineers

    Senior SWE-Bench is an open-source benchmark that measures how well AI agents perform senior software engineering tasks, highlighting both current capabilities and challenges of LLM-driven development.