augmentcode/augment-swebench-agent

The #1 open-source SWE-bench Verified implementation

What it solves

This project provides a coding agent designed to solve complex software engineering tasks, specifically those found in the SWE-bench Verified benchmark. Unlike simple coding challenges, these tasks require the agent to navigate entire codebases, iterate based on regression tests, and handle the complexity of real-world GitHub issues.

How it works

The agent uses Claude 3.7 Sonnet as its primary driver and is based on an architecture forked from Anthropic. It operates by executing tools such as bash commands, file editing, and a "sequential thinking" process for complex problem-solving. To improve accuracy, the system employs a majority vote ensembler powered by OpenAI's o1, which analyzes multiple candidate solutions generated by the agent and selects the best one.

Who it’s for

It is intended for developers and AI researchers who want a ready-to-use baseline agent for software engineering benchmarks, or those looking for a simple, extensible framework to build and test their own coding agents.

Highlights

  • Integrated Toolset: Includes built-in capabilities for bash execution, file manipulation, and sequential thinking.
  • Ensembling Logic: Uses a majority vote system to select the highest-quality solution from multiple candidates.
  • Docker Integration: Runs within Docker containers to ensure safe and isolated execution of code.
  • Flexible Modes: Supports both an interactive CLI for personal assistance and a non-interactive mode for running large-scale evaluations on SWE-bench.

Related

  • Project
    SWE-agent/SWE-agent

    An autonomous AI agent that enables LLMs to fix GitHub issues, find security vulnerabilities, and perform custom coding tasks.

  • Project
    SWE-bench/SWE-bench

    A benchmark for evaluating large language models on their ability to resolve real-world GitHub software issues by generating functional code patches.

  • Dispatch
    Claude 3.5 Sonnet SWE-bench Performance

    The upgraded Claude 3.5 Sonnet achieved a state-of-the-art 49% success rate on SWE-bench Verified, demonstrating superior reasoning and agentic coding capabilities.

  • Dispatch
    Claude 3.5 Sonnet SWE-bench Performance

    The upgraded Claude 3.5 Sonnet achieved a new state-of-the-art score of 49% on SWE-bench Verified, demonstrating superior software engineering agent capabilities over previous models.

  • Dispatch
    Claude 3.5 Sonnet Integration with GitHub Copilot

    Anthropic's Claude 3.5 Sonnet is now available in public preview on GitHub Copilot, providing developers with a high-performance coding model that outperforms other publicly available models on SWE-bench Verified.