Claude 3.5 Sonnet SWE-bench Performance

The upgraded Claude 3.5 Sonnet has achieved a 49% success rate on SWE-bench Verified, surpassing the previous state-of-the-art score of 45%. This performance demonstrates the model's improved reasoning, coding, and mathematical abilities when integrated into a minimal agent scaffold designed for real-world software engineering tasks.

Understanding SWE-bench Verified

SWE-bench is an evaluation benchmark that measures an AI agent's ability to resolve real-world GitHub issues from popular open-source Python repositories. Unlike competition-style coding tests, it requires the agent to understand a codebase, modify files, and verify the fix using unit tests.

SWE-bench Verified is a human-reviewed subset of 500 problems from the original dataset. This subset ensures that every task is solvable with the provided context, providing a more accurate measure of a coding agent's performance by removing impossible tasks.

The Agent Scaffold Design

Anthropic utilized a minimal "agent" system—a combination of the Claude 3.5 Sonnet model and software scaffolding—to maximize the model's own judgment and control. The scaffold avoids hardcoded workflows in favor of a flexible interaction loop.

Toolset and Interface

The agent is equipped with two primary tools:

  • Bash Tool: Executes bash commands in a persistent environment. It provides access to a mirror of common Linux and Python packages via apt and pip but lacks internet access.
  • Edit Tool (str_replace_editor): A complex tool for viewing, creating, and editing files. It uses a string replacement strategy (old_str to new_str) to ensure reliability; replacements only occur if there is exactly one match of the target string.

Prompting Strategy

The agent is guided by a prompt that suggests a general approach—exploring the repo, creating a reproduction script, editing source code, and verifying the fix—without enforcing strict transitions. Anthropic notes that encouraging the model to produce long, thorough responses can improve performance for those not constrained by token costs.

Performance Results

When using the same agent scaffold, the upgraded Claude 3.5 Sonnet significantly outperformed previous iterations and the prior state-of-the-art model.

Model SWE-bench Verified Score
Claude 3.5 Sonnet (new) 49%
Previous SOTA 45%
Claude 3.5 Sonnet (old) 33%
Claude 3 Opus 22%

Agent Behavior and Capabilities

Analysis of the model's logs indicates that the upgraded Claude 3.5 Sonnet exhibits stronger self-correction capabilities than previous models. It is more likely to attempt multiple different solutions when a first attempt fails, rather than repeating the same error.

In a typical workflow, the model:

  1. Explores the repository structure using the Edit Tool.
  2. Creates a standalone Python script to reproduce the reported bug.
  3. Executes the script via the Bash Tool to confirm the error.
  4. Applies minimal changes to non-test files using string replacement.
  5. Reruns the reproduction script to verify the resolution.

Technical Challenges in Software Engineering Evals

Anthropic identified four primary challenges when running the SWE-bench Verified evaluation:

  • Resource Intensity: Successful runs can require hundreds of turns and exceed 100k tokens, leading to high costs and long durations.
  • Grading Noise: System issues, such as environment setup errors or duplicate install patches, can lead to incorrect failure marks for otherwise correct model behavior.
  • Hidden Test Discrepancies: Because the model cannot see the grading tests, it may believe it has succeeded when it has either applied a "bandaid" fix instead of a refactor or failed to match the specific unit tests of the original human PR.
  • Multimodal Gaps: While Claude 3.5 Sonnet has vision capabilities, the current scaffold does not allow it to view files or URLs on the filesystem, which complicates debugging for libraries like Matplotlib.

Sources

Related