Claude 3.5 Sonnet SWE-bench Performance

The upgraded Claude 3.5 Sonnet has achieved a 49% success rate on SWE-bench Verified, surpassing the previous state-of-the-art score of 45%. This improvement is driven by the model's enhanced reasoning, coding, and mathematical abilities, combined with a minimal agent scaffolding designed to maximize the model's autonomy.

SWE-bench Verified Overview

SWE-bench is an evaluation benchmark that tests an AI's ability to resolve real-world GitHub issues from popular open-source Python repositories. Unlike competition-style coding tests, it requires the model to understand, modify, and test code within a local working copy of a repository.

SWE-bench Verified is a 500-problem subset of the original dataset, human-reviewed to ensure every task is solvable without external context. This benchmark evaluates an "agent"—the combination of the AI model and the software scaffolding (prompts and tool-parsing logic) surrounding it.

Agent Architecture and Design Philosophy

Anthropic's agent scaffold for Claude 3.5 Sonnet focuses on minimal scaffolding to give the language model maximum control over its problem-solving process. The agent operates using a prompt and two primary tools:

Tool-Using Agent Components

  • Bash Tool: Executes bash commands in a persistent environment. The tool's description provides critical instructions on escaping inputs, the lack of internet access, and how to run long-lived commands in the background.
  • Edit Tool (str_replace_editor): A complex tool for viewing, creating, and editing files. It supports commands such as view, create, str_replace, insert, and undo_edit.

Key Optimization Strategies

  • String Replacement for Edits: The agent uses a string replacement method where the model specifies an old_str to be replaced by a new_str. The replacement only occurs if there is exactly one match of old_str, reducing reliability issues associated with other editing strategies.
  • Error-Proofing: To prevent issues with relative file paths, the Edit Tool requires absolute paths for all operations.
  • Model Autonomy: The prompt suggests a general approach (exploration, reproduction, editing, and verification) but does not force a strict workflow, allowing the model to use its own judgment.

Performance Results

When using the same agent scaffold, the upgraded Claude 3.5 Sonnet significantly outperforms previous models and the prior state-of-the-art:

Model SWE-bench Verified Score
Claude 3.5 Sonnet (new) 49%
Previous SOTA 45%
Claude 3.5 Sonnet (old) 33%
Claude 3 Opus 22%

Agent Behavior and Capabilities

Analysis of the model's logs indicates that the upgraded Claude 3.5 Sonnet exhibits stronger self-correction and a greater willingness to attempt multiple different solutions rather than repeating the same mistake.

In a typical workflow, the model:

  1. Explores the repository structure using the Edit Tool.
  2. Creates a reproduction script to confirm the bug.
  3. Executes the script using the Bash Tool to observe the error.
  4. Modifies the source code using the Edit Tool.
  5. Reruns the reproduction script to verify the fix.

Implementation Challenges

Anthropic identified four primary challenges in running the SWE-bench Verified evaluation:

  1. Cost and Duration: Successful runs can take hundreds of turns and exceed 100k tokens, making the process expensive and time-consuming.
  2. Grading Accuracy: System-level issues, such as environment setup errors or duplicate install patches, can lead to incorrect failure ratings.
  3. Hidden Tests: Because the model cannot see the grading tests, it may believe it has succeeded when it has actually failed, sometimes by applying a surface-level fix rather than a deeper refactor.
  4. Multimodal Limitations: While the model has vision capabilities, the current agent implementation does not allow it to view files or URLs, which complicates debugging for tasks involving visual output (e.g., Matplotlib).

Sources

Related