SWE-bench/SWE-smith

[NeurIPS 2025 D&B Spotlight] Scaling Data for SWE-agents

What it solves

SWE-smith is designed to solve the data scarcity problem for training software engineering (SWE) agents. It provides a toolkit to scale the production of high-quality training data by automatically turning GitHub repositories into interactive training environments (SWE-gyms) and synthesizing unlimited task instances for agents to practice on.

How it works

The toolkit uses Docker to create isolated execution environments for various repositories. It follows a four-step process to build datasets:

  1. Environment Construction: Creating a Docker-based execution environment for a repository.
  2. Task Synthesis: Automatically generating task instances (such as program repair or file localization).
  3. Task Validation: Filtering tasks to ensure they are functional by keeping only those that break at least one unit test.
  4. Issue Generation: Creating the descriptive issue text that an agent must solve.

Who it’s for

It is primarily for AI researchers and developers building and training Large Language Models (LLMs) to act as autonomous software engineers, specifically those using the SWE-agent framework.

Highlights

  • Scalable Data Generation: Ability to turn any GitHub repository into a training ground for SWE agents.
  • Curation of Resources: Provides a dataset of 52k task instances and 26k trajectories.
  • Proven Performance: Used to fine-tune Qwen 2.5 Coder into SWE-agent-LM-32B, resulting in a +32% jump on SWE-bench Verified.
  • RL Support: Compatible with reinforcement learning techniques like GRPO via SkyRL.