Real-SWE Benchmark: Evaluating AI Coding Agents on Private Enterprise Codebases

The Real-SWE benchmark reveals that frontier AI coding agents struggle significantly with the complexities of private, real-world enterprise codebases. While synthetic benchmarks often show high success rates, Real-SWE demonstrates that the transition to proprietary systems—where code is not available on the public internet and business logic is deeply intertwined with infrastructure—drops resolution rates for all tested models to below 40%.

Model Performance and Leaderboard

Fable 5.1 leads the benchmark with a 38.8% resolution rate, followed by GPT-6 Astra and Gemini 3.8 Flash. The results indicate a substantial gap between AI capabilities and the requirements of a professional software engineer in a production environment.

Rank	Model	Harness	Resolution Rate
1	Fable 5.1	Claude Code	38.8%
2	GPT-6 Astra	Codex CLI	33.8%
3	Gemini 3.8 Flash	Gemini CLI	31.2%
4	GLM 5.3	Claude Code	28.8%
5	Grok 4.6	Grok Build	23.8%
5	Muse Spark 1.3	Muse Code	23.8%
7	Kimi K3	Kimi Code	18.8%
8	GPT-5.6 Sol	Codex CLI	16.2%

Resolution rates are calculated as pass@1, averaged over eight independent runs per task.

Why Enterprise Codebases are More Challenging

Real-SWE tasks are designed to be "out of distribution" for AI models because they utilize licensed private codebases from real companies. This prevents models from relying on training data found on the public internet.

Company-Specific Complexity

Enterprise engineering requires more than just syntax knowledge; it requires an understanding of internal coding patterns and business rules. For example, one task involves fixing invoice billing where the agent must navigate a NestJS/TypeScript service and interact with external services like TaxJar and InfluxDB to handle tax exemptions and VAT registrations based on specific business configurations.

Cross-Functional Implementation

Real-world changes are rarely contained within a single file. The median number of files edited in a Real-SWE reference solution is 11, compared to 6 in other benchmarks like FrontierCode and DeepSWE. This requires agents to maintain a mental map of the system across multiple services and infrastructure components, including AWS, Kubernetes, PostgreSQL, and Redis.

Analysis of Model Failures

Failure analysis shows that the most common cause of failure is "missed requirements," followed by "unverified assumptions." This suggests that while models can write code, they struggle to fully grasp the business intent and verify their changes against the existing system.

Task-Specific Variance

Performance varies wildly by task. While some tasks like "Multi-region sweep" have a 67.2% resolution rate, others such as "Analytics stream reducer" have a 0.0% resolution rate across all tested models. This indicates that certain types of complex business logic or architectural patterns remain entirely unsolvable for current frontier models.

Rollout Duration and Cost

There is little correlation between the time spent on a task and the success rate; 71.4% of rollouts under 10 minutes failed, compared to 73.4% of longer rollouts. Estimated costs per rollout range from $2.50 (Gemini 3.8 Flash) to $6.96 (Fable 5.1).

Community Insights and Counterpoints

Discussion among engineers on Hacker News highlights a divide between the benchmark's results and individual user experiences.

Tooling and Context

Some users argue that the success of an agent depends more on the harness and the "context map" provided to the model. One user suggested that building a "code atlas"—a semantically queryable map of codebase connections—can dramatically reduce the time models spend reading code and improve their awareness of nuances.

Methodology Concerns

Critics of the benchmark pointed to the lack of reproducibility and the potential for "pinky-promise benchmarking" since the codebases are private.

"TL;DR benchmarking in a completely non-reproducible manner? 'Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company'."

Additionally, some users questioned whether the private codebases were truly private or if they had leaked into the training sets of the frontier models, suggesting that contamination is a constant risk in modern LLM evaluation.

Practical Observations

Several users noted that Gemini 3.8 Flash's high performance in this benchmark aligns with their own experiences, describing it as a "hidden gem" for solving issues in well-defined enterprise environments, even if it struggles with open-ended questions.