Opus 5 24% Strict Pass on SlopCodeBench – Modest Gain, Persistent Code Quality Issues

Opus 5 scores 24 % strict pass on a subset of SlopCodeBench, but still falls far short of reliable autonomous coding

Takeaway: In a six‑hour experiment Opus 5 achieved a 24 % strict‑pass rate on 17 checkpoints across three SlopCodeBench problems—only a modest gain over Opus 4.6’s 17 % from the original paper—while inflating code volume five‑fold and triggering slop warnings on >90 % of its lines. The results suggest that today’s frontier models cannot yet maintain a codebase over iterative specifications without human steering.

Benchmark context and why it matters

SlopCodeBench (SCB) is a newly released (Mar 2026) longitudinal coding benchmark that reveals requirements incrementally, forcing a model to evolve a codebase across multiple checkpoints rather than solving a static problem. This design addresses a key limitation of prior benchmarks, which disclose the entire task up front and therefore do not test a model’s ability to preserve code quality over time.

The benchmark reports a suite of deterministic quality metrics—size, cyclomatic complexity, duplication, decomposition, lint violations, and dependency‑graph entropy—computed after each checkpoint. A strict pass requires every new and inherited test to succeed, meaning a single defect at any stage invalidates all later checkpoints.

Experimental setup

  • Models evaluated: Claude Opus 4.8, Claude Sonnet 5, and Claude Opus 5.
  • Problems selected: three from the SCB repository—circuit_eval (easy, 8 checkpoints), database_migration (medium, 5 checkpoints), and dynamic_config_service_api (hard, 4 checkpoints), totaling 17 checkpoints.
  • Execution: Each model ran in parallel with a fresh context window per checkpoint, using the official Claude code harness and identical prompts.
  • Success metric: Strict pass rate (all tests, including regressions, must pass).

Core results

Model Strict passes Pass rate
Opus 5 4 / 17 24 %
Opus 4.8 1 / 17 6 %
Sonnet 5 1 / 17 6 %
  • Opus 5’s passes were the first three checkpoints of circuit_eval and checkpoint 1 of database_migration.
  • No model completed any problem without defects; all accumulated at least one failure that persisted to later checkpoints.
  • Opus 5 wrote 29 k source lines—about three times the volume of the other models—yet only ~1.8× more production code after excluding tests.
  • Slop warnings were triggered on 93 % of Opus 5’s lines (vs. 98 % for Opus 4.8 and 89 % for Sonnet 5).

"every dollar bought correctness. nobody bought enough of it." – author’s comment on cost vs. defects

Code quality trends

Verbosity explosion

Opus 5 generated five times as many functions/callables as Opus 4.8 across the same checkpoints. However, only 14.9 % of its functions were single‑use, compared with 49.1 % for Opus 4.8 and 71.5 % for Sonnet 5.

Complexity growth

All models showed rising cyclomatic complexity over checkpoints. Opus 5 maintained the lowest mean complexity (≈2.5) but did so by creating many small functions, whereas Opus 4.8 kept fewer functions that grew to a maximum complexity of 93.

Duplication and slop density

Duplication percentages rose for Opus 4.8 (4.6 % → 16.8 %) but remained roughly flat for Opus 5 (≈2.4 %). When mapping SCB slop detectors onto a 1 kLOC slice of a production TypeScript monorepo, Opus 5’s code exhibited 11.6× the slop trigger density.

Community insights

  • @dcl: "finally the benchmark for me" – acknowledges SCB fills a missing evaluation niche.
  • @Johnny_Bonk: "Opus 5 is not much of an improvement" – echoes the author’s observation of modest gains.
  • @killingtime74: "Did you not benchmark GPT 5.6 or GLM 5.1? I can run them" – suggests broader model coverage would be valuable.
  • @Vgoose: "SCB is the first benchmark to aim at non‑functional, longitudinal requirements" – highlights the benchmark’s relevance as point‑in‑time solves become routine.
  • @robbomacrae: "SCB mirrors real software development because the agent must keep code clean" – reinforces the practical importance of strict‑pass metrics.
  • @sothatsit: "Opus 5 is a nice improvement over Opus 4.8 but not revolutionary" – aligns with the quantitative findings.
  • @delbertty: "Mostly a harness problem; constraining the agent to add rather than edit reduces slop" – points to prompt/harness design as a lever for quality.
  • @wilcoKruijer: "Reducing complexity should be the number‑1 priority" – proposes a future benchmark focus.

What the numbers tell us about autonomous coding

  1. Strict‑pass rates remain low – Even the best model (Opus 5) passes only a quarter of checkpoints, confirming that current agents cannot be trusted to evolve a codebase unattended.
  2. Higher correctness comes at massive verbosity – Opus 5’s extra tests and scaffolding increase line count dramatically without proportionate gains in pass rate.
  3. Slop metrics are pervasive – >90 % of generated lines violate at least one SCB rule, indicating that models routinely produce code that is hard to maintain.
  4. Complexity is inevitable – All models’ mean cyclomatic complexity rises, suggesting that without explicit refactoring prompts models tend to accumulate technical debt.

Recommendations for practitioners

  • Do not rely on a single model run – Use human‑in‑the‑loop review or a secondary “refactor” model to catch defects before they propagate.
  • Incorporate quality back‑pressure – Prompt the model to check cyclomatic complexity, duplication, and slop rules after each checkpoint and to refactor before proceeding.
  • Track cost vs. correctness – The experiment shows diminishing returns; spending more tokens does not guarantee higher strict‑pass rates.
  • Consider hybrid pipelines – Let a strong model generate initial checkpoints and hand off later checkpoints to a smaller, more disciplined model (as illustrated in the paper’s “handoff” diagram).

Future directions for SlopCodeBench

  • Human baseline – Adding a human‑coder reference would contextualize the 24 % pass rate.
  • Broader model roster – Including GPT 5.6, GLM 5.1, Fable, and Sol would clarify whether the observed trends are model‑specific.
  • Prompt engineering studies – Experiments with “plan‑first” or adversarial review loops could quantify the impact of steering on strict‑pass outcomes.
  • Language expansion – Porting the slop detectors to TypeScript, Rust, or Go would test whether Python’s higher slop density is language‑dependent.
  • Statistical rigor – Reporting confidence intervals for cost and quality metrics would address concerns about overlapping error bars.

Bottom line

Opus 5’s 24 % strict‑pass rate on a small SCB subset demonstrates modest progress over Opus 4.6 but also underscores persistent shortcomings: excessive verbosity, pervasive slop, and inevitable complexity growth. Until models can consistently maintain a clean, defect‑free codebase across incremental specifications, developers must continue to provide steering, review, and refactoring guidance.

References

Sources

Related