Parsimony: an open-source benchmark for how much code AI coding agents add
Parsimony is an open-source benchmark that measures how much code an AI coding agent adds when solving a task, rather than only whether it solves the task. It reuses patches that existing benchmarks already publish, counts syntax-level "coding units" instead of lines, and subtracts deleted code, producing a net figure that can be compared across models on leaderboards for DeepSWE Python, JavaScript, TypeScript and Go, SWE-bench Verified, SWE-bench Live Lite, and SWEeper-Bench PHP.
The project's stated motivation is that existing benchmarks measure solve rate and cost but do not punish excess code, which makes a codebase harder to work with in the long term.
How Parsimony scores a patch
Parsimony counts coding units — syntax elements such as statements, calls, comparisons, names and literals — rather than lines of code. Counting syntax elements means that cramming code onto fewer lines is not rewarded, and because deleted code is subtracted, the reported figure is a net value: net units = added − removed.
Parsimony does not execute anything. Each published patch is applied to the task's code, and both versions of every changed file are parsed. Tests, documentation and generated files are excluded from the count. Pass/fail results are taken from the source benchmark's published results; tests are not rerun.
The project's Method section states that "only Python is measured," even though the site also publishes JavaScript, TypeScript, Go and PHP leaderboards. That is an apparent inconsistency in the source page rather than a documented limitation.
Leaderboards and reproducibility
Parsimony publishes seven leaderboards: DeepSWE Python, DeepSWE JavaScript, DeepSWE TypeScript, DeepSWE Go, SWE-bench Verified, SWE-bench Live Lite (listed as "Live Python"), and SWEeper-Bench PHP.
Each leaderboard links to a report and a manifest in the project's GitHub repository, and each ships SHA-256 hashes for both the manifest and the dataset. For example, the DeepSWE Python manifest hash is c16b15f80c140ffb4f0a548886962dda4173a02893a65b7faa1d6796940ecf65 and its dataset hash is 41a44998c70248e0a3e2a4cf43a8838b70248537d1f3a458838f2f7e09ecef8a. The self-run leaderboard also records the DeepSWE commit used: 0b9fabbb63b9104d678fe965e1632f2dd9eaa2ea.
DeepSWE Python results: 26 models on 34 tasks
The main DeepSWE Python board covers 26 models on 34 DeepSWE tasks, with 136 attempts per model (four attempts per task). Models are ranked by mean net units added on the tasks they solved, lowest first. The site also reports a second mean that includes failed attempts, a solve rate taken from DeepSWE, and the Artificial Analysis Intelligence Index dated 2026-10-08, which is an outside score matched by model and reasoning effort and does not affect the ranking.
The site's table does not label its numeric columns; the labels below follow the site's own explanation of the ranking metric and the solve/measured counts.
| Rank | Model | Mean net units (ranking metric) | Mean net units incl. failed attempts | Solve rate | AA Intelligence Index | Measured |
|---|---|---|---|---|---|---|
| 1 | Gemini 3.1 Pro Preview (high) | 1479.5 | 1156.3 | 8.1% | 29.7 | 11/11 |
| 2 | Claude Fable 5 (xhigh) | 1557.5 | 1509.1 | 74.3% | — | 97/101 |
| 3 | Gemini 3.5 Flash (high) | 1621.5 | 1707.6 | 29.4% | 32.6 | 38/40 |
| 4 | Claude Opus 4.8 (max) | 1621.9 | 1592.2 | 63.2% | 41.8 | 83/86 |
| 5 | Kimi K3 (max) | 1641.2 | 1601.1 | 67.6% | 43.6 | 89/92 |
| 6 | Claude Sonnet 5 (max) | 1665.9 | 1563.5 | 59.6% | 38.2 | 78/81 |
| 7 | Claude Sonnet 4.6 (high) | 1684.5 | 1587.9 | 29.4% | — | 36/40 |
| 8 | Qwen3.8 Max (xhigh) | 1697.8 | 1770.5 | 55.1% | 45.4 | 74/75 |
| 9 | GLM-5.2 (max) | 1733.2 | 1591.3 | 41.9% | 33.7 | 53/57 |
| 10 | Kimi K2.7 Code | 1737.3 | 1600.1 | 24.3% | 25.8 | 32/33 |
The bottom of the ranked board is occupied by Muse Spark 1.1 (xhigh) at rank 26 with a mean of 2562.3 net units and a 52.9% solve rate, and Muse Spark 1.2 (xhigh) at rank 25 with 2484.1 net units and a 50.7% solve rate. Claude Opus 5 (max) sits at rank 23 with 2183.4 net units and a 74.3% solve rate.
Two models on the board have no Artificial Analysis score: Claude Fable 5 (xhigh) and Claude Sonnet 4.6 (high).
Self-run models are shown but not ranked
Parsimony also displays a faded set of models that the author ran personally on the first 3–10 DeepSWE Python tasks, using pi for OpenAI models and Claude Code for Anthropic models, graded with each task's own tests. These runs are explicitly not verified and not ranked.
In that set, GPT-6 Luna (high) shows the lowest mean at 1076.3 net units with a 15.0% solve rate over 3/3 measured attempts, and GPT-6 Astra (high) shows 1103.0 net units with a 66.7% solve rate over 4/4 measured attempts. GPT-5.6 Sol (high) appears in both the self-run set and the published board; the site notes that the author's run added 0.88× the code of its published run.
The stated limits of the metric
Parsimony's own Limits section is unusually candid about what the number does not mean.
- Smaller is not always better: units measure size, not readability or design.
- Models that write more code also tend to solve more tasks, with a Spearman correlation of 0.33–0.60 on the boards that include 26 or more models.
- Every model uses the same agent, mini-SWE-agent, but the agent still affects how much code gets written.
The site illustrates the size-versus-readability distinction with two implementations of the same function. A loop-based has_negative that sets a found flag and a one-line any(value < 0 for value in values) version return the same result, but only one of them is short.
Community reaction: parsimony versus readability
The Hacker News discussion pushed back on treating a low unit count as a proxy for good code. The most direct objection was that the model winning the author's own unranked runs produces code that is hard to read.
"On the one hand, I absolutely agree that we should try to measure and reduce verbosity that LLMs are slopping into our codebases. On the other hand, the fact that Astra is 'winning' this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra." — fastball
A second thread asked for practical techniques rather than measurement. One commenter asked directly whether anyone has ways to reduce the complexity and verbosity of code generated by agents, framing it as a code-quality problem rather than a benchmarking problem.
A third theme was metric coverage. One commenter asked for cyclomatic complexity as a supporting readability metric, saying they increasingly find AI-adjacent and AI-written code suffering from it. Another questioned whether fewer coding units actually means less review work for the human who has to read the patch.
A fourth theme was model coverage. One commenter argued that benchmarks should include the best coding and writing model that ordinary people can run locally — naming Qwen 3.8 27B at roughly 28GB of RAM in Q4 as the current candidate — so that the gap between open weights and leading commercial models is visible. The same commenter noted that local models are already better than what was available just over a year earlier, and drew the same comparison to text-to-image and text-to-speech models.
One commenter dismissed the premise entirely, arguing that code volume will stop mattering once people understand the scale at which a vector space operates, and that complexity is not much of an issue for an AI system.
A commenter who said they would bookmark the benchmark framed the value in measurement terms: you cannot manage what you do not measure, and moving the measurement to the abstract syntax level removes a layer of noise from comments and formatting. They added that they would like to hear more lived experience, since heuristics in this space still leave room for optimizing code generation and pipelining.
Adding your own benchmark
Parsimony accepts external results. A user downloads the example file, fills in their own code-size numbers, and opens it to get the same chart and table; the file stays in the browser. The format and sharing instructions are documented in the project's repository under docs/custom-benchmarks.md.
Sources
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch