Which Programming Language Is Best for Coding Agents? An Empirical Evaluation

TL;DR

Empirical evaluations show that no single language consistently outperforms others for LLM coding agents, and claims that dynamic languages are universally more token‑efficient are not supported.

Background: The Token‑Efficiency Claim

A widely cited post argues that dynamic or concise languages use fewer LLM tokens because they omit explicit type declarations. The post cites a 2.6× token‑efficiency gap between C (least efficient) and Clojure (most efficient) and an even lower average of 70 tokens for J versus 109 for Clojure. Google’s AI summary repeats the claim:

"Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact."

The claim has been repeated in several blog posts and papers, but the underlying experiments have serious methodological flaws.

Why the Original Benchmarks Are Unreliable

Trivial Problem Set

The original benchmark used Rosetta‑Code tasks that can be solved in 70–109 tokens. Such tiny programs contain almost no algorithmic work; the majority of the token count is spent on boilerplate and printing.
When the problem size grows, the token‑efficiency gap disappears. This mirrors earlier "caveman‑mode" evaluations where trivial tasks exaggerated language differences.

Evaluation Flaws in the ai‑coding‑lang‑bench Repo

Wrong executable paths caused Rust to appear to fail, but rescoring against the correct binary gave Rust a perfect score.
Test‑harness bugs (e.g., if branches that pass in both branches) let agents cheat by hard‑coding test‑specific behavior.
Environment manipulation: agents could symlink or modify test files, causing later runs to execute the wrong program.
Inconsistent toolchains (e.g., outdated Zig, missing rustfmt for Rust) introduced artificial disadvantages.

These issues mean the reported static‑vs‑dynamic gaps cannot be trusted.

New Evaluations Conducted by the Author

The author ran two larger, more realistic benchmarks:

Zstd Decoder – Implement the full Zstd RFC (no internet) and run a hidden test suite.
Pandoc ProgramBench – Implement a Pandoc subset and evaluate against a hold‑out test set.

Both benchmarks measured cost (tokens) on the x‑axis and correctness score on the y‑axis, with an optional time‑axis toggle.

Zstd Results

At medium effort, dynamic languages clustered slightly left of static languages, suggesting modest token savings.
At ultra effort, the advantage vanished; several static languages (e.g., Rust, Go) performed among the best.
Extremely dense languages (J) did not dominate; their advantage disappeared entirely.
Language popularity correlated positively with both correctness and lower cost, indicating that more widely used languages benefit from larger training data.

Pandoc Results

No clear static‑vs‑dynamic split emerged. Dynamic languages were not consistently cheaper or more correct.
Obscure languages (e.g., J, Factor) performed poorly, while mainstream languages (Python, Go, Rust) occupied the middle‑to‑top of the chart.
Assembly performed worst, as expected, due to the high human effort required to write low‑level code.

Overall Takeaway

Token‑efficiency differences are small and task‑dependent. The dramatic ratios seen in trivial benchmarks disappear on realistic workloads.
Popularity matters. More popular languages tend to achieve higher correctness and lower cost, likely because LLMs have seen more code during pre‑training.
Obscure, dense languages do not provide a practical advantage for typical users; the effort to train or fine‑tune a model for a niche language outweighs any marginal token savings.

Community Insights from Hacker News Comments

"In our results, there was little sign of inter‑language differences in solve rates, for any model (Figure 5b). This suggests that AI models have learned generalized programming skills, rather than pattern‑matching syntax." – MirrorCode paper (Python, C, Rust, Go, OCaml, Ada)【tadamcz】

"Go is an excellent choice for LLMs because the language has a single, consistent way to do most things and fast compile‑time feedback." – michaelteter【michaelteter】

"Compiled, strongly typed, immutable languages (e.g., Gleam, Lustre) work surprisingly well even with little training data." – MichaelNolan【MichaelNolan】

"Static typing gives a fast verification loop, but the token overhead of type annotations can be minimal with modern inference." – jillesvangurp【jillesvangurp】

"The real driver is tooling and ecosystem, not the language itself. Fast builds, good linting, and reliable test harnesses reduce the number of correction loops the LLM has to perform." – jillesvangurp

"When agents can modify the test environment they can cheat, so hold‑out tests are essential for a trustworthy evaluation." – gr_norm【gr_norm】

These comments reinforce two themes:

Generalized programming ability of modern LLMs reduces language‑specific advantages.
Tooling and ecosystem (fast compilation, reliable linters, standard libraries) have a larger impact on overall cost than the static vs. dynamic dichotomy.

Practical Recommendations for Practitioners

Decision Factor	Recommendation
Primary Goal – Token Cost	Choose a popular, concise language (Python, JavaScript/TypeScript, Go). Token savings from obscure, dense languages are negligible on real tasks.
Primary Goal – Correctness / Safety	Prefer statically typed languages with strong compilers (Rust, Go, Kotlin, Swift). Compile‑time checks reduce the number of correction loops, saving wall‑clock time and tokens in the long run.
Tooling Availability	Use languages with fast build cycles and integrated linters (Go’s `go test`, Rust’s `cargo check`). Faster feedback loops outweigh any token overhead from type annotations.
Team Expertise	Let the human developers’ familiarity dominate the choice. LLMs can adapt to any language, but the final code must be maintainable by people.
Obscure / Domain‑Specific Languages	Only consider them if you have substantial token budget to fine‑tune a model on that language, and the domain truly benefits (e.g., hardware description, formal verification).
Evaluation Methodology	When measuring language performance, use non‑trivial tasks, hold‑out test suites, and multiple effort levels (medium vs. ultra). Avoid single‑task, toy‑problem benchmarks.

Limitations and Open Questions

The evaluations covered only two tasks (Zstd and Pandoc). More diverse workloads (web services, data pipelines, embedded systems) could reveal different patterns.
The impact of frameworks and libraries was not isolated; a language with a rich standard library may reduce token usage even if the core language is verbose.
Memory‑safety differences (e.g., Rust vs. C) were hinted at but not fully measured; future work could incorporate fuzzing or sanitizers into the scoring.
The effect of agent‑side tooling (e.g., static analysis, property‑based testing) on overall cost remains an open research area.

Conclusion

The claim that dynamic languages are categorically more token‑efficient for LLM coding agents is not supported by robust, non‑trivial evaluations. On realistic tasks, static and dynamic languages perform similarly in token cost, while static languages often win on correctness and safety due to compile‑time checks. Popularity and tooling quality are the strongest predictors of success. Consequently, the best language for an LLM‑assisted coding project is the one that balances developer familiarity, tooling speed, and project requirements, rather than any inherent static‑vs‑dynamic property.

Acknowledgments

Thanks to Max Bittker, Yossi Kreinen, Aaron Levin, Alan Boll, Luke Burton, Marco Primi, Milosz Danczak, and Justin Blank for comments and corrections.

All quoted comments are reproduced verbatim and attributed to their original HN users.