The AI Cost of JavaScript Frameworks: A Data-Driven Analysis

In the era of agentic coding, the web development community continues to make technology choices based on "vibes" rather than empirical evidence. To challenge this, a recent benchmark experiment evaluated 12 JavaScript frameworks by tasking AI agents (Claude 3.5 Opus) with building a non-trivial Pokémon application. The results suggest that framework choice significantly impacts development cost, token consumption, and final performance, even when developers are no longer writing the code by hand.

The Benchmark Methodology

The experiment focused on building a functional Pokémon app using the PokéAPI, requiring search and filtering capabilities. To ensure a fair comparison, the study measured:

• Cost: Total API spend per build.
• Efficiency: Time taken and total tokens consumed.
• Quality: Lighthouse performance, accessibility, and JavaScript bundle size.
• Maintainability: Total lines of code (LOC).

Agents were instructed to use the latest versions of each framework and were provided access to official MCP (Model Context Protocol) servers or agent skills where available. The experiment ran 3–5 iterations per framework to account for the stochastic nature of LLMs.

Key Findings: Cost and Efficiency

Frameworks that favor server-side rendering or simpler abstractions proved more efficient for AI agents. Hono+htmx and Astro emerged as the most cost-effective and fastest options, while heavier frameworks like Angular and SvelteKit incurred higher costs and longer build times.

Framework	Median Cost ($)	Median Time (min)	Tokens (M)
Hono+htmx	2.67	8.6	4.26
Astro	2.76	8.2	4.13
Next.js	3.14	10.9	4.99
SvelteKit	4.28	10.8	8.53
Angular	4.97	17.2	9.76

Performance and Maintainability

While Lighthouse scores for accessibility and performance were largely saturated across all frameworks, the JavaScript bundle size and Total Blocking Time (TBT) showed clear differentiation. MPA-based frameworks (like Astro and Hono+htmx) consistently delivered smaller bundles and lower TBT compared to SPA-heavy frameworks.

Regarding maintainability, measured by non-blank lines of code, Hono+htmx required approximately 1,370 lines, whereas Angular required over 2,000. This suggests that in an agentic workflow, the verbosity of a framework directly correlates to the token cost of future maintenance and updates.

Community Insights and Critiques

Discussion surrounding these results highlights a shift in how developers view "Developer Experience" (DX) versus "Agent Experience" (AX).

• The Case for Simplicity: Many commenters noted that frameworks like Hono and HTMX are inherently easier for agents to navigate because their abstractions are closer to standard web primitives.
• The Role of Training Data: Some observers pointed out that newer frameworks (like SvelteKit 3) may have performed worse due to lack of training data, forcing agents to read migration guides repeatedly, which inflated costs.
• Limitations of the Study: Critics argued that the Pokémon app is too simple to represent complex enterprise applications. As one commenter noted, "The point of using a framework is to reduce mental load at the expense of customizability... the moment you need to change something in the framework, this mental model falls apart."

"In the agentic era, when developers are no longer typing code by hand and are thus less constrained by DX, I still see vibes ruling the day... If you’ve gone largely hands-free, then what attachment do you really have to the choices you made back when you were typing code line-by-line like an animal?" — Nolan Lawson

Ultimately, this benchmark demonstrates that AI agents are sensitive to the architectural complexity of the tools they use. As agentic development becomes standard, the industry may shift toward frameworks that prioritize low-token overhead and clear, predictable patterns over traditional DX-focused abstractions.