Netlify Agent Runners: Comparing 11 AI Models for Web Development
High-end models provide superior design but at significantly higher credit costs
Netlify's evaluation of 11 AI models reveals a stark contrast in both visual output and resource consumption. Top-tier models like Claude Opus 5 produce highly detailed, aesthetically polished sites with custom vector graphics and dark mode support, but they can consume up to 1,055 credits per run—orders of magnitude more than budget-friendly alternatives. In contrast, models such as DeepSeek V4 Flash 0731 can generate functional sites for as little as 1.3 to 3.4 credits.
Model Performance Breakdown: Design vs. Cost
Netlify tested these models using a simple prompt to build a one-page site for a neighborhood coffee shop. The results highlight a trade-off between "turnkey" high-fidelity design and iterative, low-cost development.
Frontier Closed-Source Models
- Claude Opus 5: The gold standard for visual detail and self-validation. It produces the most sophisticated designs but exhibits high variance in credit usage, occasionally spending 4x more than its own baseline.
- Claude Sonnet 5: A mid-tier option that maintains some detail but produces simpler vector graphics and less content than Opus.
- GPT 5.6 Sol (Low Effort): Outperforms Claude Sonnet 5 in basic design intuition and content richness while remaining economically competitive.
- GPT 5.6 Terra: Offers a distinct visual language that is simpler than Sol but often avoids the "AI-generated" aesthetic, though it occasionally suffers from visual glitches like missing images.
- Gemini 3.6 Flash vs. 3.1 Pro: There is a generational leap between these two. Gemini 3.6 Flash produces modern, content-rich results, whereas Gemini 3.1 Pro produces bare-bones pages that strictly follow the prompt without any creative expansion.
Open-Weight and Specialized Models
- DeepSeek V4 Flash (0731): The most cost-efficient model tested, averaging 2.4 credits. It produces surprisingly varied results, with some runs mimicking mid-tier closed models.
- GLM 5.2: A low-cost option (average 27 credits) that produces highly varied outputs across different runs, suggesting that multiple iterations are necessary to find the best result.
- Kimi K3 & K2.7 Code: While Kimi K3 is designed for long-horizon agentic tasks, it does not particularly shine in short-form design tasks. Kimi K2.7 Code is extremely cheap (19 credits) but offers minimal design or content value.
- DeepSeek V4 Pro: Less inspiring results compared to GPT 5.6 Terra at a similar price point, with occasional technical failures such as broken image links.
Technical Implementation: Agent Runners and AXIS
Netlify utilizes Agent Runners, which integrate full coding agents rather than pared-down versions. These agents are equipped with specific skills and project context to utilize Netlify-specific primitives, such as Netlify Database, AI Gateway, and Netlify Blobs.
To maintain quality, Netlify uses AXIS, an open-source evaluation framework. AXIS scores models based on functional correctness—such as whether a database is correctly implemented when required—rather than visual design. Models that fail these functional checks or exhibit inflated credit costs are excluded from Agent Runners.
Community Perspectives and Critical Analysis
Discussion among developers on Hacker News suggests that while these "one-shot" prompts are useful for ideation, they may not represent professional software development workflows.
Key Critiques
- Prompt Realism: Some developers argue that professional work involves detailed, iterative instructions rather than two-sentence prompts. One user noted:
"This 'oneshot from a simple prompt' eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application."
- Statistical Significance: Critics pointed out the small sample size (N=3), noting that probabilistic models require more runs to establish a reliable performance baseline.
- Mobile Optimization: A critical gap in the evaluation was the lack of focus on mobile-first design. Analysis by users showed that some high-credit models produced sites that were visually impressive on desktop but ineffective or slow to load on mobile devices.
- Utility of Simplicity: Some users preferred the output of cheaper models (like Gemini 3.1 or DeepSeek V4), arguing that "no-frills" text-based sites are easier to parse for actual customers than "stylistic" AI designs.
Summary of Model Credit Usage
| Model | Avg Credits | Notable Characteristic |
|---|---|---|
| Claude Opus 5 | 519 | Highest detail, highest cost variance |
| Claude Sonnet 5 | 143 | Mid-tier balance |
| GPT 5.6 Sol (Low Effort) | 141 | Strong design intuition |
| Gemini 3.6 Flash | 103 | Modern, content-rich |
| Kimi K3 | 102 | Optimized for long-horizon tasks |
| Gemini 3.1 Pro | 53 | Minimalist, strictly prompt-adherent |
| GPT 5.6 Terra | 39 | Distinct, less "AI-feeling" style |
| DeepSeek V4 Pro | 37 | Occasional technical glitches |
| GLM 5.2 | 27 | High variance in output style |
| Kimi K2.7 Code | 19 | Extremely low cost, low design value |
| DeepSeek V4 Flash (0731) | 2.4 | Maximum efficiency, surprising variety |
Sources
Related
- Dispatch
- Dispatch
- Dispatch
- Dispatch
- Dispatch