Armature Study Finds Coding Agents Prefer Neon, Stripe, and In‑House Solutions Across 17k Tool Selection Runs

Quick Takeaways

  • Across 16,893 tool‑selection sessions, Claude Code, Codex, and Cursor consistently favored Neon (database), Stripe (payments), and in‑house implementations, with only 42% of tool choices agreeing across agents.
  • Agent behavior is driven by source of information (web search vs. priors), repository context, and how vendor information is presented, leading to large discrepancies between mentions and actual selections.

Experiment Overview

Armature ran the largest known study of coding‑agent tool selection, observing 16,893 runs involving three agents (Claude Code, Codex, Cursor) across 75 repositories, 10 programming languages, and four persona types (vibe‑coder, junior, senior, enterprise engineer). Each run consisted of a realistic task (e.g., “pick a database”) followed by a simulated human orchestrator (Gemini 3.7 Flash) that approved the top recommendation and asked the agent to implement it. The study filtered the raw data to 5,292 valid sessions for public release.

Agent Information Sources Shape Decisions

  • Cursor relied on web search in roughly two‑thirds of sessions.
  • Codex performed web searches in 94% of sessions, typically using domain‑specific operators like site:auth0.com to focus on trusted sources.
  • Claude Code used its internal priors for most queries, searching the web only ~30% of the time, but when it did, it examined three times more pages than Codex. In domains where its priors were weaker (e.g., sandbox tooling), Claude searched the web ~80% of the time.
  • Agreement Rate: All three agents chose the same tool in only 42% of category‑persona‑repo cells. For voice‑agent tooling, Claude chose Twilio, Codex chose OpenAI Realtime API, and Cursor chose Vapi.
  • In‑House Preference: Claude built custom solutions in 19% of runs, nearly double the 10% rate for Codex and Cursor.

Repository Context Drives Different Winners

  • The same email‑sending request produced four different providers depending on language and repo:
    • TypeScript → Resend (55/89 runs)
    • Python → SendGrid (22/24 runs)
    • Go → Postmark (20/24 runs)
    • Java → Azure ACS (22/23 runs)
  • Vercel dominated TypeScript/Next.js projects (100% when Next.js was present) but never appeared in Python repositories, where Render was the top choice.

High Mention, Low Selection Phenomenon

Many well‑known services are frequently cited but rarely selected:

  • PayPal: mentioned 139 times, never chosen (Stripe won 124 of those sessions).
  • Adyen: mentioned 175 times, selected only 3 times.
  • LangChain: 194 mentions, selected 4 times.
  • Netlify: 152 mentions, selected 6 times.
  • Supabase: most‑cited database (242 mentions) but Neon won the majority of those sessions.

Presentation Details Can Flip Outcomes

  • Mailgun lost to Postmark when agents read “1‑day retention” on Mailgun’s free tier.
  • Supabase often lost because its bundled BaaS features (auth, storage, realtime) were presented as part of a single pricing plan, whereas agents were looking for a pure database.
  • In 5,292 sessions, 388 references to platform‑management overhead and 195 references to cost correlated with different winners, indicating that subtle wording influences agent choices more than raw price numbers.

Market Dominance vs. Competition

  • Payments: Stripe captured 90% of selections, losing only in EU‑specific regulatory cases where Paddle or Mollie were chosen.
  • Databases: Neon won 66% of selections, followed by cloud‑native options (Azure, AWS).
  • File Storage: Amazon S3 led with 45% share; Azure and GCP each held ~20%.
  • Email Services: Resend (35.6%) and Postmark (27.4%) were the top two winners.

Community Insight Highlights

"I keep telling people that we are living in the golden age of AI – like the first year or two of Google. It is all downhill as these companies push for profit and lock‑in." – natnatenathan

"Armature is on to something. Analyzing agent choices lets you tilt the market toward your product, just like traditional marketing does for humans." – ttul

"Claude Code keeps using awk, sed, and even Python for basic file editing – anyone know why this changed with the 5 series?" – IgorPartola

These comments underscore the broader industry concern that agent‑driven tool selection could become a new battleground for vendor lock‑in and SEO‑style optimization.

Implications for Developers and Vendors

  • Developers should treat agent recommendations as a starting point, not a final verdict, especially when the agent’s source (web vs. priors) is unknown.
  • Vendors can improve selection rates by simplifying pricing language, highlighting core features, and ensuring their documentation is easily discoverable by web‑search‑heavy agents like Codex.
  • Tool Builders may consider providing clear, concise “agent‑friendly” snippets (e.g., site:example.com‑friendly pages) to increase the likelihood of being chosen.

Access to Full Data

All traces, prompts, decision logs, and code diffs are publicly available at the Armature blog post. Researchers can download the dataset to conduct deeper analyses or to validate the findings presented here.

The study was funded by Armature, a YC‑backed growth‑services firm for developer tools. The authors disclose a conflict of interest but note that the methodology and raw data are openly shared for independent verification.

Sources

Related