Momentic Mo Launch: AI QA Agent Finds Bugs in Your App (QA Bench vs Codex, Claude Code, Devin)

Momentic's Mo has exited beta and is now generally available as an agentic QA engineer that drives a real application end to end, looks for bugs, and files what it finds. The launch matters because it pushes verification into the same loop that coding agents already live in: Mo runs from a coding agent session, a PR comment, the Momentic app, or the qa CLI, and a separate agent attempts to reproduce each functional bug before Mo reports it. The launch HN thread (155 points, 105 comments) mixed concrete benchmark curiosity with pointed skepticism about cost per bug, observability below the UI, vendor lock-in, and noisy onboarding.

What Mo is and where it fits

Mo is a custom agent harness built on three years of Momentic work and millions of test traces from real applications, pitched at verification rather than test authoring. The Momentic site frames the problem as: "Coding agents have made it dramatically faster to build software. The bottleneck is now knowing it works." Mo is sold as belonging inside the agent loop, where a coding agent that wrote a change can use MCP to author a test, check it in a live session, and commit the YAML alongside the code.

Site-cumulative numbers published at launch:

Tests created: 77,813
Test runs executed: 70,607,819
Changes verified: 339,166
PRs verified: 80,290
Auto-heals: 8,932,104
Bugs caught: 117,010

Named customers include Notion, Quora, Webflow, Framer, Xero, Whatnot, Retool, Runway, Bilt, Collectors, Podium, SPS Commerce, GPTZero, Vapi, Mutiny, Squire, Arena, and Kumo.

The five-part workflow

Momentic describes the product as five loops: a Knowledge base that records terminology, business rules, and expected behavior for agents to look up at each AI step; runtime recovery that re-finds controls when the UI changes mid-run; permanent healing that proposes a test-file diff for review; MCP integration so a coding agent can author and check tests; step-level failure evidence with screenshots and traces; and a CLI whose exit code can be required before merge or deploy.

Tests are written as plain-English YAML, versioned in the repository. The triage loop classifies failures as application bugs, flakiness, or environment issues, and AI triage proposes a repair for review.

QA Bench results: Mo versus Codex, Claude Code, and Devin

QA Bench graded Mo, Codex, Claude Code, and Devin on 20 areas across Gitea, Metabase, Twenty, and Saleor, giving each 60 minutes per area. Momentic reports that Mo found about 269 labeled bugs, while the next-best agent found about 86.

Momentic's own launch framing adds two honest caveats that dominated the thread. Precision: Mo at 84 percent versus Codex at 86 percent. Cost: "Mo costs $2.67 per bug vs $0.80 for Codex, less than Claude Code and Devin." A commenter, @podoman, read that back with visible confusion:

is this good? so Mo costs more money per bug? I'm confused.

The same trade-off was raised by @staindk, who asked whether the framing effectively says Codex is both better and cheaper. Momentic's own position is a recall-first one: accept lower precision and higher cost per bug in exchange for catching bugs other agents miss.

How the harness explores: DAG exploration and recall

Momentic describes Mo as a DAG-based exploration graph, and commenters picked up on that as the substantive technical claim. @2001zhaozhao argued there is something real there because the DAG systematically increases exploration surface area, and noted that in personal experience, Playwright-style QA steps within a coding agent catch certain bug classes reliably but let edge cases slip. The same commenter offered the key nuance:

i suppose that this DAG approach is only useful for codebase-wide bug sweeps? If one actually integrates it into a feature-by-feature agent loop, then the QA following each change is already narrow (since it only needs to test 1 changed feature), so this kind of exploration thoroughness optimization (in author's own words "recall") is less important.

@karinago raised the corresponding blind-spot concern: if the sales pitch is that LLMs are bad at testing because they reinforce their own assumptions, and Mo is built on millions of traces from Momentic's own product, how does the harness avoid developing blind spots at the harness level rather than the model level?

Example bugs Momentic says Mo caught

The site publishes a set of anonymized bugs by flow and impact, all labeled by detection stage. Representative examples:

A consumer fintech login-routing change silently skipped the new-device 2FA prompt, detected on a feature branch.
A rideshare app's live ETA subscription silently failed after a dependency update, detected in staging.
A wellness app's iOS lifecycle change reset the active sleep timer on screen lock, detected in CI.
A food-delivery payment-form refactor stopped pre-filling saved cards at checkout, detected eight minutes before deploy.
A retail marketplace reset all active category and price filters when moving to page two, detected in a nightly run.
A travel marketplace's validation change rejected multi-segment promo codes on the final checkout step, detected before merge.
A streaming platform's locale-detection bug selected the wrong subtitle track on first play, detected pre-release.
A one-day sweep caught 12 bugs and exposed legacy API issues that would have shipped undetected, detected before release.

These are product-marketing claims and should be treated as such; the source does not provide reproductions.

What commenters challenged: cost model, observability, and lock-in

Token cost and the BYO-subscription question

The most direct economic objection came from @aliasxneo, who framed per-token SaaS pricing as "the current subscription crisis on steroids" and argued the default should be BYOA: bring your own agent. The argument was that developers with leftover quota across multiple frontier subscriptions cannot spend it inside Momentic, forcing a second parallel spend on non-transferable tokens. @AgentMasterRace reported that the service felt too slow and expensive in brief use and preferred Claude with skills and maestro scripts. @JulianPaul asked the underlying product question plainly: why is Mo better than pointing a frontier model directly at an app with a custom Playwright-based harness on an existing subscription?

Does the agent see below the UI?

@bsp580 asked the sharpest technical observability question in the thread: how much of the failure surface is visible to the agent when the failure is not on screen?

A lot of the failures I run into aren't visible on screen: a request that returns a 200 with an error body, a retry loop, a call that quietly fails. Do the agents get the network traffic for a failed flow, or only screenshots and the view hierarchy?

@moon0917 made the same point from a concrete example in Saleor, one of the QA Bench targets. They described a price-recalculation task that was meant to run only on dirty listings, except a filtered queryset was never assigned back to the variable, so every listing recomputed on every run. Unit tests stayed green for unrelated reasons and the no-op sat in production because the final output was still valid. Their question to Momentic: does Mo catch that class of silent no-op where external state looks correct, or does it mostly hook into visible state diffs?

Determinism, lock-in, and target user

@staindk grouped the buyer questions that matter for enterprise adoption: what vendor lock-in looks like, whether Momentic code or a testing standard will be open enough to move off the platform or survive a shutdown, how deterministic the tests are, and who the intended user is, product owner, developer, or SDET.

@beavisringdin added three more: a preference for open-source tools strapped to a custom harness, an explicit question about what a YC W24 company has been doing for two years, and a roadmap question about supporting mobile development builds on real devices, offering to pay for the product if real-device support were handled.

Retry loops, pass silence, and the middle ground

@bshafiq asked how the harness handles an agent stuck clicking an element that never loads, specifically whether there is a hard max-steps limit or a smarter state-change detection to kill the run.

@daniiel argued that precision and recall are not the first metrics that matter for a QA agent, because the worst outcome is a silent pass on an already-broken flow:

a silent pass is worse than a flaky test, since the flaky one at least makes some noise.

That commenter described running mutation-testing-style evaluations, planting bugs in open-source apps and counting how many slipped through, and reported that adding a third possible answer, "I can't tell," eliminated most false passes because the engine stopped guessing. The open questions for Momentic: whether a bug Mo cannot reproduce shows up anywhere or is dropped, and whether the QA Bench bugs were planted or only labeled from existing applications.

Learn from non-bugs, PR-level scope, and the moat

@gophigure asked whether marking a finding as "not a bug" teaches Mo anything for the next run, or whether the same finding reappears. @throwaway7783 described giving Codex and Claude Code source-code access plus browser and Playwright control at a previous employer, and reported they found bugs and correlated them to the relevant UI and backend code, which suggests a benchmark variant with source-code access. @bortimus asked the simplest question in the thread: what is the moat? @maxdo pushed back on the benchmark framing itself, arguing that comparing a bare find-a-bug agent to a real engineered harness with product knowledge is not apples to apples.

Onboarding and UX friction reported by early users

Not every comment was about architecture. @ecares reported that after signing up for Mo, there was effectively no onboarding, and that the flow stalled at configuring authentication with no clear path through the pages. @ice_cream_man_9 ran Mo on one page of a personal site and listed four UI issues: the state indicator said "No bugs yet" while the tab reported "Bugs: 11," then said Mo found no bugs while the tab showed 12; all bugs were labeled "medium" with no explanation of the severity rubric and no way to reprioritize; the stop-testing button only appeared at the very end of the run rather than being available throughout; and there was no export for the discovered-bug list. @vfosnar flagged a privacy concern, saying Momentic appeared to hand their email to Slack for contact on login without prior notice, while their in-product Notifications setting for Slack was turned off.

How to read the launch

Momentic's core bet is that the testing bottleneck has moved from authoring tests to knowing whether a coding agent's change actually works, and that a purpose-built harness built on real application traces will out-recall a general coding agent. The published QA Bench numbers support that recall claim on a bounded dataset, but they also concede that Codex is slightly more precise and dramatically cheaper per bug. The thread's strongest unresolved questions are whether Mo can see silent failures that never surface in the view hierarchy, whether its pricing model survives the rise of bring-your-own-agent workflows, and whether teams can extract their YAML tests if the vendor relationship ends. Those are the questions a buyer should insist on before replacing a hand-built Playwright harness.