bug0inc/qa-agent

Bug0 AI – AI‑powered Playwright regression testing

What it is – An open‑source TypeScript library that lets you write Playwright browser tests in natural language. The library sends each step to LLMs (Claude, Gemini, OpenAI’s computer‑use agent, etc.), gets back actions, runs them in the browser, and validates the result with a multi‑model consensus engine. It also caches successful steps in Redis so later runs are fast and cheap.

Key capabilities

Feature	How it works
Natural‑language steps	runSteps() takes an array of {description, data?, waitUntil?, ai?} objects. The description is plain English (e.g., “Click Acme Circles T‑Shirt”). The library formats a prompt, calls the configured LLM, and translates the model’s response into Playwright actions.
Multi‑model assertion engine	For each assertion the library calls Claude and Gemini in parallel. If they disagree, a third “arbiter” model (configurable) decides, or you can fail immediately (fail‑on‑disagreement).
Smart caching	Successful step executions are stored in Redis keyed by userFlow + step.description. On a cache hit the step is replayed without any LLM call, dramatically reducing latency and cost.
Auto‑healing	When a cached step fails, the library falls back to the LLM to recompute the action, then updates the cache.
Video assertions	Optional video: true on an assertion records the whole step run, uploads the .webm to Gemini’s Files API, and lets Gemini evaluate the assertion against the video (useful for transient UI like toasts).
AI gateway support	You can route all LLM calls through Vercel AI Gateway, OpenRouter, Cloudflare AI Gateway, or talk directly to providers (Anthropic, Google, OpenAI).
CUA mode	When mode: "cua" is selected, the library uses OpenAI’s computer‑use agent (gpt‑5.5 + built‑in computer tool) for visual, screenshot‑driven automation.
Dynamic placeholders	{{run.*}}, {{global.*}}, {{data.*}}, and {{email.*}} let you inject random emails, IDs, or values extracted from received emails into steps.
Telemetry & logging	Optional OpenTelemetry traces (Axiom) and structured Pino logs.
Email extraction	Built‑in emailsink provider (disposable inbox) or custom provider to fetch verification codes, etc.

Typical workflow

1. Create a Playwright project – npm init playwright@latest bug0-ai-project (choose TypeScript).
2. Install the library – npm install @bug0/ai dotenv.
3. Add your LLM credentials – at least one Anthropic key and one Google Generative AI key (or use a gateway). Put them in a .env file.
4. Configure once – call configure({ ai: { gateway: "vercel", models: { … } }, redis: { url }, telemetry: { … } }) at the top of your test suite.
5. Write a test – use runSteps({ page, userFlow, steps, assertions, test, expect }). Each step is a plain English description; assertions are also natural language strings.
6. Run – npx playwright test example.spec.ts --project chromium. The report includes an AI‑generated summary.

Example test (TypeScript)

import { test, expect } from "@playwright/test";
import { runSteps, configure } from "@bug0/ai";

configure({
  ai: { gateway: "vercel" },
});

test("Shopping cart", async ({ page }) => {
  await runSteps({
    page,
    userFlow: "Add product to cart",
    steps: [
      { description: "Navigate to https://demo.vercel.store" },
      { description: "Click Acme Circles T-Shirt" },
      { description: "Select color", data: { value: "White" } },
      { description: "Add to cart", waitUntil: "My Cart is visible" },
    ],
    assertions: [{ assertion: "You can see My Cart with Acme Circles T-Shirt" }],
    test,
    expect,
  });
});

Configuration highlights

• Model slots – eight slots (step execution, user‑flow low/high, primary/secondary/arbiter assertions, utility, CUA). Defaults are Gemini‑flash for cheap steps and Claude‑4.5‑haiku for primary assertions.
• Redis – set REDIS_URL or pass via configure({ redis: { url } }) to enable caching and global placeholders.
• Gateways – set AI_GATEWAY_API_KEY, OPENROUTER_API_KEY, etc., depending on the chosen gateway.
• CUA mode – requires OPENAI_API_KEY with access to the computer tool; set mode: "cua" and gateway: "none".
• Telemetry – provide AXIOM_TOKEN/AXIOM_DATASET or configure them programmatically to get OpenTelemetry traces.

Who might use it

• Teams that already generate UI code with AI agents (e.g., Bug0’s FactoryKit) and need a regression suite that tolerates frequent DOM changes.
• QA engineers who want to write tests quickly without maintaining brittle selectors.
• Developers looking to combine LLM‑driven automation with traditional Playwright reliability features like caching and video capture.

Limitations noted in the README

• The test‑suite itself is still small; more edge‑case coverage is welcome.
• Video assertions bypass the gateway and require a direct Google API key.
• CUA mode cannot be mixed with a gateway; it must talk directly to OpenAI.
• Multi‑step caching is not yet implemented (only single‑step actions are cached).

License – FSL‑1.1‑ALv2 (a permissive open‑source license).

---

All information above is taken directly from the repository’s README; no additional features have been inferred.