bug0inc/passmark

What it solves

Passmark is a Playwright library designed to eliminate the fragility of traditional selector-based browser regression testing. It allows developers to write tests in natural language, which prevents tests from breaking whenever the DOM structure changes, while providing high-confidence assertions through a multi-model AI consensus engine.

How it works

Passmark integrates with Playwright to execute browser steps described in plain English. It uses AI models to translate these descriptions into actions. To ensure stability and accuracy, it employs several key mechanisms:

• Multi-Model Consensus: Assertions are validated by both Claude and Gemini in parallel. If they disagree, a third arbiter model resolves the conflict.
• Intelligent Caching: Successful actions are stored in Redis; subsequent runs use the cached action first, falling back to AI execution and "auto-healing" the cache if the action fails.
• Execution Modes: It supports standard ARIA accessibility snapshots for efficiency and a "CUA mode" (Computer Use Agent) using OpenAI's computer-use agent for visual, screenshot-driven automation.
• Video Assertions: For transient UI elements like toast messages, it can record the step run and use Gemini's Files API to evaluate assertions against the video.

Who it’s for

It is primarily for QA engineers and developers who want to maintain stable end-to-end browser tests that don't require constant updates to CSS selectors or XPaths, especially when working with AI coding agents that frequently modify the UI.

Highlights

• Natural Language Steps: Write tests as a sequence of descriptions rather than rigid code selectors.
• Consensus-Based Validation: Reduces flakiness by using multiple LLMs to verify test outcomes.
• Auto-healing Cache: Combines the speed of traditional tests with the AI's ability to adapt to UI changes.
• Flexible AI Routing: Supports direct provider SDKs or AI gateways like Vercel, OpenRouter, and Cloudflare.
• Integrated Email Testing: Built-in support for disposable email providers to test verification flows.