web-infra-dev/midscene

Midscene.js – Vision‑driven GUI Agent for E2E Testing

What it is – A JavaScript library that lets you write end‑to‑end (E2E) tests for web, mobile, and desktop applications in natural language. It uses a vision model to “see” the UI, locate elements from screenshots, perform actions (click, type, scroll) and verify what the user would actually see.

Key ideas

• Natural‑language test steps – Write aiAct, aiWaitFor, aiAssert calls (or YAML cases) describing what a user should do and what should appear.
• Vision‑based locating – No CSS selectors or DOM queries needed; the agent matches UI elements by appearance, so it works on canvas‑based controls, icon‑only buttons, cross‑origin iframes, etc.
• Cross‑platform API – Same PlaywrightAgent, PuppeteerAgent, or generic Agent can be used for web browsers, Android/iOS devices, HarmonyOS, and remote desktop sessions.
• Model‑agnostic & low‑cost – Supports a range of multimodal LLMs (Qwen‑VL, Gemini‑Flash, Doubao‑Seed, UI‑TARS, etc.) with configurable pricing; screenshot‑only input keeps token usage cheap.
• HTML replay reports – After a run you get a visual report with screenshots, element locations, AI decisions and assertion results for easy debugging.

How you use it

1. Create a Midscene Test project (or add the Agent to an existing Playwright/Puppeteer test).
2. Configure a vision model (e.g., Qwen3.x or gemini‑3.5‑flash).
3. Write steps either in a YAML file:

   cases:
     - name: Find headphones under $100
       steps:
         - aiAct: Search for headphones and filter the results to prices under $100
         - aiWaitFor: The filtered search results are visible
         - aiAssert: There are products in the search results, and each costs less than $100
   

   or directly in

   await agent.aiAct('Search for headphones and filter the results to prices under $100');
   await agent.aiWaitFor('The filtered search results are visible');
   await agent.aiAssert('There are products in the search results, and each costs less than $100');
   

4. Run the test – Midscene captures screenshots, sends them to the model, performs the actions, and produces an interactive HTML report.

Supported platforms

Platform	Integration
Web	Playwright, Puppeteer, or a Chrome‑bridge mode
Android	ADB‑controlled devices/emulators
iOS	WebDriverAgent‑controlled devices/simulators
HarmonyOS	HDC‑controlled devices
Desktop (macOS/Windows/Linux)	Remote RDP or native keyboard/mouse control
Custom UI	Provide screenshots + action hooks

Benchmarks – Reported pass rates of 93 %+ on AndroidWorld, 78 %+ on MobileWorld, and 96 %+ on AppControlBench with sub‑$1 model cost for a full suite, demonstrating both accuracy and cost‑effectiveness.

Getting started

• Install the package: npm i @midscene/web (or the Playwright‑specific bundle).
• Follow the quick‑start guide on the website or the Playground Chrome extension to try a test without any code.
• For larger suites, create a Midscene Test project (@midscene/test) and add custom TypeScript Nodes for reusable business logic.

Community & ecosystem

• Official docs: https://midscenejs.com
• Sample repo: https://github.com/web-infra-dev/midscene-example
• SDKs in other languages (Python, Java, etc.) are maintained by the community.
• Active Discord, X, and Lark groups for support.

Why it matters – Traditional UI test frameworks rely on brittle selectors and DOM knowledge. Midscene flips the script: the AI looks at the UI just like a human, making tests more resilient to UI changes and enabling testing of visual aspects (colors, borders, canvas drawings) that are hard to assert with code alone.

---

Midscene.js is MIT‑licensed and welcomes stars and contributions.