web-infra-dev/midscene

What it solves

Midscene.js is a GUI agent designed to eliminate the fragility of traditional UI automation. Instead of relying on fragile DOM selectors or accessibility trees that break during refactors, it uses AI-powered vision to interact with interfaces based on how they look to a human user.

How it works

The project uses multimodal models (such as Qwen, Gemini, and UI-TARS) to perform element localization based solely on screenshots. This allows it to target elements on any surface where a screenshot can be taken—including web browsers, Android, iOS, HarmonyOS, and desktop apps—using natural language instructions.

Who it’s for

It is primarily built for developers and QA engineers performing end-to-end (E2E) testing and UI automation across multiple platforms.

Highlights

• Vision-Driven: Operates from screenshots alone, enabling it to target elements in <canvas>, native apps, and cross-origin iframes.
• Cross-Platform: Supports web, mobile (iOS/Android/HarmonyOS), and desktop applications through a single API.
• Natural Language: Allows users to describe automation steps in plain English rather than writing complex selectors.
• Integration: Integrates with existing tools like Playwright, Puppeteer, and Vitest.