web-infra-dev/midscene-skills

AI-powered, vision-driven UI automation for every platform.

What it solves

Midscene Skills provides a way to automate UI interactions across different platforms using natural language. It removes the need for writing complex, fragile same-element same-element same-element scripts by allowing an AI to "see" and interact with the screen regardless of the platform.

How it works

The project is built on Midscene.js, which uses vision-based automation. Instead of relying on the underlying code of an application, it operates entirely from screenshots. It requires models with strong visual grounding capabilities to accurately locate UI elements on the screen based on natural language instructions.

Who it’s for

Developers and QA engineers who need to automate tasks across web browsers, desktop OS (macOS, Windows, Linux), and mobile devices (Android, iOS, HarmonyOS), as well as those integrating AI agents into their workflows.

Highlights

  • Cross-platform support: Works across Browser, Chrome Bridge, Desktop, Android, iOS, and HarmonyOS.
  • Vision-driven: Operates via screenshots, making it more reliable for cross-platform use.
  • Natural language control: Allows users to interact with UI elements using plain English instructions.
  • E2E Testing: Includes a scaffold for managing AI-powered end-to-end tests via Vitest.

Related

  • Project
    volcengine/rtc-aigc-demo

    A demo project showcasing an interactive AIGC pipeline that integrates RTC, ASR, LLM, and TTS for real-time voice-based AI interactions.

  • Project
    horang-labs/tessera

    Tessera is a local desktop/web app that lets you run multiple Claude Code, Codex, or OpenCode AI coding agents in parallel, each in its own isolated Git worktree. It provides a Kanban board, split‑pane UI, chat view for terminals, full Git integration, and mobile remote access, all while using the provider CLIs already installed on your machine.

  • Project
    apple-aiml-research/ml-hierarchical-confusion-matrix

    Neo is an open‑source JavaScript library (npm package `@apple/hierarchical-confusion-matrix`) that provides an interactive visualisation for confusion matrices with hierarchical and multi‑output class labels. Install via npm, feed it a spec and an array of `{actual, observed, count}` records, and embed the matrix in any web page. It supports flat, hierarchical (`:`), multi‑output (`,`), and combined label structures, and comes with a live demo, unit tests, and a full TypeScript source tree.

  • Project
    SamurAIGPT/AI-Influencer-Generator

    An open-source pipeline that uses Stable Diffusion, gTTS, and SadTalker to create and animate consistent virtual AI influencers for social media.