tencentmusic/page-eyes-agent

What it solves

PageEyes Agent simplifies UI automation and testing by allowing users to control applications using natural language instead of writing complex scripts. It enables automated UI inspections and testing across multiple platforms, reducing the manual effort required to create and maintain automation suites.

How it works

Built on the Pydantic AI framework, the agent translates natural language instructions into UI actions. It can operate in two primary modes: a lightweight mode using a Vision Language Model (VLM) and a hybrid mode that combines a specialized vision model (OmniParserV2) for element perception with a standard Large Language Model (LLM) for path planning. This architecture allows even smaller LLMs to handle complex navigation tasks without needing a full VLM for every step.

Who it’s for

It is designed for QA engineers and testers who need to perform cross-platform UI automation, as well as developers looking to integrate AI-driven UI interaction into their workflows.

Highlights

• Cross-Platform Support: Works across Web, Android, iOS, HarmonyOS Next, and Electron desktop applications.
• Natural Language Driven: Executes tasks and performs assertions based on plain text instructions.
• Flexible Model Integration: Supports various models including DeepSeek, OpenAI, and Qwen.
• Detailed Reporting: Generates execution logs and visual reports to help testers verify results.