Core-Mate/OpenGUI

What it solves

OpenGUI provides a framework for AI agents to see, understand, and operate Android application interfaces on real devices. It solves the difficulty of automating complex mobile workflows—such as social media management, regression testing, and repetitive in-game tasks—by enabling agents to perform screen understanding, task planning, and action execution.

How it works

The system uses a multi-layered orchestration approach consisting of a backend graph and an Android client. A "Plan Supervisor" breaks goals into executable steps, which are then passed to an "Executor Graph" that manages a loop of taking screenshots, using a Vision Language Model (VLM) to analyze the UI, and executing actions via Android's AccessibilityService. The system supports model routing, allowing different models (e.g., Claude Opus for planning and Doubao Pro for vision) to be assigned to different roles to balance performance and cost. It can be deployed as a full stack or as a plugin for DeepSeek Harness.

Who it’s for

It is designed for developers and testers who need to automate Android app operations, researchers performing lead research on mobile platforms, and users wanting to dispatch mobile tasks remotely via IM channels like Discord, Telegram, or Feishu.

Highlights

• Long-running task support: Designed for workflows that can last several hours with built-in progress tracking and recovery.
• Role-based model routing: Ability to split planning and VLM execution across different model providers.
• Remote dispatch: Built-in support for triggering tasks via REST API or IM platforms (Discord, Telegram, Feishu) while phones remain on standby.
• Structured workflows: Implements a "plan-before-action, summarize-after-execution" cycle to ensure reliability and provide clear results.