OpenGUI – Mobile GUI Agent Framework

What it does (in plain language)

OpenGUI lets a large‑language‑model‑based AI agent see an Android phone’s screen, understand what’s on it, and act on the device – tapping, swiping, typing, opening apps, etc. It does this by using real screenshots from a connected phone and sending the image plus a natural‑language instruction to a model that can process images. The model decides which UI actions to perform, and OpenGUI executes them via Android’s accessibility service.

Who it’s for

• Developers of AI agents who want to give their bots the ability to control real Android apps (testing, automation, content creation).
• Product testers who need a scriptable way to run UI‑level tests and capture reproducible screenshots.
• Power users of the WorkBuddy or DeepSeek Harness platforms who want a plug‑in that adds phone‑control capabilities without running a full backend.

Main pieces

Component	Role
WorkBuddy / DeepSeek Harness plugins	Thin host‑side runtimes that expose a /opengui … command in the chat UI. They handle installing the plugin, authorising it, and forwarding user prompts to the backend.
Full‑stack backend (optional)	Graph‑based task executor written in TypeScript/Node. It receives a request, routes it to a vision‑language model (Claude, GPT, Gemini, etc.), builds a plan, and streams actions to the Android client.
Android client	Small app running on the phone (Android 11+). It provides screenshots via ADB/scrcpy, receives action commands (tap, swipe, type) through a WebSocket, and performs them using an AccessibilityService.
CLI (pnpm opengui …)	Direct command‑line interface for developers who prefer scripting instead of the chat plugins.
IM channel adapters	Optional Discord/Telegram/Feishu gateways that let you trigger tasks from chat services.

How you would use it

1. Install the host plugin – run a one‑line curl … | bash command on macOS (WorkBuddy) or follow the DeepSeek Harness install prompt.
2. Connect a phone – enable USB debugging, plug it in, and grant the Android app the required accessibility/overlay permissions.
3. Send a natural‑language command – e.g. /opengui Open Settings and report the Android version.
4. The plugin forwards the prompt to the backend, which asks the vision model to look at the current screen, decide the next UI action, and send that back to the Android client.
5. The phone executes the action; the model can iterate, take screenshots, and finally return a summary or a set of issue reports.

What it can do today

• Basic phone navigation – open apps, tap buttons, swipe, type text, go back.
• App testing – run a described workflow, capture steps and screenshots, and produce a bug report.
• Content creation – fill forms, draft social‑media posts (e.g., a TikTok draft) directly on the device.
• Visible execution – you can watch the phone screen in the WorkBuddy workbench and intervene or stop the task.
• Integration points – Discord, Feishu, Telegram, or a generic REST API for remote dispatch.

What it cannot do (limitations)

• Requires a vision‑capable model (image input + tool calling). If you use a text‑only model, the agent cannot interpret the screen.
• Only one OpenGUI task per DeepSeek Harness session; concurrent tasks need separate sessions.
• Real‑world reliability varies with device OEM quirks, screen layouts, and network latency.
• The full backend (graph agent, Redis/PostgreSQL, etc.) is optional but needed for advanced routing, multi‑step planning, and persistent task IDs.

Getting started quickly (WorkBuddy example)

cd "$HOME" && curl -fsSL https://raw.githubusercontent.com/Core-Mate/OpenGUI/main/workbuddy-plugin/install.sh | bash
# restart WorkBuddy, then in its chat:
/opengui Prepare a TikTok draft about "AI safety" – stop before publishing.

The plugin will:

1. Install a private Node runtime.
2. Register the OpenGUI skill and connector.
3. Ask you to authorise the phone.
4. Execute the command and return a draft with screenshots.

Where to find more info

• Installation & troubleshooting – workbuddy-plugin/INSTALL.md and deepseek-harness-plugin/README.md
• Full‑stack deployment – docs/get-started.md
• CLI reference – docs/codex-remote-control.md
• System diagram – the README’s Mermaid flowchart shows the data path from chat → model → executor → Android client.

License & commercial note

The code is released under the Business Source License 1.1, which permits non‑production use. Production or commercial deployments require a separate commercial license from Core‑Mate (change‑date 2030‑04‑29 when it becomes Apache 2.0).

---

All details above are taken directly from the repository’s README; no external assumptions have been added.