HKUDS/PhoneCLI

What it solves

PhoneCLI addresses the inefficiency, cost, and brittleness of traditional GUI agents that rely on a continuous loop of screenshots and Vision-Language Model (VLM) calls. By treating repetitive navigation as a deterministic process rather than a reasoning problem, it reduces latency (from seconds to sub-second), lowers API costs, and improves task success rates on mobile devices.

How it works

PhoneCLI uses a three-tier execution strategy to handle mobile tasks:

1. Offline Compilation: A BFS crawler explores an app's interface to create an "app map" (a YAML file) that records screens, interactive elements, and navigation paths. This turns complex GUI navigation into a set of callable, deterministic commands.
2. Online Execution: When a task is requested, the system first attempts to use the compiled CLI commands for navigation at zero model cost. If a command fails or the task is novel, it falls back to an on-device model, and finally to a cloud-based VLM.
3. On-Device Optimization: It utilizes a specialized 3B parameter model (OpenPhone) designed to fit within mobile hardware constraints (GPU/NPU memory) to handle reasoning and verification locally, ensuring privacy and speed.

Who it’s for

It is designed for developers and researchers building mobile AI agents that need to operate reliably and efficiently on Android and iOS devices without relying exclusively on expensive cloud APIs.

Highlights

• Hybrid Execution: Combines deterministic CLI-style replays with VLM reasoning to cut cloud calls and increase success rates.
• App Mapping: Pre-computes navigation graphs for apps, allowing sub-second execution of routine tasks.
• On-Device First: Prioritizes local execution (CLI $\rightarrow$ Device Model $\rightarrow$ Cloud) to minimize latency and maximize privacy.
• OpenPhone-3B: Includes an open-weights 3B model optimized for GUI tasks through SFT and RL (GRPO).
• Cross-Platform: Supports both iOS (via WebDriverAgent) and Android (via ADB/AndroidLab).