droidrun/mobilerun

What it solves

Mobilerun provides a way to control Android and iOS devices using natural language commands. It eliminates the need for manual scripting of mobile interactions by allowing LLM agents to navigate apps, perform multi-step workflows, and extract data from mobile interfaces.

How it works

The framework uses a "Portal" app installed on the device to bridge the gap between the LLM and the mobile OS. It combines accessibility trees (UI state) with screenshots for visual understanding. Agents can then execute actions like tapping, swiping, and typing. It supports various LLM providers (OpenAI, Anthropic, Gemini, etc.) and offers two modes of operation: a local framework for machine-based control and a cloud-managed service for hosted devices.

Who it’s for

• QA Engineers: For mobile app testing and regression checks.
• Developers: To build custom mobile automation workflows via a Python API.
• Business Users: To automate repetitive mobile tasks or extract data from native apps using natural language.
• AI Researchers: To create agents capable of interacting with real-world mobile environments.

Highlights

• Cross-Platform: Supports both Android and iOS devices.
• Vision Mode: Can use screenshots for apps that lack accessibility tree data.
• Reasoning Mode: Employs a manager-executor pattern for complex, multi-step planning.
• Extensible: Allows for custom tools, structured output, and app-specific guidance cards.
• Observability: Integrates with Arize Phoenix and Langfuse for execution tracing.