rounak/PhoneAgent

An AI agent that can get things done across iPhone apps.

What it solves

PhoneAgent provides a way for AI agents to interact with and control mobile devices. It bridges the gap between high-level AI reasoning and the low-level technical actions required to operate an iPhone or Android device, allowing an agent to perform tasks like opening apps, tapping elements, and entering text.

How it works

The project operates in two primary modes:

  1. In-app iPhone agent: A standalone SwiftUI app that uses the OpenAI Responses API to process prompts and execute actions directly on the device.
  2. External bridge: A JSON-RPC server that allows external AI agents (such as Codex or OpenClaw) to send commands to a mobile device.

For iOS, it uses XCTest-hosted actions against a simulator or physical device. For Android, it uses adb (Android Debug Bridge) combined with UiAutomator to execute commands.

Who it’s for

Developers and AI researchers building mobile automation agents that need a standardized way to control iOS and Android devices via API calls.

Highlights

  • Cross-platform support: Works with both iOS (simulators and physical devices) and Android (emulators and physical devices).
  • Unified RPC surface: Provides a shared set of actions for both platforms, including get_tree, get_screen_image, tap, and scroll.
  • Voice and Text input: The in-app iOS agent supports keyboard input, microphone support, and a custom wake-word mode.
  • Localhost-oriented security: The RPC bridge is restricted to 127.0.0.1 to maintain control over device access.

Related

  • Project
    raulvidis/hermes-android

    A remote Android device control system that allows AI agents to interact with mobile apps and hardware via a WebSocket relay.

  • Project
    minitap-ai/mobile-use

    An open-source AI agent that automates Android and iOS devices using natural language to perform tasks and scrape data from apps.

  • Project
    Hacker-Valley-Media/Interceptor

    A tool that lets AI agents control your real browser and macOS apps using your existing sessions and logins, avoiding the need for separate automated browser instances.

  • Project
    droidrun/mobile-harness

    A Python API and Markdown-based instruction set that enables AI agents to control Android and iOS devices locally or in the cloud.

  • Project
    ghost-in-the-droid/android-agent

    An open-source framework that gives AI agents a mobile body, enabling them to control Android and iOS devices via a unified tool surface and MCP server.