zhongerxin/iPhone-use

What it solves

Controlling a real iPhone by hand is slow and repetitive. This project lets AI coding assistants like Codex, WorkBuddy, or 豆包工作 operate your actual iPhone over USB, so you can describe a task in plain language and the assistant opens apps, reads screens, taps, scrolls, and types on its own. It also shows the phone's screen in a sidebar for live monitoring.

How it works

The project uses WebDriverAgent (WDA), an open-source automation service that runs on the iPhone itself. The assistant communicates with WDA through a local MCP (Model Context Protocol) server, which forwards commands over USB and streams the screen. A one-time setup installs and signs WDA on your phone using your Apple account and Xcode. After setup, the assistant can call a set of 18 tools to inspect the screen, find UI elements, tap, swipe, type text, perform batched actions, and recover from failures. When element location fails, it falls back to looking at a screenshot and tapping by coordinates. The screen preview widget runs in the assistant's sidebar and supports direct clicking, dragging, and scrolling.

Who it's for

Developers and power users who use Codex, WorkBuddy, or 豆包工作 on macOS and want those assistants to automate tasks on a physical iPhone connected via USB. It requires a Mac with full Xcode, Python 3.9+, Node.js and npm, an Apple developer account for signing, and an unlocked iPhone with developer mode enabled.

Highlights

• Local USB connection: all traffic stays on your Mac; no cloud or Appium Server needed.
• Reuses existing work: healthy WDA sessions and previous builds are reused to avoid repeated setup.
• Compact observation and combined actions: the control tree is simplified, screenshots are only taken when needed, and batch/scroll/collect tools cut down on assistant round-trips.
• Asynchronous setup: downloads, builds, and launches return a job ID you can query, avoiding duplicate starts.
• Screenshot fallback: if element lookup fails, the system returns a screenshot and the assistant selects a tap point by pixel-to-point conversion.
• Clear failure semantics: an operation lock prevents conflicting actions, and ambiguous results are marked so the assistant reads the actual state before retrying.
• Live screen with pause/resume: the preview shows the iPhone shell, status, and gestures; if the user needs to unlock or authenticate, a mask overlays the screen and resumes automatically after.