ShawnPana/phone-harness

What it solves

It allows AI agents (such as Claude Code or Codex) to directly control a physical smartphone. It removes the need for complex setups like jailbreaking, Xcode, or WebDriverAgent by using existing system mirroring and debugging tools to provide a way for an LLM to "see" and "touch" a mobile device.

How it works

The project acts as a thin transport layer between an agent and the phone:

• For iPhone: It uses macOS iPhone Mirroring to treat the phone as a window. It "sees" via screen captures and Apple's Vision-framework OCR and "acts" by sending HID-level CGEvents (taps, drags, typing) to that window.
• For Android: It uses adb (Android Debug Bridge) over USB or Wi-Fi. It "sees" using screencap and the device's own accessibility tree for precise text and element location, and "acts" using adb input commands.

Who it’s for

Developers and AI researchers who want to build agents capable of automating real-world mobile tasks on physical iOS and Android devices without modifying the device's system software.

Highlights

• No Device Modification: Works without jailbreaking or installing specialized apps on the phone.
• Cross-Platform: Supports both iPhone (via macOS Sequoia+) and Android.
• Agent-Ready: Designed to be registered as a "skill" for LLMs, allowing them to automatically call phone-control functions.
• Stateless Architecture: No daemon is required; each invocation is self-contained.