onehumanbeing/esp32-iphone-agent

Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.

What it solves

This project allows an AI agent to control a real iPhone without requiring a Mac, an Apple developer account, jailbreaking, or installing any software on the device. It bypasses the traditional need for XCTest or WebDriverAgent by interacting with the phone as a human would: by seeing the screen and using a pointer.

How it works

  • Eyes: The system uses AirPlay mirroring via UxPlay to stream the iPhone's screen to a computer, which then converts the stream into JPEG frames for the AI model.
  • Hands: A cheap ESP32-S3 board is plugged into the iPhone's USB port, acting as a standard USB HID (Human Interface Device) mouse and keyboard. With iOS AssistiveTouch enabled, the agent can move a pointer, click, and type.
  • Brain: An OpenAI-compatible vision model acts as both a planner (creating a step-by-step goal) and an actor (deciding the specific action for each screenshot).
  • Closed-Loop Control: The server tracks the pointer's relative position and wiggles it before taps to calibrate and ensure accuracy. It also verifies if the screen changed after an action to detect and retry missed taps.

Who it’s for

Developers and AI researchers who want to automate iPhone apps (including games) using vision-based AI agents without the friction of Apple's developer ecosystem.

Highlights

  • Hardware-based control: Uses a $7 ESP32-S3 board to simulate mouse/keyboard input.
  • No-install approach: Requires no apps or developer signatures on the iPhone.
  • Vision-driven: Works with any OpenAI-compatible vision model, including local models.
  • Real-time monitoring: Includes a web UI with a live phone screen, action logs, and a step-by-step reasoning wall.
  • Cross-platform: The server can run on macOS or Linux.

Written about in

Related

  • Project
    RainerKuemmerle/g2o

    A C++ framework for optimizing graph-based nonlinear error functions, commonly used for SLAM and Bundle Adjustment in robotics and computer vision.

  • Project
    realsenseai/realsense-ros

    A ROS wrapper for Intel RealSense cameras that enables depth and color sensor data to be streamed as ROS topics for robotic applications.

  • Project
    commaai/opendbc

    A Python API and database of CAN bus definitions that allows developers to read vehicle data and send control commands to modern cars for advanced driver-assistance applications.

  • Project
    autorope/donkeycar

    A minimalist Python library for building and teaching self-driving robots, featuring support for deep learning, computer vision, and GPS autopilots.