OpenMinis/iCTRL

An AI-friendly iOS/iPadOS automation testing tool built on XCUITest, but more than a testing tool: agents like Codex, Claude Code, Hermes, OpenClaw and Minis can operate your iPhone or iPad wirelessly through an encrypted JSON-RPC 2.0 API (tap, type, read the screen, launch apps, and more).

What it solves

iCTRL turns an iPhone into a controllable automation endpoint for AI agents. It addresses the difficulty of programmatically driving iOS devices by providing a stable, high-performance API that allows an agent to interact with the screen, apps, and system settings as if it were a human user.

How it works

The project runs a long-lived XCUITest runner on the iPhone that exposes a JSON-RPC 2.0 API over HTTP. Because it is designed for AI, the device provides its own documentation via GET /skill and GET /schema, allowing an agent to learn how to interact with the phone at runtime. It uses synthesized touch events to bypass standard UI test delays, achieving fast response times for taps, swipes, and screen reading.

Who it’s for

This is for developers and researchers building AI agents that need to perform tasks on iOS, such as navigating apps, reading screen content, or automating workflows that require real-world device interaction.

Highlights

  • Self-describing API: Agents can discover capabilities and methods dynamically without pre-configured knowledge.
  • High performance: Optimized for speed with ~280ms tap latency and sub-second screen snapshots.
  • Enhanced visibility: Uses on-device Vision OCR to read text and elements that the standard accessibility tree might miss.
  • Robust security: Features encrypted, authenticated communication with pairing approvals required directly on the device.
  • Flexible deployment: Can be installed via SideStore (free Apple ID) or Xcode, and supports remote operation over LAN or local-only modes.

Related

  • Project
    FoundationAgents/ReCode

    ReCode is an LLM agent framework that uses recursive code generation to unify high-level planning and low-level action, enabling dynamic granularity control for complex tasks.

  • Project
    Dwinovo/minecraft-numen

    Numen is an open‑source Minecraft mod that embeds an embodied AI agent controllable by large language models. Users install the mod, configure an OpenAI/Anthropic API key, summon a bot, and interact with it via natural language, voice, or scripted Lua actions. The project provides a three‑layer architecture (Minecraft core, Lua action API, LLM sandbox) and a plugin system for extending the agent’s capabilities.

  • Project
    SaseQ/discord-mcp

    A Model Context Protocol (MCP) server that allows AI assistants to manage Discord servers, including channel, member, and moderation tasks.

  • Project
    ayi-ai/nie-grassroots-logic

    An AI-ready methodology toolkit that transforms the frameworks from Nie Huihua's book on grassroots China into executable skills for AI agents to analyze power structures and local governance.