lycorp-jp/sim-use

Give your AI agent eyes and hands on iOS Simulator and Android emulator/devices.

What it solves

sim-use provides a way for AI agents to interact with mobile applications on iOS Simulators and Android emulators or devices. It bridges the gap in the agentic development loop by allowing LLMs to observe the current screen state and perform actions (like tapping or typing) to verify that the features they have built are working correctly.

How it works

The tool operates as a cross-platform CLI that interfaces with Apple's Accessibility APIs, the iOS Simulator HID pipeline, and Android's AccessibilityService. It uses a cycle of "observe, act, verify":

  1. Observe: The ui command converts the complex accessibility tree of a screen into a token-efficient, compact outline that an LLM can easily reason about without consuming excessive tokens.
  2. Act: The agent can perform actions using aliases (e.g., tap @9) or specific identifiers, removing the need for the exact coordinates of elements.
  3. Verify: The agent runs the ui command again to see the result of its action.

To maintain speed, it uses a per-device background daemon to reduce initialization costs, enabling round-trip interactions in approximately 300ms.

Who it’s for

It is designed for developers building AI agents for mobile app development, testing, and automation who need a high-performance, cross-platform interface to drive mobile simulators and devices.

Highlights

  • Token-Efficient: Outline representations are ~16x more compact than raw JSON accessibility trees.
  • Cross-Platform: A single command surface for both iOS and Android.
  • AI-Native: Includes alias-cached taps and a bundled agent skill (e.g., for Claude) to teach the AI how to use the CLI.
  • Comprehensive Input: Supports tapping, swiping, hardware buttons, and IME-safe Unicode pasting to bypass keyboard composition issues.
  • Batching: iOS-only support for chaining multiple steps into one invocation to reduce latency.
  • Viewer: A bundled web app that renders the accessibility tree onto an SVG canvas for visual debugging.