lycorp-jp/sim-use

What it solves

sim-use provides a unified command-line interface (CLI) that allows AI agents to observe and interact with mobile screens on iOS Simulators, Android emulators, and Android devices. It bridges the gap in the agentic mobile development loop by enabling agents to verify their work by actually "seeing" and acting upon the UI of the apps they build.

How it works

The tool drives Apple's Accessibility APIs, the iOS Simulator HID pipeline, and Android's AccessibilityService through a single command surface. It converts complex accessibility trees into a token-efficient, compact outline that LLMs can easily reason about. To act, agents can use aliases (like @9) cached from the last screen observation, meaning they don't need to provide exact coordinates for every interaction.

Who it’s for

It is designed for developers building AI agents for mobile app development, testing, and automation who need a cross-platform way to programmatically control mobile environments.

Highlights

• Token-Efficient Observation: Converts raw accessibility trees into a compact outline, reducing token usage by approximately 16x compared to raw JSON.
• Cross-Platform: A single set of commands works across iOS Simulators, Android emulators, and Android devices.
• Agent-Native Design: Includes alias-cached taps, structured JSON envelopes for errors, and a bundled agent skill to quickly teach AI clients (like Claude) how to use the CLI.
• High Performance: Uses a background daemon to minimize initialization costs, achieving round-trip times of around 300ms.
• Comprehensive Interaction: Supports taps, swipes, text input, pasting (including IME-safe Unicode), hardware button presses, and video recording/streaming.