lycorp-jp/sim-use

Give your AI agent eyes and hands on iOS Simulator and Android emulator/devices.

What it solves

sim-use provides a way for AI agents to interact with mobile applications on iOS Simulators, Android emulators, and Android devices. It bridges the gap in the agentic development loop by allowing agents to observe the screen state and perform actions (like tapping or typing) to verify their work or automate mobile workflows.

How it works

The tool is a cross-platform CLI that interfaces with Apple's Accessibility APIs, the iOS Simulator HID pipeline, and Android's AccessibilityService. It simplifies the complex accessibility trees of mobile OSs into a token-efficient, text-based outline that LLMs can easily reason about. It uses a per-device background daemon to reduce initialization overhead, enabling fast observe-act round trips (approximately 300ms).

Who it’s for

It is designed for developers building AI agents that need to automate mobile app testing, verification, and interaction across both iOS and Android platforms.

Highlights

  • Token-Efficient Observation: Converts raw accessibility data into a compact outline, reducing token usage by roughly 16x compared to JSON.
  • Cross-Platform Consistency: Uses a single command surface for both iOS and Android, allowing one agent loop to work on both platforms.
  • Alias-Based Interaction: Supports @N aliases for elements, allowing agents to tap elements based on the last observed screen without needing coordinates.
  • Comprehensive Control: Supports a wide range of actions including taps, swipes, text input, hardware button presses, and video streaming/recording.
  • Agent-Native Integration: Includes a bundled agent skill (e.g., for Claude) to quickly teach AI clients how to use the CLI.

Related

  • Project
    block/trailblaze

    Trailblaze is a CLI tool that allows AI coding agents to control iOS, Android, and web applications, turning agent-led sessions into deterministic, replayable automated tests.

  • Project
    callstack/agent-device

    A tool that lets AI coding agents drive and verify mobile apps by reading accessibility-tree snapshots (not screenshots), acting through UI refs, and capturing evidence — available as a CLI, MCP server, or typed Node.js API.

  • Project
    injaneity/pi-computer-use

    A Pi extension that enables AI agents to interact with desktop applications by observing UI elements and performing actions like clicking and typing across macOS, Windows, and Linux.

  • Project
    remorses/usecomputer

    usecomputer is a cross‑platform CLI (and TypeScript library) that lets AI agents take screenshots, map coordinates, and perform mouse/keyboard actions (click, drag, type, scroll) on macOS, Linux and Windows. It uses a native Zig backend, provides a coord‑map system for accurate screen‑to‑image mapping, streams global input events via SSE, and supports Kitty Graphics Protocol for agent‑friendly image delivery.

  • Project
    droidrun/mobile-harness

    Mobile Harness is a Python‑based, Markdown‑driven library that lets LLM agents control Android and iOS devices (locally via ADB/Portal or remotely via Mobilerun cloud). It offers a unified API for UI interaction, screenshots, app launch, and accessibility‑tree queries, and includes benchmarks showing near‑perfect performance on AndroidWorld and iOSWorld task suites.