injaneity/pi-computer-use

Let Pi control your apps on MacOS & Windows

What it solves

pi-computer-use allows AI agents to interact with desktop applications across macOS, Windows, and Linux. It is designed for scenarios where an application lacks a reliable API or terminal command, forcing the agent to interact with the graphical user interface (GUI) directly.

How it works

The project provides a Pi extension that equips agents with a set of tools to find open windows, observe the visible interface, search for specific buttons or text, and perform physical actions like clicking, typing, and scrolling. On macOS, it uses a helper app requiring Accessibility and Screen Recording permissions. On Windows, it leverages platform accessibility APIs. On Linux, it utilizes the AT-SPI2 accessibility bus and X11/Wayland protocols.

Who it’s for

Developers building AI agents that need to automate tasks within standard desktop software on various operating systems.

Highlights

  • Cross-platform support: Works on macOS (14+), Windows, and Linux.
  • UI Observation: Tools for inspecting and searching the visible interface for controls and text.
    • Action Execution: Ability to click, type, scroll, and wait for UI changes.
  • State Management: Uses immutable, state-scoped observations and cached progressive outline queries to manage desktop surfaces.
  • Parallelism: Supports running independent roots in parallel.

Related

  • Project
    lycorp-jp/sim-use

    A cross-platform CLI that enables AI agents to observe and act on iOS and Android screens via token-efficient UI outlines and simplified interaction commands.

  • Project
    justhil/pi-app

    pi Desktop is a cross‑platform Electron app that provides a graphical UI for the existing *pi* coding‑agent. It shares the same `~/.pi/agent` files as the CLI, showing sessions in a timeline, offering Git‑style diffs, file browsing, parallel workers, extension adapters, themes, and optional voice input.

  • Project
    remorses/usecomputer

    usecomputer is a cross‑platform CLI (and TypeScript library) that lets AI agents take screenshots, map coordinates, and perform mouse/keyboard actions (click, drag, type, scroll) on macOS, Linux and Windows. It uses a native Zig backend, provides a coord‑map system for accurate screen‑to‑image mapping, streams global input events via SSE, and supports Kitty Graphics Protocol for agent‑friendly image delivery.

  • Project
    iFurySt/open-codex-computer-use

    open-computer-use is an open‑source MCP server that lets AI agents control a desktop (macOS, Linux, Windows). It provides JSON‑based tools (list apps, press keys, move cursor, etc.) and integrates with agents like Codex, Claude Code, Gemini CLI via one‑line install commands.

  • Project
    openclaw/Peekaboo

    Peekaboo is a macOS‑only CLI and menu‑bar app that captures the screen, builds a structured UI map via accessibility APIs, and lets you automate clicks, typing, key presses, scrolling, etc. It also includes an LLM‑driven “agent” that can execute natural‑language instructions, and it can be exposed to external Model‑Client‑Protocol (MCP) clients such as Codex or Claude Code.