injaneity/pi-computer-use

What it solves

pi-computer-use allows AI agents to interact with desktop applications across macOS, Windows, and Linux. It is designed for scenarios where an application lacks a reliable API or terminal command, forcing the agent to interact with the graphical user interface (GUI) directly.

How it works

The project provides a Pi extension that equips agents with a set of tools to find open windows, observe the visible interface, search for specific buttons or text, and perform physical actions like clicking, typing, and scrolling. On macOS, it uses a helper app requiring Accessibility and Screen Recording permissions. On Windows, it leverages platform accessibility APIs. On Linux, it utilizes the AT-SPI2 accessibility bus and X11/Wayland protocols.

Who it’s for

Developers building AI agents that need to automate tasks within standard desktop software on various operating systems.

Highlights

• Cross-platform support: Works on macOS (14+), Windows, and Linux.
• UI Observation: Tools for inspecting and searching the visible interface for controls and text.
• • Action Execution: Ability to click, type, scroll, and wait for UI changes.
• State Management: Uses immutable, state-scoped observations and cached progressive outline queries to manage desktop surfaces.
• Parallelism: Supports running independent roots in parallel.