lahfir/agent-desktop

Native desktop automation CLI for AI agents. Control any application through OS accessibility trees with structured JSON output and deterministic element refs.

What it solves

AI agents often struggle to interact with desktop applications because they rely on inefficient methods like screenshots or pixel matching. This tool provides a structured, machine-readable way for agents to observe and control any application by tapping into the operating system's accessibility trees.

How it works

The tool uses a Rust-based CLI to traverse the OS accessibility tree, converting UI elements into compact, machine-readable JSON snapshots. It uses a unique reference system (e.g., @s8f3k2p9:e1) to allow agents to target specific buttons, text fields, or menus without needing to re-process the entire screen. It supports a "progressive skeleton traversal" technique, where an agent first gets a shallow overview of an app to save tokens before drilling down into specific regions.

Who it’s for

  • AI Agent Developers: Building autonomous agents that need to perform tasks in native desktop apps (like Slack, Finder, or Xcode).
  • Software Engineers: Integrating desktop automation into existing workflows via the provided C-ABI (FFI) for languages like Python, Node, or Go.
  • Automation Engineers: Creating reliable, headless-by-default desktop automation scripts.

Highlights

  • Token Efficiency: Reduces LLM token usage by 78–96% through progressive skeleton traversal.
  • Native Performance: Built in Rust as a fast, single binary with C-ABI bindings for in-process calls.
  • Headless Interaction: Supports semantic operations that don't require stealing focus or moving the physical cursor.
  • Session Tracing: Includes a built-in trace viewer to record and replay agent interaction sessions.