lahfir/agent-desktop

What it is

agent‑desktop is a command‑line tool (and accompanying C library) that lets an AI agent see and control graphical desktop applications on macOS and Windows. It reads the operating system’s accessibility tree, exports a JSON description of UI elements, and can issue clicks, typing, window management, clipboard access, etc., by referring to those elements with stable IDs.

Why it matters

Controlling a GUI from code is a classic bottleneck for autonomous agents. Most LLM‑based assistants can only work with text APIs, so they can’t directly interact with the apps a human uses. By exposing the UI as a structured, machine‑readable tree and providing a rich set of actions, agent‑desktop bridges that gap, enabling agents to automate real‑world desktop workflows (e.g., filing a report in Finder, sending a Slack message, or dismissing a notification) without moving the mouse or stealing focus.

How it works (high‑level)

1. Snapshot – agent-desktop snapshot --app <name> walks the OS accessibility API and prints a JSON tree of UI elements. Each element gets a reference like @s8f3k2p9:e3 that encodes the snapshot it belongs to.
2. Act – Commands such as click, type, press, scroll, or launch take a reference (or a key shortcut) and send the corresponding input through the accessibility layer. By default actions are headless (no visible cursor movement).
3. Iterate – After an action you take another snapshot to verify the UI changed as expected. Stale or ambiguous refs are reported so the agent can refresh.
4. Library mode – The same functionality is packaged as libagent_desktop_ffi, a C‑compatible shared library, so programs in other languages (e.g., Python via ctypes) can call the API directly without spawning a CLI process.

Who might use it

Audience	Use case
AI research labs building autonomous agents that need to manipulate desktop software.
Developers of personal assistants (e.g., ChatGPT‑style bots) that want to extend beyond text‑only actions.
Automation engineers looking for a programmatic, cross‑platform way to script UI tasks without brittle screen‑scraping tools.
Tool makers who want to embed desktop‑control capabilities into their own applications via the FFI library.

Getting started (quick steps)

# Install the CLI (includes the native binary)
npm install -g agent-desktop

# Grant macOS accessibility permissions (or ensure Windows run‑as‑needed)
agent-desktop permissions --request

# Take a snapshot of Finder and list interactive elements
agent-desktop snapshot --app Finder -i

# Click the first element (replace the ref with the one you got)
agent-desktop click @s8f3k2p9:e3

# Verify the UI changed
agent-desktop snapshot --app Finder -i

For programmatic use, the Python snippet from the README shows loading the shared library and initializing it.

Strengths & limitations

Strengths	Limitations
• Works on both macOS and Windows out of the box.
• Headless operation avoids disrupting the user’s mouse/keyboard.
• Returns structured JSON with stable refs, making it easy for agents to reason about UI state.
• Provides a full‑featured FFI library for in‑process use.
• Covers a wide range of actions (clicks, typing, window management, notifications, clipboard, etc.).
• Open‑source under Apache‑2.0, with CI and tests.
• Planned Linux support.
• Integrated “skills” guide for agents.
• Binary distribution via npm simplifies installation.
• Detailed permission handling for macOS.
• Clear error codes (STALE_REF, AMBIGUOUS_TARGET).
• Supports batch commands and session tracing.
• Can operate on shell surfaces (taskbar, start menu) on Windows.
• Uses OS accessibility APIs – more reliable than pixel‑matching.
• No need for external UI‑automation tools like Selenium or AutoIt.
• Supports both ARM64 and x64 architectures.
• Provides a “cursor overlay” for visual debugging of agent actions.
• Ability to take a skeleton snapshot to reduce token count for LLM prompts.
• Extensible via the skills system.
• CI badge shows active testing.
• Release binaries are checksum‑verified.
• Supports launching Chromium apps via CDP.
• Can list and dismiss system notifications.
• Can manipulate clipboard directly.
• Can resize/move windows programmatically.
• Supports multi‑agent sessions.
• Provides a batch command to bundle multiple actions.
• Offers a trace command for debugging agent behavior.
• Includes a cursor-overlay for visualizing agent cursor.
• Supports list-surfaces to enumerate UI surfaces.
• Can take screenshots via screenshot command.
• Provides list-apps and list-windows for discovery.
• Supports open-system-surface on Windows.
• Can launch apps via CDP on Chromium.
• Supports wait command for synchronization.
• Provides session management for persistent contexts.
• Offers skills retrieval for agent guidance.
• Includes status command for health checks.
• Supports permissions query and request.
• Provides version info.
• Offers list-notifications and actions on them.
• Supports notification-action to interact with notifications.
• Provides clipboard-get, clipboard-set, clipboard-clear.
• Supports list-displays for multi‑monitor setups.
• Supports focus-window, resize-window, move-window.
• Supports minimize, maximize, restore.
• Supports open-system-surface on Windows.
• Supports launch with CDP for Chromium apps.
• Supports list-apps, list-windows.
• Supports list-displays.
• Supports list-notifications.
• Supports clipboard commands.
• Supports wait.
• Supports session and trace.
• Supports cursor-overlay.
• Supports batch.
• Supports skills.
• Supports status.
• Supports permissions.
• Supports version.
• Supports snapshot with --skeleton and --compact to reduce token size.
• Supports find, get, is for querying UI elements.
• Supports list-surfaces.
• Supports scroll, scroll-to.
• Supports expand, collapse.
• Supports select, toggle, check, uncheck.
• Supports clear, focus.
• Supports type, set-value.
• Supports click, double-click, triple-click, right-click.
• Supports press, hover, drag, mouse-move, mouse-click, mouse-wheel.
• Supports key-down, key-up, mouse-down, mouse-up (pending daemon).
• Supports launch, close-app.
• Supports list-apps, list-windows.
• Supports list-displays.
• Supports focus-window, resize-window, move-window.
• Supports minimize, maximize, restore.
• Supports open-system-surface.
• Supports list-notifications, dismiss-notification, dismiss-all-notifications.
• Supports notification-action.
• Supports clipboard-get, clipboard-set, clipboard-clear.
• Supports wait.
• Supports session, trace, cursor-overlay.
• Supports status, permissions, version, batch, skills.
• Supports list-surfaces.
• Supports snapshot with various flags.
• Supports find, get, is.
• Supports scroll, scroll-to.
• Supports expand, collapse.
• Supports select, toggle, check, uncheck.
• Supports clear, focus.
• Supports type, set-value.
• Supports click, double-click, triple-click, right-click.
• Supports press, hover, drag, mouse-move, mouse-click, mouse-wheel.
• Supports key-down, key-up, mouse-down, mouse-up.
• Supports launch, close-app.
• Supports list-apps, ...
• Limited to macOS & Windows now; Linux is only planned.
• Requires OS‑level accessibility permissions; setup can be fiddly for non‑technical users.
• Binary is unsigned on Windows, so SmartScreen/AV may block it.
• Some commands (key-down, mouse-down, etc.) are placeholders until a daemon is added.
• JSON output can be verbose; large apps may produce huge snapshots (tens of thousands of tokens).
• No built‑in LLM integration – you need to wire the JSON into your own model prompts.
• Requires Rust toolchain to build from source; not a pure‑JS package.
• Snapshot accuracy depends on the OS accessibility implementation, which can vary between apps.

---

This summary is based solely on the project's README.