lahfir/agent-desktop

Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates it — refs stay stable and actions stay safe to retry, instead of guessing from pixels.

What it solves

agent-desktop provides a reliable way for AI agents to interact with desktop applications. Unlike traditional methods that rely on visual pixel-guessing, it uses OS accessibility trees to see the real UI structure of any application, ensuring that element references remain stable and actions are safe to retry.

How it works

Built with Rust, the tool operates by observing the desktop's accessibility APIs to generate snapshots of the UI. It uses a "progressive skeleton traversal" method to reduce token usage by providing a shallow overview of dense apps and allowing the agent to drill down into specific regions of interest. It provides a native CLI and a C-ABI library for integration into various languages (Python, Node, Go, etc.). For Chromium-based apps, it can launch them with a Chrome DevTools Protocol (CDP) endpoint to allow deeper web-content driving via frameworks like Playwright or Puppeteer.

Who it’s for

Developers building AI agents that need to operate a computer desktop, specifically those requiring stable, structured UI interaction rather than purely visual-based automation.

Highlights

  • Accessibility-based interaction: Uses OS accessibility trees instead of pixels for stable element references.
  • Token efficiency: Reduces token usage by 78–96% in dense apps via progressive skeleton traversal.
  • Multi-language support: Offers a native Rust CLI and C-ABI bindings for Python, Swift, Go, Ruby, Node, and C.
  • Chromium Interop: Supports CDP for driving web contents in Chromium apps while keeping native menus and windows on the accessibility path.
  • Agent Cursor Overlay: Provides a visual, customizable cursor on macOS to show what an agent is doing in real-time.
  • Session Management: Supports shared sessions and tracing, allowing multi-agent workflows to share a namespace and export session timelines to HTML.

Related

  • Project
    vercel-labs/agent-browser

    A native Rust CLI that lets AI agents (or scripts) control Chrome via simple commands, using accessibility‑based element refs (`@eN`). It supports opening pages, taking snapshots, clicking/filling, extracting text, screenshots, network mocking, tab management, batch execution, and an experimental WebMCP tool system.

  • Project
    lycorp-jp/sim-use

    A cross-platform CLI that enables AI agents to observe and act on iOS and Android screens using token-efficient UI outlines and alias-based interactions.

  • Project
    keon/browser-control

    A fast Rust CLI that lets AI coding agents control a real browser via the Chrome DevTools Protocol using shell-native commands and stable element references.

  • Project
    Eynzof/Hermes-CN-Desktop

    A native Windows and macOS desktop client for Hermes Agent, providing a streamlined interface and bundled runtime for managing AI agents.

  • Project
    agent-sh/computer-use-linux

    A Rust-based MCP server and CLI that allows AI agents to control a Linux desktop, supporting Wayland and X11 across multiple desktop environments.