agent-sh/computer-use-linux

Linux desktop control over MCP — AT-SPI, GNOME Shell, Wayland portals, ydotool

What it solves

It enables AI agents to interact with a live Linux desktop by overcoming the limitations of existing tools that are often macOS-only or rely on imprecise pixel-based OCR. It provides a way for agents to perform semantic actions (like clicking a button by its name) rather than just clicking specific coordinates.

How it works

The project acts as a Model Context Protocol (MCP) server that interfaces with Linux desktop environments. It uses several methods to gain control:

  • Observation: It reads accessibility trees (via AT-SPI), takes screenshots, and lists windows using compositor-aware backends (GNOME, KDE, Hyprland, i3, and COSMIC).
  • Input: It drives mouse and keyboard actions using Wayland portals or ydotool as a fallback.
  • Semantic Selection: It uses accessibility data to allow agents to select elements by role, name, or text, falling back to pixel coordinates when necessary.

Who it’s for

  • AI Agent Developers: Anyone building agents that need to operate a Linux desktop via MCP.
  • Linux Users: Users wanting to integrate desktop automation into workflows using tools like Claude Desktop, Claude Code, or Hermes Agent.

Highlights

  • Wayland Support: Built with a "Wayland-first" approach using desktop portals.
  • Compositor Awareness: Supports multiple window managers including GNOME, KDE, Hyprland, i3, and COSMIC.
  • Semantic Control: Supports interacting with UI elements by name and role via AT-SPI.
  • Diagnostic Tools: Includes a doctor command to generate a JSON readiness report for troubleshooting system requirements.