agent-sh/computer-use-linux

What it solves

computer-use-linux provides a way for AI agents (via the Model Context Protocol - MCP) to control a real Linux desktop. While most existing computer-use tools are macOS-only or rely on simple screen-scraping, this project enables agents to interact with Linux environments across various desktop environments (GNOME, KDE, Hyprland, i3, and COSMIC) with high reliability, specifically supporting Wayland and X11.

How it works

It operates as an MCP server and CLI tool written in Rust. It interfaces with the Linux accessibility tree (AT-SPI) to identify UI elements semantically rather than relying solely on pixel coordinates. For input, it uses a variety of backends including the RemoteDesktop portal, ydotool, wtype, and xdotool to simulate clicks, keystrokes, and scrolls. It also integrates with various window managers to list, focus, and resize windows, and provides a structured JSON "doctor" report to verify system readiness.

Who it’s for

Developers building AI agents that need to interact with local Linux desktop applications, as well as users of MCP-compatible hosts like Claude Desktop, Claude Code, Hermes Agent, and Codex Desktop.

Highlights

• Wayland Support: Implements multiple fallback mechanisms to ensure pointer and keyboard input works on Wayland compositors.
• Semantic Selection: Uses AT-SPI to allow agents to click or interact with elements by role, name, or text instead of just coordinates.
• Cros-Desktop Compatibility: Supports a wide range of environments including GNOME, KDE Plasma, Hyprland, i3, and COSMIC.
• Comprehensive Diagnostics: Includes a doctor command that provides a machine-readable readiness report of the system's capabilities.
• Safety Annotations: Provides MCP ToolAnnotations to help hosts distinguish between read-only observation and destructive UI mutations.