agent-sh/computer-use-linux

Native Linux desktop control for AI agents (MCP, Pi, CLI) — semantic AT-SPI trees, screenshots, window targeting, and input across GNOME, KDE, Hyprland, niri, COSMIC, i3, and X11.

What it solves

computer-use-linux enables AI agents to control a real Linux desktop. It overcomes the common limitation where most "computer use" tools are macOS-only or rely on fragile X11-only methods, providing a robust way for agents to interact with various Linux window managers and desktop environments, including Wayland-first support.

How it works

It operates as a Rust-based Model Context Protocol (MCP) server and CLI. It interacts with the desktop through several mechanisms:

  • Accessibility Trees: Uses AT-SPI to read the UI structure and perform semantic actions (like clicking a specific button by name) rather than relying solely on pixel coordinates.
  • Compositor Awareness: It uses specific backends for different environments (GNOME, KDE, Hyprland, niri, i3, COSMIC) to list and focus windows accurately.
  • Input Simulation: Employs the org.freedesktop.portal.RemoteDesktop interface on Wayland, with ydotool as a fallback, and xdotool for X11 sessions.
  • Visual Feedback: Includes an on-screen indicator that shows a tinted cursor and status pill so users can see exactly what the agent is doing in real-time.

Who it’s for

  • AI Agent Developers: Those building agents that need to interact with Linux software.
  • MCP Host Users: Users of clients like Claude Desktop, Codex Desktop, or Hermes Agent who want to give their agents control over a Linux machine.
  • System Administrators: Users needing to automate complex desktop UI tasks via LLMs.

Highlights

  • Broad Linux Support: Works across GNOME, KDE/KWin, Hyprland, niri, i3, and COSMIC.
  • Wayland-First: Specifically designed to handle the complexities of Wayland pointer and keyboard actions.
  • Semantic Interaction: Supports targeting elements by role, name, or text via AT-SPI, reducing reliance on coordinate-based clicking.
  • Detailed Diagnostics: Includes a doctor command that provides a structured JSON readiness report to identify blockers.
  • Safety Annotations: Uses MCP ToolAnnotations to distinguish between read-only observation, setup mutators, and destructive desktop actions.

Related

  • Project
    mrpulor-gh/nuphus-mcp

    A desktop and browser automation MCP server that enables AI agents to control the mouse, keyboard, and Chrome, and perceive the screen using local OCR and vision LLMs.

  • Project
    CursorTouch/Windows-MCP

    Windows‑MCP is an open‑source MCP server that exposes Windows UI‑automation tools (mouse, keyboard, window management, screenshots, browser DOM mode, etc.) to any LLM‑based AI agent. Installable via PyPI, it runs on Windows 10/11 with Python 3.14+, supports local `stdio` or network transports (SSE/HTTP), and includes optional auth, TLS, IP allow‑lists, and OAuth for secure remote access.

  • Project
    mediar-ai/mcp-server-macos-use

    An MCP server that allows AI models to control macOS applications by leveraging accessibility APIs to traverse the UI and simulate user inputs.

  • Project
    ip2a/mcpstore

    A high-performance MCP management system built with Rust that orchestrates Model Context Protocol services for AI agents and applications.