AmrDab/clawdcursor

What it solves

Clawd Cursor provides a safe, cross-platform way for AI agents to control a desktop computer. It replaces brittle, expensive vision-only approaches (which rely on screenshots for every action) with a tiered perception system that prioritizes accessibility trees and OCR, making desktop automation faster, cheaper, and more private.

How it works

The system acts as a Model Context Protocol (MCP) server that integrates with agents like Claude Code or Cursor. It uses a "UI State Compiler" to fuse accessibility data and OCR into a map of the screen, assigning stable IDs (el_NN) to elements so the agent can interact with them by identity rather than pixel coordinates.

It employs a "cheapest-tier-first" perception ladder:

1. Structured (a11y): Uses the accessibility tree (free).
2. OCR: Uses OS-level text recognition (cheap).
3. DOM: Uses Chrome DevTools Protocol for web-based content (medium).
4. Vision: Uses screenshots as a last resort (expensive).

Every action is routed through a single safety gate (safety.evaluate()) for human confirmation or blocking, and the system verifies actions by checking the live screen for deviations after an event occurs.

Who it’s for

Developers building AI agents that need to interact with native desktop applications, legacy software, or any GUI without a public API, across Windows, macOS, and Linux.

Highlights

• Tiered Perception: Reduces token costs by avoiding vision models unless absolutely necessary.
• Action Verification: Detects if a UI action actually produced the intended result instead of assuming success.
• Cross-Platform: Native support for Windows, macOS, and Linux.
• Safety First: Centralized safety chokepoint and a visible "desktop control in progress" banner with a kill switch.
• MCP Native: Easy integration with any MCP-capable host (Claude Code, Cursor, Windsurf, etc.).