AmrDab/clawdcursor

What is Clawd Cursor?

Clawd Cursor is a local MCP (Model‑Control‑Protocol) server that lets any AI‑powered “agent” safely control a real desktop. It turns the screen into a structured UI map by merging the operating‑system accessibility tree with cheap OCR, assigning each element a stable el_NN identifier. Agents then issue actions (click, type, open app, etc.) by referring to those IDs instead of raw pixel coordinates.

Key points from the README:

Feature	Why it matters
Cross‑OS (Windows, macOS, Linux)	Works with any desktop the user runs.
Vision‑free primary perception	Uses the free accessibility tree first; OCR is a cheap fallback; full‑screen screenshots are only used when absolutely needed, keeping token‑cost low and keeping data local.
Self‑verification	After an action the server re‑reads the UI and reports a DEVIATION if the result doesn’t match the expectation (expect flag).
Single safety gate	Every request passes through safety.evaluate() which can allow, ask for user confirmation, or block the action.
MCP‑compatible	Exposes a standard stdin/stdout MCP interface, so any tool that supports MCP (Claude Code, Cursor, VS Code, Zed, OpenAI Codex, etc.) can plug it in with a one‑line config.
Toolbox	7 compound tool groups (computer, accessibility, window, system, browser, task, …) built on ~98 granular primitives, all routed through the safety gate.

The engine compiles a UI map (compile_ui), lets the agent find elements by name (find_button, find_field), performs actions, and then verifies the outcome. If verification fails, the loop can retry or report the deviation.

---

Who might use it?

• Developers building AI agents that need to interact with native desktop apps (email clients, office suites, IDEs, etc.) without writing custom UI automation code.
• Teams using Claude Desktop, Cursor, or other MCP‑enabled editors who want a drop‑in “desktop control” extension.
• Researchers experimenting with safe, locally‑run AI agents who want a deterministic, verifiable way to let the model manipulate the OS.

---

Quick start (from the README)

1. Install – npm i -g clawdcursor (or use the one‑liner npx -y clawdcursor).
2. Grant consent – clawdcursor consent --accept (and on macOS clawdcursor grant for Accessibility + Screen‑Recording).
3. Add to an MCP host – e.g. in VS Code’s .vscode/mcp.json:

   {
     "servers": {"clawdcursor": {"command": "clawdcursor", "args": ["mcp", "--compact"]}}
   }
   

   Similar JSON snippets are provided for Claude Code, Cursor, Zed, OpenAI Codex, etc.
4. Ask the agent – “Open Outlook and reply to the latest email from Sarah.” The agent will:
   • Compile the UI map,
   • Locate the “Reply” button by its stable ID,
   • Click it, type the response, and
   • Verify the email was sent.

---

How it works under the hood

1. Perception tiering – The server first reads the OS accessibility tree (free). If the needed element isn’t found, it runs OCR (still cheap). Only when a task truly needs pixel‑level data does it take a screenshot and optionally feed it to a vision model.
2. Stable element IDs – Every UI element gets an el_NN identifier that survives DPI changes, window moves, and layout shifts, so the agent can refer to the same element across multiple steps.
3. Safety checkpoint – All calls pass through safety.evaluate(). The gate can automatically allow, request a human confirmation, or block the action.
4. Verification – After an action, the server re‑observes the UI (using the same tiered perception) and returns a DEVIATION if the state doesn’t match the expected outcome.
5. Toolbox – 7 high‑level tool groups expose ~150 verbs (click, type, read_tree, open_app, etc.). A “granular” surface of ~98 primitives is also available for debugging or custom hosts.

---

Limitations & considerations

• OS prerequisites – macOS needs Xcode command‑line tools; Linux requires tesseract-ocr, python3-gi, gir1.2-atspi-2.0, and on Wayland ydotool for synthetic input.
• Vision fallback – Pure‑canvas apps (e.g., games, Paint) still rely on screenshot‑based clicks, which are slower and cost more tokens.
• Safety UI – A small on‑screen banner with a blinking red dot indicates when the desktop is being driven; double‑clicking it kills the session.
• Local‑only – No cloud component; all processing stays on the user’s machine, which is great for privacy but means the host must have enough compute for OCR and any optional vision model.

---

Bottom line

Clawd Cursor provides a privacy‑preserving, verifiable, cross‑platform bridge between LLM‑driven agents and the real desktop. By preferring accessibility data over raw screenshots and by enforcing a single safety gate, it aims to make “let the AI use my computer” both cheap and safe.