To3akaRin/mac-computer-use

TL;DR

mac‑computer‑use is an open‑source Agent Skill that lets a large‑language‑model‑driven agent control a macOS desktop. It can observe windows, read UI elements via macOS Accessibility (AX) or Chrome DevTools Protocol (CDP), issue clicks/keystrokes, and verify the result, all from a local command‑line tool that follows the open Agent Skills specification.

---

What it does (in plain language)

• Observe – List open apps, get window geometry, read UI controls (buttons, text fields, etc.) and, for Chromium‑based windows, inspect the page’s DOM via CDP.
• Act – Send native macOS input events (click, hover, scroll, type) or CDP input commands to interact with the target UI.
• Verify – Read back the UI state, take screenshots with ScreenCaptureKit, and return structured JSON so the agent knows whether the action succeeded.

The whole workflow is packaged as a skill that any agent runtime supporting the Agent Skills spec (Claude Code, Codex, Cursor, etc.) can load and invoke.

---

Who might use it

• Developers building AI assistants that need to automate macOS applications (e.g., opening a file, filling a form, running a script) without writing custom AppleScript or Selenium code.
• Researchers experimenting with embodied AI who want a reproducible, sandboxed way to let an LLM manipulate a real desktop.
• Power‑users who want a plug‑and‑play CLI to script UI actions that are hard to express with pure shell commands.

---

How to get started (step‑by‑step)

1. Install the skill via the Vercel Skills CLI:

   npx skills add To3akaRin/mac-computer-use
   

   This copies the mac-computer-use/ directory into your local skill store.
2. Make sure your Mac meets the requirements:
   • macOS 14 or newer, Apple Silicon (Intel may work but is untested).
   • Xcode Command‑Line Tools (Swift 6) – xcode-select --install.
   • Node 22.4+ (only for the CDP part).
   • Grant Accessibility and Screen Recording permissions to the terminal or the agent app you’ll run.
3. Run a simple observation command to see what windows are available:

   ./.skills/mac-computer-use/scripts/run.sh native windows
   

   The output is JSON listing each window’s ID, title, size, and process.
4. Pick a window and probe its UI (example with TextEdit):

   ./.skills/mac-computer-use/scripts/run.sh native ax \
       --window <WINDOW_ID> \
       --snapshot-out ./artifacts/window.json
   

   This writes a JSON description of all Accessibility elements in that window.
5. Perform an action – e.g., type “Hello” into the first text field:

   ./.skills/mac-computer-use/scripts/run.sh native type \
       --window <WINDOW_ID> \
       --element "AXTextField" \
       --text "Hello"
   

6. Verify – read the field back or take a screenshot:

   ./.skills/mac-computer-use/scripts/run.sh native shot \
       --window <WINDOW_ID> \
       --output ./artifacts/after.png
   

   The JSON result tells the agent whether the text was set correctly.

All commands support a --dry-run flag that checks parameters without actually sending input, which is useful for safety.

---

Core technical details (for the curious)

Aspect	Implementation
Language	Swift 6 for native macOS tooling; Node .js scripts for CDP interaction.
UI observation	macOS Accessibility (AX) APIs for native apps; Chrome DevTools Protocol (CDP) for Chromium‑based windows.
Input generation	CGEvent‑based mouse/keyboard events for native apps; CDP Input domain (click, insertText, etc.) for web pages.
Verification	Reads back AX values, CDP read‑only assertions, and captures screenshots via ScreenCaptureKit.
Packaging	Follows the open Agent Skills spec – a SKILL.md metadata file plus a scripts/ launcher that the runtime calls.
Zero runtime npm deps	The CDP client is a tiny self‑contained script; the skill does not require a separate server or API keys.
Safety rules	• Actions only after ≥2 s of user inactivity, ≤15 s wait. • --dry‑run preview mode. • Stale window snapshots are rejected.
Output format	JSON objects containing target, channel (native or cdp), status (success, unknown, refused), and any evidence (screenshots, AX readbacks).

---

Limitations & open questions

• App coverage – Works out‑of‑the‑box for apps that expose clean AX trees or a CDP port. Some third‑party tools may have custom controls that are hard to locate.
• Intel/macOS 14 – Not officially validated; the CI runs on Apple Silicon/macOS 15.
• No remote execution – The skill must run locally on the Mac; a Linux sandbox cannot control the desktop.
• Documentation language – Most guides (references/*.md) are currently in Chinese, which may hinder non‑Chinese contributors.

---

Quick reference cheat‑sheet

Command	Purpose	Example
native windows	List all windows	run.sh native windows
native ax	Dump Accessibility tree for a window	run.sh native ax --window 123 --snapshot-out out.json
native type	Type text into a specific AX element	run.sh native type --window 123 --element AXTextField --text "Hi"
native click	Click a UI element by coordinates or AX path	run.sh native click --window 123 --point 0.5,0.2
cdp snapshot	Capture DOM and screenshot of a Chromium page	node scripts/cdp.mjs snapshot --endpoint http://127.0.0.1:9222 --target PAGE_ID
native doctor	Run a self‑check (permissions, Swift, Node)	run.sh native doctor

---

Bottom line

mac‑computer‑use turns a macOS desktop into a deterministic, observable‑act‑verify interface that any LLM‑powered agent can call. It bridges the gap between high‑level language reasoning and low‑level UI manipulation without needing a cloud service, making it a solid building block for autonomous macOS agents.