openclaw/Peekaboo

Peekaboo – macOS‑only screen‑aware automation

What it is – Peekaboo is a native macOS tool (CLI + menu‑bar app) that can see what’s on the screen, turn that visual information into a structured UI map, and then drive the UI with clicks, typing, key‑presses, scrolling, etc. It bundles a small “agent” that can be given a natural‑language instruction (e.g. “Open Safari, go to github.com, and search for Peekaboo”) and will orchestrate the observation‑action loop for you.

Why it matters – Modern AI‑assisted workflows often need a reliable way to let a language model interact with a desktop. Peekaboo provides the low‑level primitives (screen capture, accessibility inspection, synthetic input) and a thin LLM‑driven agent, all packaged for macOS 15+.

Key capabilities

Area	Commands / Features	Typical use
Observation	see, screen list, window list	Capture a screenshot or get a JSON UI tree of a specific app/window.
Interaction	click, type, press, scroll, drag, set-value, action	Perform semantic UI actions (click a button, type into a field) without bringing the app to the foreground.
App‑level control	app, window, menu, menubar, dock, dialog, space	List windows, target a specific PID, manipulate macOS menus or dialogs.
Agent / workflow	agent, capture	Give a natural‑language description; the built‑in agent plans a sequence of see + action commands, optionally using an LLM provider you configure.
Integration	mcp, browser, tools	Expose Peekaboo’s command set to external “Model‑Client‑Protocol” (MCP) clients such as Codex, Claude Code, Cursor, etc.

How you use it

1. Install – via Homebrew (brew install openclaw/tap/peekaboo), an npm MCP package, or a signed DMG for the menu‑bar app.
2. Grant permissions – screen‑recording and accessibility (plus a synthetic‑input permission) are required; peekaboo permissions status helps verify.
3. Run a command – e.g. peekaboo see --app Finder --json to get a UI map, then peekaboo click "Address and search bar" --app Safari --window-id 12345 to interact.
4. Agent mode – after configuring a model provider (peekaboo config), run peekaboo agent "Open Safari, go to github.com, and search for Peekaboo" --allow-foreground.

Who it’s for – Developers building AI‑driven macOS assistants, power‑users who want scriptable UI automation, and anyone integrating a language model with native macOS apps.

Project health – Actively maintained (CI badge, recent 4.9.0 release with new scoped menu‑bar clicks), open‑source MIT license, and a small ecosystem of community ports (Windows, Linux).

---

All details are taken directly from the repository’s README.