remorses/usecomputer

usecomputer – Desktop‑automation CLI for AI agents

What it is – usecomputer is a cross‑platform command‑line tool (and accompanying npm library) that lets an AI agent interact with a real computer screen. It can take screenshots, move the mouse, click, drag, scroll, type text, press shortcuts, and even stream global input events. The heavy lifting is done by a native Zig backend, so the tool runs without a Node.js runtime after installation.

Why it matters for AI – Modern “computer‑use” agents (Claude, OpenAI, Anthropic, OpenCode, etc.) need a reliable way to close the perception‑action loop:

1. Capture the screen (or a specific window) → feed the image to a vision model.
2. Decide what to do based on the model’s output.
3. Act by moving the mouse, clicking, typing, etc.
4. Repeat. usecomputer provides the exact CLI commands and a TypeScript API that match this loop, plus a coord‑map system that translates screenshot‑pixel coordinates back to real‑screen coordinates, eliminating the common “off‑by‑a‑few‑pixels” problem.

Key features

• Screenshot (full‑desktop or window‑scoped) with automatic scaling (max 1568 px) and a coordMap string that records the mapping between screenshot space and screen space.
• Mouse: move, click (single/double), drag (straight or quadratic‑bezier curves), scroll.
• Keyboard: type (with optional delay, stdin streaming, chunking) and press for single keys or chord shortcuts.
• Observe: stream all mouse/keyboard events as Server‑Sent Events (SSE) – useful for agents that need to watch what a user does.
• Native Zig backend – no Node.js runtime needed after npm install -g usecomputer.
• Cross‑platform: works on macOS, Linux (X11/XWayland), and Windows (interactive desktop only). Currently the global‑event observe command is macOS‑only.
• Agent‑friendly graphics: when AGENT_GRAPHICS=kitty the screenshot is emitted inline via the Kitty Graphics Protocol, allowing agents like OpenCode to receive the image without a separate file read.
• Library API – the npm package mirrors the CLI’s native functions (screenshot, click, mouseMove, type, press, observe, etc.) and provides TypeScript types for strong editor support.

Typical workflow

# 1️⃣ Capture the screen (or a window) and get the coord‑map
usecomputer screenshot ./screen.png --json
# → {"path":"./screen.png","coordMap":"0,0,1600,900,1568,882",…}

# 2️⃣ Feed the PNG to your vision model (e.g., Claude, OpenAI, etc.)
#    The model returns an action, e.g. click at (400,220)

# 3️⃣ Execute the action using the same coord‑map
usecomputer click -x 400 -y 220 --coord-map "0,0,1600,900,1568,882"

The same pattern works for drags, scrolls, or typing.

Installation

npm install -g usecomputer   # installs the CLI and the TypeScript library

On macOS you must grant the terminal Accessibility and Input Monitoring permissions; on Linux ensure an X11 DISPLAY is set; on Windows run in an interactive desktop session.

Using it from code (TypeScript example)

import * as usecomputer from 'usecomputer'

const shot = await usecomputer.screenshot({ path: './tmp.png', display: null, window: null, region: null, annotate: null })
const map = usecomputer.parseCoordMapOrThrow(shot.coordMap)
const point = usecomputer.mapPointFromCoordMap({ point: { x: 400, y: 220 }, coordMap: map })
await usecomputer.click({ point, button: 'left', count: 1 })

All functions accept null for optional native fields, mirroring the CLI’s argument shape.

Who should use it

• Developers building AI agents that need to control a desktop (computer‑use tools, autonomous assistants, RPA‑style bots).
• Researchers experimenting with vision‑language models that act on a GUI.
• Anyone needing a lightweight, scriptable way to automate macOS/Linux/Windows without heavyweight UI‑automation frameworks.

Related ecosystem

• npx skills add remorses/usecomputer registers a skill for agents like OpenCode, teaching them the screenshot‑→‑act‑→‑screenshot loop.
• The kitty‑graphics‑agent plugin can automatically set AGENT_GRAPHICS=kitty so screenshots appear directly in an LLM’s context window.
• Example integrations for OpenAI, Anthropic, and OpenCode are provided in the README, showing how to translate the model’s tool calls into usecomputer actions.

---

All details above are taken directly from the repository’s README; no additional features have been inferred.