remorses/usecomputer

What it solves

usecomputer provides a way for AI agents to interact with a computer's desktop environment. It solves the problem of bridging the gap between an LLM's decision-making and the actual execution of mouse and keyboard actions on macOS, Linux, and Windows.

How it works

It is a CLI tool and Node.js library backed by a native Zig binary. It allows agents to take screenshots, move the mouse, click, drag, and type text. To ensure accuracy, it uses a "coord-map" system that maps coordinates from a scaled screenshot (optimized for LLM vision models) back to the actual screen coordinates.

Who it’s for

Developers building AI agents (such as those using Claude Code or OpenCode) that need to automate desktop tasks and interact with GUI applications.

Highlights

• Cross-platform support: Works on macOS, Linux (X11), and Windows.
• Coordinate Mapping: Automatically handles scaling between model-friendly screenshots and real screen pixels.
• Window-scoped captures: Can capture specific application windows to improve model focus and accuracy.
• Advanced Input: Supports curved drag paths using quadratic Bezier control points and global input event streaming (via SSE) on macOS.
• Agent-friendly output: Integrates with the Kitty Graphics Protocol to send images directly to an agent's context window.