iFurySt/open-computer-use

What is open‑computer‑use?

open-computer-use (often invoked as ocu) is an open‑source Computer‑Use service that lets AI agents control a real desktop (macOS, Linux, Windows) through a standardized MCP (Message‑Based Control Protocol) interface. It implements the same kind of “non‑intrusive” UI automation that OpenAI’s Codex Computer Use demonstrated, but it is fully community‑maintained and can be plugged into any MCP‑compatible client such as Codex, Claude Code, Gemini CLI, or custom agents.

Core capabilities

Feature	Description
Cross‑platform	Runs on macOS 14+, Linux, and Windows. macOS needs Accessibility & Screen‑Recording permissions; other OSes need no extra setup.
MCP server	Starts a local MCP JSON‑RPC server exposing nine native tools (e.g., `list_apps`, `press_key`, `get_app_state`) plus a `query` tool on macOS for targeted UI element lookup.
JavaScript REPL / one‑off eval	`ocu js <code>` runs a snippet in a sandboxed JS environment that can call the native `cua` bindings; `ocu repl` keeps the session alive for interactive scripting.
Skill packaging	Provides ready‑made skills for popular agents (Codex, Claude Code, Gemini CLI, opencode, DeepSeek Harness). Installing a skill writes the necessary MCP config into the agent’s config file.
Permission helper	`ocu doctor` (macOS) reports whether Accessibility, Screen Capture and other required permissions are granted, helping hosts automate onboarding.
Cursor Motion	An optional macOS cursor‑motion subsystem (open‑source) that moves the mouse based on public information from the Software.Inc team.
Automation scripts	Supports batch calls via `--calls` or `--calls-file`, allowing a sequence of UI actions to be executed in a single process.
Extensible CLI	Commands for installing into various agents, inspecting capabilities, running stress tests, and configuring persistent settings (e.g., screenshot format).

Quick start (for a human user)

# Install the npm package globally
npm i -g open-computer-use   # provides the `open-computer-use` and `ocu` commands

# First run – macOS will ask for Accessibility & Screen Recording permissions
open-computer-use   # or just `ocu`

Plugging into an AI agent

# Example: add to Codex (writes to ~/.codex/config.toml)
open-computer-use install-codex-mcp

# Or add manually to any MCP client
cat <<EOF > ~/.my‑agent‑config.json
{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"]
    }
  }
}
EOF

After the agent knows the MCP endpoint, it can issue calls like list_apps or press_key and receive structured JSON results.

How it works under the hood

Native runtime – A small compiled binary (bundled with the npm package) talks to the OS’s accessibility APIs to enumerate windows, read UI state, and synthesize mouse/keyboard events.
MCP bridge – The binary is launched by the ocu mcp command and speaks a JSON‑RPC protocol that the agent consumes. This keeps the agent language‑agnostic; any client that can speak MCP can use the service.
JavaScript bindings – The cua object exposed in the REPL mirrors the native tools, letting scripts call await cua.listApps() etc., making it easy to prototype automation logic.

Who might use this?

Developers building AI‑driven assistants that need to manipulate a real desktop (e.g., opening files, clicking UI, reading screen content).
Researchers evaluating “computer‑use” capabilities of large language models without relying on proprietary OpenAI services.
Power‑users who want a local, privacy‑preserving alternative to cloud‑hosted computer‑use APIs.

Documentation & community

Demos – Video demos for Codex App, Gemini CLI, and Linux are linked in the README.
Guides – docs/locked-use.en.md explains setup and permissions; docs/targeted-query.md shows how to locate UI elements by text/role.
Configuration – ocu config lets you view or change persistent settings (e.g., screenshot image format).
License – MIT, so you can freely modify and redistribute.

Bottom line

open-computer-use is a practical, open‑source implementation of the “computer use” paradigm that lets any MCP‑compatible AI agent control a desktop locally. It provides a ready‑made MCP server, skill packages for major agents, and a rich CLI for testing and scripting, making it a solid building block for anyone experimenting with AI‑driven UI automation.