shhivv/arc-cua

Superfast action layer for computer-use agents, on MacOS.

What it solves

arc-cua addresses the inefficiency of using expensive frontier LLMs to handle every single UI interaction in computer-use agents. It separates high-level planning from low-level execution, allowing a planner to hand off bounded subtasks to a faster, specialized decision model that handles the repetitive "observe-click-observe" loop.

How it works

The system operates as a two-layer architecture:

  1. Planner/Agent: A high-level LLM or deterministic planner generates a subtask payload containing a goal, inputs, verification criteria, and constraints.
  2. Decision Loop (arc-cua): A fast decision model (such as JEV or OpenAI's Decisions API) receives the subtask and executes the UI loop. It observes the desktop state using a hybrid of Accessibility (AX) and local OCR, builds a legal action space of available controls, and selects the next action until the goal is verified or the subtask is blocked.

The project also includes arc-driver, a standalone macOS driver that allows agents to read and act on application windows in the background without moving the user's pointer or changing the frontmost app.

Who it’s for

Developers building computer-use agents who want to reduce the number of expensive reasoning calls and improve execution speed and reliability on macOS and in Chrome browsers.

Highlights

  • Background Control: Acts on macOS apps in the background, meaning the user can continue working while the agent operates on minimized or hidden windows.
  • Hybrid Perception: Combines macOS Accessibility (AX) for semantic controls and Apple Vision OCR for apps with poor accessibility.
  • Decision-Model Separation: Plugs into various decision backends (JEV, OpenAI Decisions) to handle low-level UI navigation.
  • Safety Guards: Includes allowed_risks to prevent the model from clicking consequential buttons (like "delete" or "purchase") unless explicitly permitted.
  • Secret Handling: Supports secret_inputs to redact sensitive data from the decision model's view while still allowing the runtime to enter the values.
  • Cross-Platform Browser Support: Includes a ChromeBackend for background browser automation across macOS, Linux, and Windows.

Related

  • Project
    openma-ai/open-managed-agents

    An open-source, self-hosted alternative to Claude Managed Agents and OpenAI Agents API that provides durable sessions, sandboxed tools, and encrypted credential storage.

  • Project
    lbx154/Argus

    An autonomous agent framework for long-running research and engineering that uses a four-role system to ensure persistent, reviewed autonomy without constant human supervision.

  • Project
    yohasebe/openai-chat-api-workflow

    An Alfred 5 workflow that integrates OpenAI's API to provide system-wide access to GPT models, image generation, file analysis, and speech tools on macOS.

  • Project
    AaravKashyap12/advise-project-approach

    An agent skill that forces AI agents to provide evidence-based project planning and architectural advice by researching real-world comparables and constraints.