ComputerUseAgent/AutoCua

What it solves

AutoCua is a framework for building AI agents that can use a computer just like a person: clicking real buttons, typing into real apps, browsing your own Chrome profile, and even driving a paired iPhone. It solves the fragmentation problem—most agents work only inside a browser or a sandbox—by providing a single harness that works across desktop apps, browsers, shell terminals, and mobile (iOS). It also tackles common failure points of such agents: guessing at screen coordinates, being blocked by permission dialogs, hitting context limits on long tasks, and sending telemetry.

How it works

Under the hood, AutoCua runs a hierarchy of specialized agents. A parent agent watches the screen and decides what to do, then delegates to sub-agents for code work, browser interactions, or web research. Each step, the agent receives a numbered list of interactive elements (built from the OS accessibility tree plus OCR) and a screenshot with those numbers painted on, so it never has to guess pixel locations. Input is injected at the operating system's own input layer (e.g., macOS's kCGHIDEventTap, Windows SendInput, or even a kernel driver for UAC prompts), so apps see genuine mouse and keyboard events. The harness supports multiple modes—desktop, shell, web (via a Rust-based Chrome extension), and iOS—and includes a compression agent that watches the conversation and inserts a handoff summary mid-run to prevent context exhaustion.

Who it's for

AutoCua is aimed at developers, testers, and power users who want to automate real computer tasks—from checking a macOS version to filling an online shopping cart—without building a bespoke integration for each app. It's open-source (MIT) and installable via pip, with a simple Python API: one run_agent() call with mode, provider, model, and task. A desktop app is also included for visual monitoring.

Highlights

• Works across Windows, macOS, and Linux (GNOME), plus iOS via a paired device or simulator.
• Drives real desktop apps with genuine mouse/keyboard events, no accessibility hacks.
• Reuses your existing browser profile and cookies via a lightweight Chrome extension (no debugging port in normal runs).
• Multi-agent architecture (parent, sub-agents, compression agent) for complex tasks.
• No telemetry—the only external communication is to your chosen LLM provider.
• Supports 48 models from 9 providers, and can be extended.
• One-line API: run_agent(mode, provider, model, task).