trycua/cua

Scale computer-use 2.0 with open-source drivers, cross-OS fleets, and benchmarks for training, evaluation, and data generation.

What it solves

Cua provides a comprehensive infrastructure for "computer-use" AI agents, allowing them to interact with desktop operating systems (macOS, Windows, Linux, Android) across both local and cloud environments. It solves the problem of providing agents with a consistent API to perform actions like clicking, typing, and taking screenshots without interfering with the user's own cursor or focus.

How it works

Cua operates through several integrated components:

  • Cua Drivers: These allow agents to drive native desktop apps in the background, ensuring that AI actions do not steal the cursor or focus from the user.
  • Cua Sandbox: An SDK that provides a unified API for managing agent-ready sandboxes (VMs or containers) across different operating systems.
  • Cua-Bench: A benchmarking and RL environment that allows developers to evaluate agents on datasets like OSWorld and ScreenSpot, and export trajectories for training.
  • Lume: A virtualization tool specifically for macOS and Linux VMs on Apple Silicon, providing near-native performance using Apple's Virtualization.Framework.

Who it’s for

  • AI Agent Developers: Those building agents that need to interact with desktop software autonomously.
  • ML Researchers: Those training or evaluating computer-use models using RL environments and benchmarks.
  • Software Engineers: Developers needing scalable, ephemeral macOS, Windows, or Linux environments for automated testing or agentic workflows.

Highlights

  • Cross-OS Support: Unified API for macOS, Windows, Linux, and Android.
  • Background Execution: Drivers that allow agents to interact with apps without stealing the user's cursor.
  • Integrated Benchmarking: Built-in support for OSWorld, ScreenSpot, and Windows Arena.
  • Apple Silicon Optimization: High-performance macOS virtualization via Lume.
  • Flexible Deployment: Supports both cloud-hosted sandboxes and local QEMU-based environments.

Related

  • Dispatch
    Driving macOS Apps in the Background: Cua Driver's Solution for Agent-Based Automation

    Cua Driver addresses the long-standing challenge of running UI automation on macOS without disrupting the user's session, enabling agents to interact with applications in the background by leveraging a novel approach to event posting and window management. This innovation paves the way for more seamless and concurrent agent-driven workflows.

  • Project
    Paca-AI/paca

    An open-source, self-hosted project management platform where AI agents and humans collaborate as equal teammates within a Scrum team.

  • Project
    opensandbox-group/OpenSandbox

    A general-purpose sandbox platform for AI applications that provides secure, isolated environments for code execution, agent evaluation, and RL training.

  • Project
    madarco/agentbox

    A tool for running multiple AI agents in parallel within sandboxed VMs or Docker containers, either locally or in the cloud, with persistent IDE and shell access.

  • Project
    paradigmxyz/centaur

    A self-hosted agent platform for teams that provides shared AI agents with secure Kubernetes sandboxes and Slack integration for executing real operational work.