Tongyi-MAI/OpenGCUA

OpenGCUA – Open Generalist Computer‑Use Agent

What it is

• An open‑source stack that lets large language models (including frontier models like GPT‑6 Astra, Claude Opus 5, Gemini 3.8 Flash, etc.) operate a real computer. It combines visual observation (screenshots, accessibility trees), programmatic APIs, shell commands, and GUI actions into code‑executed workflows.
• The project releases three main pieces:
  1. macOS real‑device harness – a menu‑bar app (CuaDriver) that runs an agent on your own Mac.
  2. Cross‑Device SDK – a Python/Rust API that abstracts computers, browsers, Android phones, and Linux machines, exposing windows, apps, and accessibility data.
  3. Evaluation trajectories – recorded runs of eight frontier models on the OSWorld‑v2 benchmark, showing how OpenGCUA improves scores and reduces steps.

Why it matters

• Current “computer‑use agents” are usually closed, GUI‑only, and rely on a fixed harness. OpenGCUA moves the decision‑making (what to observe, how to act, how to organise work) into the model itself, and adds code execution so a model can write and run scripts that perform many actions in one step.
• The released harness and SDK let researchers experiment with the full stack, reproduce the paper’s results, and build their own GCUA models.

Key components

Component	Description
OSWorld integration example	Sample agents that use screenshots or code actions to solve tasks in the OSWorld‑v2 benchmark.
macOS real‑device harness	A SwiftUI/AppKit menu‑bar app (CuaDriver) that grants accessibility and screen‑recording permissions, runs the agent, and shows a floating task panel.
Cross‑Device SDK	Rust core with Python bindings (qwen_intelligence). Provides runtime.add_macos, runtime.session, app.ui.ax.observe(), etc., for unified control of macOS, Android, browsers, and Linux.
Evaluation trajectories	JSON/trace files for eight frontier models (GPT‑6 Astra, Claude Opus 5, Gemini 3.8 Flash, GLM‑5.3‑Flash, Qwen 3.8‑Max, Qwen 3.8‑Flash, Kimi K3, DeepSeek‑V4.1‑Flash).

Getting started (macOS)

# Clone and set up a virtual environment (uv recommended)
git clone https://github.com/Tongyi-MAI/OpenGCUA.git
cd OpenGCUA
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e ./packages/sdk -e ./packages/harness/runtime

# Install the native driver app and browser engine
.venv/bin/mu setup macos
.venv/bin/mu doctor macos
.venv/bin/mu setup browser --engine chromium
.venv/bin/mu doctor browser --engine chromium

# Configure your API key and model
cp packages/harness/configs/macos_demo.json packages/harness/configs/local.json
# edit local.json → set api_key, model_name, llm_base_url, llm_protocol

# Grant Accessibility & Screen Recording permissions to CuaDriver.app, then verify
.venv/bin/mu doctor macos

# Run the interactive terminal agent
.venv/bin/python packages/harness/runtime/mac_agent.py

The same SDK can be used from any Python script to control devices, e.g.:

import asyncio
from qwen_intelligence import Runtime

async def demo():
    async with Runtime() as rt:
        await rt.add_macos(name="local-mac")
        async with rt.session() as mu:
            mac = await mu.register("local-mac", name="mac")
            app = await mac.apps.get("com.apple.TextEdit")
            await mu.emit(await app.ui.ax.observe())
            await mu.emit(await app.windows.list())

asyncio.run(demo())

Results highlighted in the paper

• On OSWorld‑v2, OpenGCUA raises the mean partial score of all eight models from 45.3 → 64.9 (+19.6 points) and cuts average steps per task by 53 %.
• Code‑execution adds 6.1 points and reduces steps by 14.8 % compared to a GUI + CLI baseline.
• Real‑device tests on four macOS tasks show GPT‑6 Astra completing 4/4 tasks with far fewer reasoning tokens than Qwen 3.8‑Max.

Roadmap (as of the 2026‑10 preview)

• Release a dedicated GCUA model that learns to combine code execution, observation, and task tracking.
• Optimize execution efficiency (shorter reasoning, smarter harness decisions).
• Publish a full real‑device benchmark covering macOS, browsers, and phones.
• Improve reliability, failure recovery, and multi‑device continuity.

Community & contribution

• Contribute harness or SDK improvements, add new platform adapters, or help build evaluation environments.
• Open issues with research questions, submit pull requests, or join the WeChat/Discord groups (links in the README).

License

• Apache 2.0 (see LICENSE).

Citation

@misc{opengcua2026,
  title = {{OpenGCUA}: Toward a Fully Open Stack for Generalist Computer Use},
  author = {Hanzhang Zhou and Long Li and Quyu Kong and Lei Wang and Panrong Tong and Long Chen and Tianyu Xia and Gongjie Zhang and Xu Zhang and Chen Liu and Jianan Zhang and Chenglin Cai and Zhengwen Zeng and Steven Hoi and Yue Wang},
  year = {2026},
  eprint = {2610.22200},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2610.22200}
}

---

OpenGCUA is a genuine open‑source effort aimed at making generalist computer‑use agents transparent, reproducible, and extensible for the research community.