openai/openai-cua-sample-app

What it solves

This project provides reference implementations for building "computer-use agents"—AI systems that can operate software interfaces to complete tasks by inspecting the screen, choosing actions, and verifying results.

How it works

The system uses a loop where a model writes code to interact with software. A persistent runtime maintains state and helper functions between calls, allowing the model to combine multiple actions and process observations without needing a round trip to the model for every single step. The repository includes two distinct implementations:

• JavaScript/Playwright: Controls a web browser using Playwright locators and screenshots.
• Python/PyAutoGUI: Controls a desktop environment via screenshots, mouse input, and keystrokes.

Who it’s for

Developers looking to implement agentic workflows where an AI needs to control a browser or a desktop OS to perform tasks.

Highlights

• Dual-Runtime Support: Includes both a browser-based (Playwright) and a desktop-based (PyAutoGUI) agent.
• Persistent State: Uses a runtime that keeps state available between model calls to reduce latency and context usage.
• Scenario Labs: Includes a set of lab templates (Kanban board, drawing canvas, hotel booking) to test and prompt the agents.
• Traceability: Provides a web console to inspect recorded traces and screenshot timelines to review agent behavior.