openai/openai-cua-sample-app

What it solves

This project provides reference implementations for "computer-use" agents—AI systems capable of operating software interfaces to complete tasks. It addresses the challenge of building autonomous agents that can inspect a screen, decide on an action, execute it via code, and verify the outcome through a feedback loop.

How it works

The project implements a persistent runtime loop where the AI model writes and executes code to interact with software. It includes two distinct implementations:

• JavaScript/Playwright: Uses a headless browser to perform actions and capture state.
• Python/PyAutoGUI: Controls a desktop environment by capturing screenshots and simulating mouse and keyboard inputs.

Both implementations share a common architecture consisting of a web console for managing tasks, a core agent loop that handles model-code-feedback interactions, and a set of "labs" (mock environments like booking or kanban systems) to test agent performance.

Who it’s for

This repository is designed for developers and researchers building agentic systems who want to understand how to implement computer-use capabilities using standard libraries and the OpenAI API. It serves as a practical guide for building persistent, stateful agent loops.

Highlights

• Includes two complete agent implementations using Playwright (web) and PyAutoGUI (desktop).
• Features a shared web console for managing scenarios, editing prompts, and monitoring agent runs.
• Provides "lab" templates (booking, kanban, paint) to simulate real-world tasks.
• Demonstrates how to use the OpenAI SDK ComputerToolset for building robust agent loops.
• Includes built-in support for replaying agent traces and inspecting screenshot timelines to debug agent behavior.