xlang-ai/OSWorld

What it solves

OSWorld provides a standardized benchmark and environment for evaluating multimodal AI agents on their ability to perform open-ended tasks within real computer operating systems. It addresses the difficulty of testing agents in authentic desktop environments where they must interact with various software applications and system settings.

How it works

The project creates a virtualized desktop environment (supporting Ubuntu and Windows) that AI agents can interact with. It supports multiple hosting providers, including VMware, VirtualBox, Docker, AWS, and Daytona, allowing for both local and cloud-based parallel evaluation. Agents receive observations (such as screenshots) and execute actions (such as mouse clicks or keyboard input) to complete tasks. The system includes a set of benchmark tasks across different domains (Office, Daily, Professional) and provides tools to record execution trajectories and calculate success rates.

Who it’s for

It is designed for researchers and developers building multimodal agents and "computer-use" AI that need a rigorous way to measure how well their models can navigate real-world software and operating systems.

Highlights

• Multi-Platform Support: Compatible with VMware, VirtualBox, Docker, AWS, and Daytona.
• Parallel Evaluation: AWS integration allows for large-scale parallelization to significantly reduce evaluation time.
• Real-World Tasks: Includes a diverse set of benchmark tasks spanning professional and daily computer use.
• Comprehensive Tooling: Provides scripts for running baseline agents, manual task examination, and result visualization.