OSWorld: a benchmark for evaluating multimodal agents on open-ended tasks in real computer environments

What it solves

OSWorld provides a standardized benchmark and environment for testing multimodal AI agents on open-ended tasks within real computer operating systems. It addresses the difficulty of evaluating how well AI agents can actually navigate a desktop, use various software applications, and complete complex workflows in a realistic setting rather than in a simulated or restricted environment.

How it works

The project creates a controlled virtual machine (VM) environment (supporting Ubuntu and Windows) that agents can interact with. It supports multiple hosting providers including VMware, VirtualBox, Docker, and AWS for scalable parallel evaluation. Agents receive observations (such as screenshots) and execute actions (like mouse clicks or keyboard inputs) to complete tasks. The system includes a set of benchmark tasks across different domains (Office, Daily, Professional) and a verification system to score the agent's success rate based on the final state of the environment.

Who it’s for

It is designed for AI researchers and developers building multimodal agents, specifically those focusing on "computer-use" capabilities, GUI navigation, and autonomous software interaction.

Highlights

• Real OS Environments: Uses actual virtual machines instead of simplified simulators.
• Multi-Platform Support: Compatible with VMware, VirtualBox, Docker, and AWS.
• Scalable Evaluation: Supports parallel execution of tasks to significantly reduce evaluation time.
• Diverse Task Set: Includes a wide range of tasks across office productivity, daily activities, and professional software use.
• Verification Tools: Provides scripts for result analysis, video recording of agent trajectories, and manual task examination.