xlang-ai/OSWorld-V2

What it solves

OSWorld-V2 provides a standardized benchmark and environment for evaluating AI agents capable of "computer use." It addresses the difficulty of testing agents on long-horizon, real-world tasks that require interacting with a full operating system, including browsers, applications, and file systems, while preventing benchmark leakage by gating task definitions and assets.

How it works

The project creates a controlled virtual machine (VM) environment (via Docker or AWS) where agents can execute actions. It uses a set of gated Python task classes and assets hosted on Hugging Face to define goals and evaluate success. To ensure realism and stability, it employs mocked websites and a self-hosted GitLab instance for specific tasks. The system supports multi-environment parallel execution and provides tools for manual task examination and trajectory recording.

Who it’s for

It is designed for AI researchers and developers building multimodal agents that can perceive screen content (via screenshots) and interact with a computer interface to complete complex, multi-step workflows.

Highlights

• Real-world Simulation: Tests agents on actual OS environments rather than simplified APIs.
• Leakage Prevention: Task classes and assets are distributed through gated Hugging Face datasets to prevent agents from finding answers in their training data.
• Flexible Infrastructure: Supports both local Docker deployments and scalable AWS-based parallel evaluation.
• Comprehensive Tooling: Includes a trajectory viewer, manual examination scripts, and migration tools for agents moving from OSWorld 1.0.