xlang-ai/OSWorld

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

What it solves

OSWorld provides a standardized benchmark and environment for evaluating multimodal AI agents on their ability to perform open-ended tasks within real computer operating systems. It addresses the difficulty of testing agents in realistic, complex desktop environments where they must interact with various software and files to achieve a goal.

How it works

The project creates a sandboxed desktop environment using virtual machines (VMs) across multiple providers, including VMware, VirtualBox, Docker, Modal, Daytona, and AWS. AI agents interact with this environment by receiving observations (such as screenshots) and executing actions (such as mouse clicks or keyboard inputs). The system supports both single-threaded and large-scale parallel execution to accelerate evaluation, and includes tools for recording trajectories and calculating success rates across different domains like Office, Daily, and Professional tasks.

Who it’s for

It is designed for researchers and developers building multimodal agents and "computer-use" AI that need a rigorous way to actually test if their models can navigate a real OS and complete complex software tasks.

Highlights

  • Multi-Platform Support: Works across VMware, VirtualBox, Docker, and cloud providers like AWS and Modal.
  • Real-World Complexity: Tests agents on actual software and operating systems rather than simulated APIs.
  • Parallel Evaluation: AWS integration allows for massive parallelization to reduce evaluation time to under one hour.
  • Comprehensive Metrics: Provides detailed success rate statistics categorized by task domain and category.

Related

  • Project
    benchflow-ai/benchflow

    A universal environment framework that runs AI agents against task environments and scores them through a standardized contract, treating benchmarks as frozen environments.

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    allenai/ScienceWorld

    A text-based virtual environment for benchmarking AI agents on tasks derived from a standardized elementary science curriculum.

  • Project
    benchflow-ai/skillsbench

    A benchmark for evaluating how AI agents leverage and compose modular skills—instructions, scripts, and resources—to perform specialized workflows.

  • Project
    harbor-framework/harbor

    Harbor is a Python framework for running and scaling benchmarks of AI agents and language models. Installable via `pip`/`uv`, it provides a CLI (`harbor run`) that lets you specify a dataset (e.g., Terminal‑Bench‑2.0), an agent (Claude‑Code, OpenHands, etc.), and a model, then executes the benchmark locally or on cloud providers (Daytona, Modal, etc.) with configurable parallelism. It also supports creating custom benchmarks and generating RL roll‑outs.