xlang-ai/OSWorld-V2

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

What it solves

OSWorld-V2 is a benchmark designed to evaluate the ability of AI agents to interact with real-world operating system environments. It provides a standardized way to test how well agents can perform complex tasks across a desktop OS, including interacting with various applications, mocked websites, and GitLab instances, while preventing benchmark leakage by gating task definitions and assets.

How it works

The project provides a controlled environment using virtual machines (via Docker or AWS) running Ubuntu. It uses a set of gated Python task classes and assets (distributed via Hugging Face) to define goals and evaluate success. Agents interact with the OS through a backend that captures screenshots and observations. To ensure reproducibility, the benchmark uses pinned releases that synchronize the code, task files, assets, and VM images. It also includes mocked websites and a self-hosted GitLab setup to simulate realistic web-based workflows without relying on live, unstable external sites.

Who it’s for

It is intended for researchers and developers building multimodal AI agents (such as those based on Claude, GPT, or Qwen) that are designed to operate computers and automate desktop-level tasks.

Highlights

  • Multi-Provider Support: Compatible with Docker for local Linux servers and AWS for large-scale parallel evaluation.
  • Leakage Prevention: Task classes and assets are gated on Hugging Face to prevent agents from finding answers in their training data.
  • Reproducible Releases: Uses a strict versioning system (e.g., v2.1) to pin all components, including VM images and task assets.
  • Realistic Simulation: Includes mocked websites and GitLab integration to test complex, multi-app workflows.
  • Trajectory Analysis: Provides tools for manual task examination and trajectory viewing to debug agent behavior.

Related

  • Project
    xlang-ai/OSWorld

    OSWorld is an open‑source benchmark that lets you run multimodal LLM agents inside real Ubuntu/Windows virtual machines. It provides a Python library (`desktop‑env`) to launch VMs via VMware, VirtualBox, Docker, AWS, Modal, or Daytona, a set of realistic desktop tasks, reference GPT‑4o agents, and tools for scoring and visualising results.

  • Project
    scaleapi/SWE-bench_Pro-os

    A challenging benchmark for evaluating LLMs and AI agents on long-horizon software engineering tasks by requiring them to generate patches for real-world codebase issues.

  • Project
    allenai/ScienceWorld

    A text-based virtual environment for benchmarking AI agents on tasks derived from a standardized elementary science curriculum.

  • Project
    hkust-nlp/Toolathlon

    Toolathlon is a benchmark for assessing language agents' general tool-use capabilities across 600+ diverse tools in realistic, long-horizon software environments.

  • Project
    simular-ai/Agent-S

    An open-source computer use agent framework that operates a real GUI via mouse, keyboard, and screen to complete natural-language tasks across desktop and web applications.