xlang-ai/OSWorld

OSWorld – A Benchmark for Multimodal Agents in Real Desktop Environments

What it is – OSWorld is an open‑source benchmark that provides fully‑featured virtual machines (Ubuntu and Windows) and a Python library (desktop‑env) for evaluating multimodal AI agents (LLMs with vision, tool‑use, etc.) on open‑ended, real‑world computer tasks. The repository ships VM images, a unified environment API, and a set of 300+ task specifications ranging from office work to software development.

Key components

• DesktopEnv – a thin wrapper that launches a VM via VMware, VirtualBox, Docker (KVM), Modal, Daytona, or AWS and exposes actions (mouse/keyboard, file ops, OS commands) and observations (screenshots, OCR, UI hierarchy). Installable via pip install desktop‑env.
• Benchmark data – JSON files describing each task, required credentials, and evaluation metrics. A data‑viewer web UI lets you explore examples.
• Reference agents – scripts (run.py, run_multienv.py) that run a baseline GPT‑4o‑based agent using pure‑screenshot observations. Parallel execution is supported to speed up large‑scale evaluation.
• Evaluation tools – show_result.py and a manual‑examination script to compute per‑domain success rates, generate videos, and inspect individual trajectories.

Why it matters – Most existing LLM benchmarks are text‑only or simulated. OSWorld lets researchers test agents in actual operating systems, requiring perception, reasoning, and tool use under realistic constraints (passwords, network proxies, Google account OAuth, etc.). The project also provides a public leaderboard (via a verification process) and a “OSWorld‑Verified” update that improves signal quality and adds cloud‑scale parallel evaluation.

Getting started

1. Install – Clone the repo and run pip install -r requirements.txt (or just pip install desktop‑env). You need Python ≥ 3.10.
2. Choose a provider –
   • Desktop: install VMware Workstation/Fusion (or VirtualBox) and run vmrun -T ws list to verify.
   • Docker: ensure KVM support, install Docker, then set provider_name=docker.
   • Cloud: follow the AWS/Modal/Daytona guides for sandboxed VM images.
3. Run a quick demo – python quickstart.py creates a VM, logs in, and performs a simple right‑click action.
4. Run the baseline – set OPENAI_API_KEY and execute the run.py or run_multienv.py commands shown in the README to evaluate the GPT‑4o agent.
5. Inspect results – python show_result.py prints success rates; add --detailed for per‑task scores.

Extending the benchmark

• Implement your own agent by conforming to the mm_agents interface (see its README).
• Plug in a new provider by adding a class under desktop_env/providers/ that implements reset, step, and close.
• Contribute new task definitions or VM images via pull‑requests.

Community & support – The project maintains a Discord, accepts PRs, and provides a verification pipeline for leaderboard submissions. Recent updates (2025‑07‑28) introduced OSWorld‑Verified, added AWS parallelism, and refined benchmark signals.

License – Apache 2.0.

---

TL;DR – OSWorld is a real‑OS, multimodal agent benchmark with ready‑to‑run VM environments, a Python API, baseline agents, and evaluation scripts, aimed at measuring how well LLM‑based agents can actually use a computer.