ClawBench – Benchmark for AI Browser Agents

What it is – An open‑source benchmark that measures how well AI agents can complete everyday tasks on real‑world websites (booking travel, ordering food, applying for jobs, managing email, etc.). It ships two corpora of test cases (V1 = 152 tasks on 143 sites, V2 = 129 tasks on 63 sites) and a 5‑layer recording pipeline that captures each run, then scores it with an LLM‑based evaluator against human references.

---

Quick start (run a benchmark yourself)

# 1️⃣ Clone the repo
git clone https://github.com/TIGER-AI-Lab/ClawBench.git && cd ClawBench

# 2️⃣ Install the CLI (choose one)
uv tool install clawbench-eval          # preferred, fast
# or
pipx install clawbench-eval
# or
python -m pip install clawbench-eval

# 3️⃣ Install a container engine (Docker or Podman) – the benchmark runs Chrome inside a container.
# e.g. on macOS: brew install --cask docker && open -a Docker

# 4️⃣ First run builds the container image (Chromium + ffmpeg + noVNC + agent harnesses)
clawbench   # creates a default models/models.yaml you can edit

# 5️⃣ Edit the config to point to your LLM judge (e.g. DeepSeek‑v4‑pro) and any credentials you need.
$EDITOR models/models.yaml

# 6️⃣ Run a task (interactive TUI lets you pick a model and a test case)
clawbench

The CLI provides sub‑commands for batch runs, rescoring, analysis, and reproducing published results (clawbench-run, clawbench-batch, clawbench-rescore, …).

---

Main features

Feature	What it gives you
Live‑web tasks	Tests run against real websites (no mocks).
Two corpora	V1 (152 tasks, 143 sites) and V2 (129 tasks, 63 sites) covering 15 life categories.
5‑layer trace recording	Captures browser video, DOM snapshots, network logs, agent actions, and final outcome.
LLM‑based evaluator	An agentic judge (DeepSeek‑v4‑pro by default) compares each run to human reference traces and produces a success score.
Leaderboard	HuggingFace Space shows scores per corpus, per harness, and per rubric.
Extensible	Add new tasks or models by supplying a JSON spec + rubric; contribution guide included.
Container‑isolated harness	Chrome + required dependencies run inside Docker/Podman, ensuring reproducibility across OSes.
Python CLI & library	Installable via PyPI (clawbench-eval) and usable as a library for custom pipelines.
Data download	All task definitions, rubrics, and raw traces are hosted on HuggingFace datasets.

---

When you’d use ClawBench

• Research – evaluate a new LLM‑agent, a tool‑use planner, or a retrieval‑augmented system on realistic web‑based workflows.
• Model comparison – run the same set of tasks on multiple agents and see a side‑by‑side leaderboard.
• Error analysis – the 5‑layer traces let you replay a run, inspect screenshots, DOM changes, and network traffic.
• Benchmark contribution – add a new everyday task (e.g., a new e‑commerce checkout flow) and share it with the community.

---

Resources

• Live leaderboard – https://huggingface.co/spaces/TIGER-Lab/ClawBench
• Paper (arXiv) – https://arxiv.org/abs/2604.08523
• Dataset – https://huggingface.co/datasets/NAIL-Group/ClawBench
• Task explorer – https://claw-bench.com/tasks
• GitHub issues / FAQ – for troubleshooting and community questions.

---

ClawBench is a genuine AI‑agent benchmark, not a simple tutorial list or a trivial demo. It provides a full evaluation pipeline, data, and a public leaderboard for frontier AI agents working on everyday online tasks.