TIGER-AI-Lab/ClawBench

Open-source benchmark for browser AI agents on daily tasks.

ClawBench – Benchmarking AI Agents on Real‑World Web Tasks

What it is – ClawBench is an open‑source benchmark that measures how well AI‑driven browser agents can complete everyday online activities (booking travel, ordering food, applying for jobs, managing email, etc.) on live websites. It ships two corpora of tasks (V1 = 152 tasks on 143 sites, V2 = 129 tasks on 63 sites) for a total of 281 tasks across 163 real‑world sites.

Why it matters – Most research on LLM agents evaluates them on synthetic or static environments. ClawBench pushes agents into the wild, requiring them to navigate real web pages, handle CAPTCHAs, fill forms, and interact with dynamic content. The current best agents succeed on only about one‑third of the tasks, highlighting a major gap between research prototypes and practical usefulness.

Key components

  • Task specifications – JSON files describing the goal, step‑by‑step rubric, and expected outcome for each task.
  • 5‑layer recording pipeline – Captures raw browser traces, video, audio, DOM snapshots, and a high‑level rubric‑based evaluation.
  • Agentic evaluator – An LLM judge (default: deepseek‑v4‑pro) compares the agent’s trace to human reference traces and produces a success score.
  • Leaderboard – Hosted on Hugging Face Spaces, showing per‑model scores for V1 and V2.
  • Data – All task definitions, rubrics, and reference traces are available as Hugging Face datasets (NAIL‑Group/ClawBench and TIGER‑Lab/ClawBenchV2Trace).

How to use it

  1. Install – pip install clawbench-eval (or via uv/pipx).
  2. Configure – Edit models/models.yaml to point to your agent implementation and add an API key for the judge model (deepseek‑v4‑pro).
  3. Run – The clawbench command launches an interactive TUI for selecting a model and a task, or you can script runs with clawbench-run / clawbench-batch.
  4. Containerized harness – The first run builds a Docker/Podman image containing Chromium, ffmpeg, noVNC, and any agent‑specific dependencies. Subsequent runs reuse the cached image for fast execution.
  5. Analyze – After a run, clawbench-analyze produces a report with success/failure, rubric scores, and video playback of the browser session.

Extending the benchmark – Contributions are encouraged. To add a new task, create a JSON spec and a rubric, then submit a PR. To add a new model, provide a driver script that conforms to the clawbench CLI interface.

Resources

Bottom line – ClawBench provides a concrete, reproducible way to evaluate the real‑world usefulness of browser‑based AI agents, offering both a rich set of live web tasks and a standardized scoring pipeline.

Related

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    suyoumo/ClawProBench

    A transparent live-first benchmark harness for evaluating AI agent capabilities and performance within the OpenClaw runtime using deterministic grading.

  • Project
    Accio-org/CommerceAgentBench

    Commerce Agent Bench is an open‑source benchmark for evaluating LLM‑based agents on 107 realistic, stateful commerce tasks (CLI, browser, API, file). It provides Docker‑isolated mock services, a reproducible OpenClaw harness, multiple provider routes, and a public leaderboard. The repo includes installation instructions, example runs, and a reproducibility contract, but does not ship any agents itself.

  • Project
    ByteDance-Seed/EdgeBench

    EdgeBench is a benchmark of 134 real-world tasks designed to evaluate how autonomous AI agents learn and improve through iterative interaction with an environment over long horizons.

  • Project
    harbor-framework/terminal-bench-science

    A benchmark for evaluating AI agents on expert-curated research workflows across various scientific domains, verified in a terminal environment.