TIGER-AI-Lab/ClawBench
Open-source benchmark for browser AI agents on daily tasks.
What it solves
ClawBench is an open-source benchmark designed to evaluate how well AI browser agents can complete real-world, everyday online tasks. It addresses the gap in measuring end-to-end success for agents performing complex activities like booking travel, ordering food, applying for jobs, and managing email on live websites.
How it works
ClawBench operates by spinning up an isolated Docker or Podman container containing Chromium and a specific agent harness. An AI agent then drives the browser to navigate and interact with live websites. To ensure accurate scoring, the system uses a request interceptor that captures data across five layers (including video recordings, request logs, action logs, agent messages, and interception data), which is then compared against human references by an agentic evaluator.
Who it’s for
This tool is designed for developers and researchers building AI agents, specifically those focusing on browser automation and "computer use" capabilities, who need a standardized way to measure their agent's success rate on diverse, real-world tasks.
Highlights
- Extensive Task Library: Includes V1 (153 tasks) and V2 (130 tasks) covering 144 live websites across 15 life categories.
- Isolated Execution: Uses Docker/Podman containers to ensure a clean, isolated environment for browser interactions.
- Five-Layer Recording: Captures comprehensive execution traces (MP4 video, JSONL requests, action logs, etc.) for detailed analysis.
- Two-Stage Scoring: Employs a request interceptor and an inline LLM judge to automatically produce pass/fail results.