suyoumo/ClawProBench

What it solves

ClawProBench is a benchmark harness designed to evaluate the capabilities of AI agents operating within the OpenClaw runtime. It addresses the need for transparent, live-execution benchmarks that use deterministic grading to measure how models perform on real-world tasks rather than static datasets.

How it works

The system operates as a live-first benchmark harness that executes tasks across various scenarios. It uses a CLI (run.py) to manage the benchmark process, including inventory checks, dry runs, and full execution. The harness bridges to the OpenClaw runtime to run agents and then applies scenario-specific grading logic located in custom_checks to determine success. It supports multi-trial runs (e.g., 3-try runs) to calculate stability metrics like pass^3 and pass@3.

Who it’s for

This tool is for AI researchers and model developers who need to evaluate agentic capabilities and ensure their models can reliably execute tasks within the OpenClaw ecosystem.

Highlights

• Live-First Execution: Evaluates models inside the actual OpenClaw runtime rather than using simulated environments.
• Deterministic Grading: Uses structured reports and custom check logic to ensure consistent scoring.
• Comprehensive Profiles: Offers multiple benchmark profiles (core, intelligence, coverage, native, full) to balance between a quick ranking suite and extended capability testing.
• Robust Execution: Supports checkpoint resume, cross-environment resume, and the ability to re-queue execution failures.
• Detailed Metrics: Calculates a FinalScore that weights stable repeated success more heavily than one-off wins.