suyoumo/ClawProBench

ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.

What it solves

ClawProBench is a benchmark harness designed to evaluate the capabilities of AI agents operating within the OpenClaw runtime. It addresses the need for transparent, live-execution evaluation with deterministic grading, moving away from static evaluations to test how agents actually perform in a real runtime environment.

How it works

The system uses a set of benchmark profiles (such as core, intelligence, and native) to run AI agents through various scenarios. It supports multi-trial runs (e.g., 3-try runs) to measure stability and success rates. The harness manages the execution of agents via the OpenClaw CLI, tracks traces, and applies deterministic grading logic—including custom checkers—to determine if a task was completed successfully.

Who it’s for

It is intended for AI researchers, model developers, and developers working with agentic frameworks who need to rigorously test and rank the performance of their models in live-runtime agent tasks.

Highlights

  • Live-First Execution: Evaluates models directly inside the OpenClaw runtime rather than using static datasets.
  • Comprehensive Scenario Set: Includes 102 active scenarios across 6 domains, with additional incubating scenarios.
  • Deterministic Grading: Uses structured reports and custom checkers to ensure consistent and reproducible results.
  • Advanced Metrics: Implements a FinalScore that weights stable repeated success (pass^3) and overall quality.
  • Resilient Execution: Supports checkpoint resume and re-running only execution failures to optimize evaluation costs.

Related

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    TIGER-AI-Lab/ClawBench

    ClawBench is an open‑source benchmark that evaluates AI browser agents on 281 real‑world web tasks (e.g., booking travel, ordering food). It provides task specs, a multi‑layer trace recorder, and an LLM judge (deepseek‑v4‑pro) to score success. Install via `pip install clawbench-eval`, configure your model and judge API key, then run tasks in a Docker/Podman‑based Chromium harness. Results are posted to a public leaderboard and the full data are available on Hugging Face.

  • Project
    nikilster/clawflows

    A powerful workflow system for OpenClaw that provides over 100 prebuilt, deterministic automation routines to turn an AI agent into a proactive personal assistant.

  • Project
    Accio-org/CommerceAgentBench

    Commerce Agent Bench is an open‑source benchmark for evaluating LLM‑based agents on 107 realistic, stateful commerce tasks (CLI, browser, API, file). It provides Docker‑isolated mock services, a reproducible OpenClaw harness, multiple provider routes, and a public leaderboard. The repo includes installation instructions, example runs, and a reproducibility contract, but does not ship any agents itself.

  • Project
    harbor-framework/terminal-bench

    Terminal-Bench is a continuous benchmark for measuring the capabilities of frontier AI agents using a diverse and evolving set of difficult tasks.