claw-eval/claw-eval

What it solves

Claw-Eval provides a trustworthy and reproducible framework for evaluating the general agentic capabilities of foundation models. It addresses the issue of "lucky runs" in agent evaluation by requiring consistent success across multiple trials, ensuring that a model's performance is not due to chance.

How it works

The project uses a dataset of 300 human-verified tasks across nine categories, including general productivity, multimodal perception, and multi-turn conversations. It employs a "Pass^3" methodology where a task is only marked as passed if the model succeeds in all three independent trials. Evaluation is conducted through full-trajectory auditing across three dimensions: completion, safety, and robustness.

Who it’s for

This tool is designed for AI researchers and developers building autonomous agents and foundation models who need a rigorous, scientific foundation for benchmarking their agentic capabilities.

Highlights

• Pass^3 Metric: Requires three consecutive successful runs to eliminate fluke results.
• Comprehensive Task Set: 300 tasks across general, multimodal, and multi-turn splits.
• Trajectory Auditing: Grades agents on completion, safety, and robustness.
• Multimodal Support: Includes tasks for webpage generation, video QA, and document extraction.
• Sandbox Integration: Supports sandbox isolation and full-trace tracking for transparent evaluation.