benchflow-ai/benchflow

What it solves

BenchFlow provides a universal framework for running and scoring AI agents against task environments. It treats benchmarks as "frozen environments," allowing developers to test any ACP agent across various benchmarks, single-agent or multi-agent patterns, using a standardized contract for results.

How it works

BenchFlow uses a three-layer model consisting of Rollouts, Scenes, Roles, and Verifiers. It can pull benchmark datasets from external Git repositories or connect to hosted environments. The framework supports various sandboxes (such as Docker, Daytona, and Modal) to isolate agent execution. It can also capture and upload trajectories (execution logs) from local coding agent sessions for evaluation purposes.

Who it’s for

• Eval researchers and paper writers who need to run standardized evaluations.
• Task authors creating new benchmark tasks and environments.
• Agent builders who want to integrate their agents into the BenchFlow ecosystem.
• Benchmark adapters looking to onboard third-party benchmarks into a unified format.

Highlights

• Universal Compatibility: Works with various agents (e.g., Claude Code, Codex, Gemini CLI) and models.
• Flexible Execution: Supports single-agent, multi-agent (e.g., coder + reviewer), and multi-round patterns with progressive disclosure.
• Sandbox Hardening: Integrates with Docker, Daytona, and Modal for secure agent execution.
• Trajectory Capture: Includes tools to redact secrets and upload local agent session trajectories for evaluation.
• Task Authoring Tools: Provides a CLI for scaffolding, validating, and migrating benchmark tasks.