harbor-framework/harbor

What it solves

Harbor provides a standardized way to evaluate and optimize AI agents and language models. It removes the friction of setting up complex environments for benchmarks, allowing developers to test arbitrary agents (such as Claude Code or OpenHands) against various datasets in isolated, scalable environments.

How it works

Harbor acts as an execution harness that connects agents and models to specific benchmark datasets. It can launch these environments locally using Docker or scale them across cloud providers like Daytona, Modal, and LangSmith to run thousands of experiments in parallel. It also supports generating rollouts for reinforcement learning (RL) optimization.

Who it’s for

It is designed for researchers and developers building AI agents or LLMs who need to measure performance, build custom benchmarks, or generate data for RL optimization.

Highlights

• Broad Agent Support: Evaluates a wide range of agents including Claude Code, OpenHands, and Codex CLI.
• Massive Parallelism: Supports running experiments in thousands of environments simultaneously via various cloud providers.
• Multimodal Benchmarking: Official harness for Terminal-Bench-2.0 and supports other datasets like SWE-Bench and Aider Polyglot.
• RL Ready: Capable of generating rollouts for RL optimization.