harbor-framework/terminal-bench

What it solves

Terminal-Bench provides a standardized way to measure the capabilities of frontier AI agents. It addresses the need for a high-quality, diverse, and evolving set of difficult tasks to track progress and compare the performance of different agent builders.

How it works

It functions as a continuous benchmark where tasks are published as tagged releases. Users can run the benchmark using the Harbor framework, specifying an agent and a model to test against the dataset. The system supports sandboxing environments (developed using Modal) to ensure tasks are executed safely and consistently, and includes "oracle" solutions to verify that the environment is working correctly.

Who it’s for

This tool is designed for frontier agent builders and researchers who want to evaluate their AI agents' ability to handle complex, real-world terminal-based tasks.

Highlights

• Continuous Evolution: The benchmark is updated over time with new, challenging tasks to prevent stagnation.
• Diverse Task Set: Includes a wide range of difficult tasks across various domains.
• Community Driven: Actively encourages contributions of new tasks through a structured proposal and review process.
• Integration: Built on the Harbor framework and integrates with Modal for CI/CD and leaderboard experiments.