harbor-framework/terminal-bench-science

Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains

What it solves

Terminal-Bench-Science addresses the need for rigorous, expert-curated evaluation of AI agents on real scientific research workflows. It measures how well frontier AI agents can perform tasks drawn from life, physical, earth, mathematical, and engineering sciences, giving scientists a direct voice in shaping AI progress and providing a continuous benchmark that evolves alongside AI capabilities.

How it works

The benchmark contains 71 expert-authored tasks (growing toward 100+) that produce objectively verifiable outcomes in a terminal environment. Tasks go through a structured flow: experts propose ideas, implement them, and submit pull requests; automated checks (static validation, Docker builds, oracle/no-op runs, similarity detection, cheat trials, and multi-agent trials) plus parallel domain and technical reviews ensure quality. The system runs agents via the Harbor framework, letting users test specific agents and models against the tasks. A leaderboard, task dashboard, and tagged releases track performance and development.

Who it's for

This is for AI researchers, model developers, and scientists who want to understand or improve agent performance on scientific research tasks. It is also for domain experts across scientific fields who want to contribute their own realistic workflows to steer AI development. The contribution process explicitly invites researchers from life, physical, earth, mathematical, and engineering sciences.

Highlights

  • 71 expert-curated tasks across five major scientific domains, with a public leaderboard and task dashboard
  • Community-driven: experts propose, build, and review tasks, with a contribution call and Discord channel
  • Automated quality assurance including Docker builds, oracle/no-op validation, similarity checks, cheat trials, and multi-agent runs
  • Continuous benchmark with tagged releases on Harbor Hub and DOI for citation
  • Includes a canary string to prevent training-data contamination and detect leakage

Related

  • Project
    harbor-framework/terminal-bench

    Terminal-Bench is a continuous benchmark for measuring the capabilities of frontier AI agents using a diverse and evolving set of difficult tasks.

  • Project
    harbor-framework/harbor

    Harbor is a framework for evaluating and optimizing AI agents and language models by running them in parallelized, containerized environments across various cloud providers.

  • Project
    benchflow-ai/skillsbench

    A benchmark for evaluating how AI agents leverage and compose modular skills—instructions, scripts, and resources—to perform specialized workflows.

  • Project
    benchflow-ai/benchflow

    A universal environment framework that runs AI agents against task environments and scores them through a standardized contract, treating benchmarks as frozen environments.

  • Project
    FrontierCS/Frontier-CS

    Frontier-CS is an open-ended, verifiable benchmark for evaluating AI models on difficult, research-grade computer science problems that go beyond traditional, saturated coding tests.