harbor-framework/terminal-bench-science

Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains

What it solves

Terminal-Bench-Science provides a standardized way to measure how well AI agents can handle complex, real-world scientific research workflows. It addresses the gap between general AI capabilities and the specialized, expert-level tasks required in scientific domains like the life, physical, earth, mathematical, and engineering sciences.

How it works

The project is a continuous benchmark consisting of expert-curated tasks that are objectively verified in a terminal environment. It uses the Harbor framework to run agents against these tasks. The benchmark evolves through a community-driven pipeline where domain experts propose, build, and review tasks to ensure they remain challenging and relevant to frontier AI models.

Who it’s for

This benchmark is designed for AI researchers developing agents and for scientific domain experts who want to contribute tasks that reflect the actual work they need AI systems to support.

Highlights

  • Expert-Curated: Tasks are authored and reviewed by domain experts across five broad scientific domains.
  • Objectively Verifiable: All tasks produce outcomes that can be verified within a terminal environment.
  • Community-Driven: Features a transparent pipeline for task proposals, implementation, and review.
  • Contamination Protection: Uses canary strings to help training-data pipelines detect and exclude benchmark data from training corpora.

Related

  • Project
    harbor-framework/terminal-bench

    Terminal-Bench is a continuous benchmark for measuring the capabilities of frontier AI agents using a diverse and evolving set of difficult tasks.

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    AISBench/benchmark

    AISBench Benchmark is an open‑source evaluation framework for LLMs, multimodal models, and AI agents. It supports both accuracy (using many public datasets) and performance (latency, throughput, cache) testing of locally‑run or service‑based models, with extensive recent additions such as Harbor‑agent benchmarking, prefix‑cache stress tests, and response‑anomaly detection.

  • Project
    synthetic-sciences/openscience

    An open-source AI workbench for scientific research that automates the research loop, including literature review, coding, and running experiments.

  • Project
    zli12321/LHTB

    Long‑Horizon Terminal‑Bench (LHTB) is a 46‑task benchmark that measures how well LLM agents can keep working inside a Docker‑based terminal for hundreds of steps. It ships a patched version of the open‑source Harbor harness that adds a “continue‑until‑timeout” loop and sandboxed verifier isolation, preventing agents from cheating. Results show current frontier models solve only a minority of tasks, making LHTB a hard, frontier‑level evaluation suite for autonomous AI agents.