zli12321/LHTB

Long‑Horizon Terminal‑Bench (LHTB)

What it is – LHTB is a research benchmark that evaluates large‑language‑model (LLM) agents when they are dropped into a real, stateful terminal (a Docker‑style sandbox) and asked to keep working for hundreds of steps. Unlike typical coding tests that end once a program is produced, LHTB measures how well an agent can sustain useful, progressive work over long horizons, using hidden verifiers that check the final state rather than the agent’s own logs.

Key features

• 77 tasks in version 2.0, split into 46 original “interactive‑workflow” tasks and 31 new reconstruction tasks (e.g., reproducing the behavior of a scientific tool from its outputs). The tasks span games, reverse‑engineering, scientific computing, geospatial analysis, OCR, chemistry, control‑systems, etc.
• Containerised terminal – each task runs inside a sandboxed environment; the agent interacts via a shell‑like interface.
• Hidden, rebuild‑from‑artifact verifiers – after the agent declares it is done, a separate verifier rebuilds the expected artifact and checks it. The agent never sees the verifier’s internals, preventing cheating.
• Continue‑until‑timeout – for long‑horizon tasks the harness keeps the agent alive until the overall time budget expires, re‑running the verifier after each checkpoint and only allowing a binary “pass/fail” signal. This forces the agent to keep improving rather than stopping early.
• Modified Harbor harness – the repo ships a patched version of the open‑source Harbor evaluation harness that implements the continue‑until‑timeout logic and isolates the verifier to stop agents from reading hidden grader files.
• Extensive results – the README includes leaderboards for October 2026 (15 models, 3‑hour per‑task budget) and earlier snapshots, showing mean rewards, solve rates, and cost‑per‑task analyses.

Why it matters – LHTB pushes LLM agents beyond one‑shot code generation toward continuous problem‑solving in realistic, stateful environments. Success on this benchmark suggests an agent can manage long‑running workflows such as data pipelines, scientific simulations, or multi‑step software debugging, which are core use‑cases for autonomous AI assistants.

How to use it

1. Pick a task – each task lives under tasks/ or tasks-2.0/ and follows the standard Harbor layout (source files, task.toml, hidden verifier, etc.).
2. Configure Harbor – copy one of the example YAMLs in configs/examples/ (e.g., terminus2_openai.yaml) and point it at the desired task directory.
3. Run the harness – invoke the provided scripts (or your own runner) to launch the agent inside the sandbox. The harness will enforce the time budget, checkpoint the verifier every 30 min, and apply the continue‑until‑timeout policy.
4. Inspect results – after the run finishes, the harness writes a scorecard.json and logs showing the final reward (0 – 1) and whether the hidden verifier passed.

Resources linked in the repo

• Blog post with overview: https://zli12321.github.io/LHTB/
• Interactive data website (Hugging Face Space): https://huggingface.co/spaces/IntelligenceLab/Terminal-Data-Website
• Full dataset of task definitions and benchmark submissions: https://huggingface.co/datasets/IntelligenceLab/Long-Horizon-Terminal-Bench
• Leaderboard page: https://zli12321.github.io/LHTB/leaderboard.html
• The modified Harbor harness (harbor/) with patch details.

Citation – The benchmark is described in the arXiv paper Long‑Horizon Terminal‑Bench (arXiv:2607.08964). The README provides a BibTeX snippet for citation.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.