zli12321/LHTB

Long‑Horizon Terminal‑Bench (LHTB)

What it is – LHTB is a research benchmark that evaluates large‑language‑model (LLM) agents on long‑running terminal‑based tasks. It contains 46 distinct tasks that require an agent to keep working inside a Docker‑container for hundreds of steps, rather than producing a single output and stopping.

Why it matters – Most existing coding or reasoning benchmarks are short‑horizon: the model writes one script, the test runs, and the job is done. LHTB pushes agents to maintain state, handle interactive programs, and recover from failures over a 90‑minute (or longer) budget. The results show that even the strongest current models solve only a fraction of the tasks, highlighting a clear gap in long‑term autonomous capability.

Key components

Task set – 46 “Harbor” task definitions covering games, multimodal analysis, reverse‑engineering, scientific simulation, earth‑energy systems, security, research reproduction, and professional APEX‑style workflows.
Modified Harbor harness – A fork of the open‑source Harbor evaluation framework that adds two crucial features:
1. continue_until_timeout – agents keep running until the per‑task timeout, with the harness re‑invoking the hidden verifier after each attempt.
2. Verifier isolation – the verifier runs in a separate sandbox and the agent’s process tree is frozen during verification, preventing the agent from peeking at hidden test data.
Leaderboard & dataset – Results for 21 frontier models (July 2026 snapshot) are published on a live leaderboard and the full per‑run data are available as a Hugging Face dataset.
Example configs – Ready‑to‑run YAML files for a no‑API‑key “oracle” run, OpenAI‑compatible APIs, OpenRouter, and a full‑benchmark run.

How to use it

Install the harness – either the stock harbor package (single‑shot only) or the LHTB‑patched version shipped in the repo:
```
pip install -e harbor   # editable install with continue‑until‑timeout
```

Clone the repo (Git LFS for large assets):

git lfs install
git clone https://github.com/zli12321/LHTB.git
cd LHTB
git lfs pull

Run a quick sanity check (no API key needed):

harbor run -c configs/examples/oracle_smoke.yaml

Run your own agent – set your API key in the environment and point to a config, e.g.:

export OPENAI_API_KEY=sk-...
harbor run -c configs/examples/terminus2_openai.yaml   # runs a subset

Run the full benchmark:

export OPENAI_API_KEY=sk-...
harbor run -c configs/examples/full_benchmark.yaml

Results are written under ./jobs/.

Results snapshot (July 2026) – The strongest model (Grok 4.5) achieved a mean reward of 0.505 and solved 13/46 tasks. Even with a 3‑hour budget, the best‑performing model (GPT‑5.6‑sol) only reached a mean reward of 0.600 and solved 17/46 tasks, confirming that the benchmark remains far from saturated.

Where to learn more

Blog post: https://zli12321.github.io/LHTB/
Paper (arXiv): https://arxiv.org/abs/2607.08964
Live leaderboard: https://zli12321.github.io/LHTB/leaderboard.html
Dataset: https://huggingface.co/datasets/IntelligenceLab/Long-Horizon-Terminal-Bench

License – Apache 2.0.

LHTB is a genuine, research‑grade benchmark for long‑horizon LLM agents, not a simple tutorial or collection of links.

zli12321/LHTB

Long‑Horizon Terminal‑Bench (LHTB)

Related