Stanford CS329A Lecture on Agentic Evaluations and Long Horizon Tasks

METR Time Horizon Evaluation

The METR benchmark measures how long a model can reliably complete a task by reporting the time horizon at 50% and 80% success rates, calibrated against human professionals.

  • For short tasks (1–30 seconds) success rates are high; for HCAST tasks (1 minute–30 hours) success varies widely; for RE‑Bench tasks (up to eight hours) success rates are low.
  • The 50% horizon has grown from about two seconds for GPT‑2 (2019) to roughly 59 minutes for Claude 3.7 Sonnet (2025), a doubling approximately every seven months.
  • At 80% reliability the horizon is much shorter: Claude 3.7 Sonnet achieves about 15 minutes, showing a large gap between 50% and 80% performance.
  • Improvements are driven by better logical reasoning, code generation, tool use, error recovery, and context engineering, but reliability remains a limiting factor.

GDPVal Economic Value Benchmark

GDPVal evaluates whether model output is good enough compared to work by industry professionals with over a decade of experience, reporting win rates across 44 occupations and nine sectors.

  • Win rates rose from 12.4% for GPT‑4o to 47.6% for Claude Opus 4.1, indicating a roughly linear improvement over the last two years.
  • Tasks are well‑specified, often multimodal, and have an average value of about $400 per task in the gold subset.
  • Models perform better on short, well‑defined tasks and decline on longer, ambiguous ones; they struggle when they must infer what to work on without explicit context.
  • Human oversight that provides context and task framing makes AI assistance cost‑effective in many professions.

DeepScholar‑Bench Research Synthesis

DeepScholar‑Bench tests whether models can generate a literature‑review section for academic papers, scoring knowledge synthesis, retrieval quality, and citation verifiability.

  • Current systems (OpenAI deep research, Gemini deep research, etc.) score below 19% overall on this live benchmark.
  • Models often produce coherent English but miss key facts; retrieval quality is moderate, with document importance scores under 12.5%.
  • Verifiability can be high (e.g., DeepScholar base ~90% precision) but synthesis and verifiability rarely excel together.
  • Failure modes stem from incomplete source retrieval, inability to assess document importance, inefficient extraction of key facts, and trade‑offs between synthesis quality and citation accuracy.

Failure Modes and Challenges

Across the benchmarks, recurrent agent failure modes include:

  • Poor planning: inability to break a task into appropriate steps.
  • Incorrect tool selection: choosing tools that do not advance the task.
  • Premature abandonment: stopping before success due to repetitive behavior or confusion.
  • Repetitive action loops: repeatedly performing the same high‑probability action after failure.
  • Additional challenges: models need extensive context to know what to work on, struggle with ambiguous prompts, and show limited generalization beyond software and knowledge‑work domains.

Key Takeaways

  • Capability (time horizon) is improving exponentially, but reliability (80% horizon) lags significantly.
  • Economic value benchmarks show linear gains and highlight the importance of explicit context and human‑in‑the‑loop framing.
  • Research‑synthesis tasks reveal that models still cannot match expert‑level knowledge retrieval, key‑fact extraction, and citation verification simultaneously.
  • Progress in agentic AI therefore requires better planning, tool use, error recovery, and methods to supply or infer the necessary context for long‑horizon goals.

Sources