harveyai/harvey-labs

A benchmark built to evaluate and improve agent capabilities for supporting legal work.

What it solves

It provides a standardized way to evaluate how well LLM-based agents can handle real-world legal work. Because legal tasks are complex and require high precision, this benchmark helps developers measure agent performance across various legal practice areas.

How it works

The project consists of two main components: a dataset of tasks (which include specific instructions, relevant documents, and scoring rubrics) and an execution harness. This harness allows developers to run their agents against the tasks and automatically evaluate their performance based on thedefined rubrics.

Who it’s for

It is designed for AI researchers and developers building LLM agents specifically for the legal domain.

Highlights

  • Comprehensive coverage with over 1,600 tasks across 24+ legal practice areas.
  • Includes a dedicated execution harness for running and scoring agents.
  • Uses an all-pass rubric scoring system and LLM judges for evaluation.
  • Open-source framework for comparing agent performance via dashboards.

Related

  • Project
    InternLM/WildClawBench

    An end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using tools like browsers and bash terminals.

  • Project
    lopopolo/harness-engineering

    A framework for improving AI agent output by shaping the environment, context, and tools around them rather than modifying the model itself.

  • Project
    TIGER-AI-Lab/ClawBench

    ClawBench is an open‑source benchmark that evaluates AI browser agents on 281 real‑world web tasks (e.g., booking travel, ordering food). It provides task specs, a multi‑layer trace recorder, and an LLM judge (deepseek‑v4‑pro) to score success. Install via `pip install clawbench-eval`, configure your model and judge API key, then run tasks in a Docker/Podman‑based Chromium harness. Results are posted to a public leaderboard and the full data are available on Hugging Face.

  • Project
    harbor-framework/harbor

    Harbor is a Python framework for running and scaling benchmarks of AI agents and language models. Installable via `pip`/`uv`, it provides a CLI (`harbor run`) that lets you specify a dataset (e.g., Terminal‑Bench‑2.0), an agent (Claude‑Code, OpenHands, etc.), and a model, then executes the benchmark locally or on cloud providers (Daytona, Modal, etc.) with configurable parallelism. It also supports creating custom benchmarks and generating RL roll‑outs.

  • Project
    zli12321/LHTB

    Long‑Horizon Terminal‑Bench (LHTB) is a 46‑task benchmark that measures how well LLM agents can keep working inside a Docker‑based terminal for hundreds of steps. It ships a patched version of the open‑source Harbor harness that adds a “continue‑until‑timeout” loop and sandboxed verifier isolation, preventing agents from cheating. Results show current frontier models solve only a minority of tasks, making LHTB a hard, frontier‑level evaluation suite for autonomous AI agents.