harveyai/harvey-labs

A benchmark built to evaluate and improve agent capabilities for supporting legal work.

What it solves

Harvey LAB addresses the difficulty of measuring how well AI agents perform in professional legal settings. It provides a standardized way to test whether an agent can handle complex, real-world legal tasks accurately.

How it works

The project consists of two main components: a comprehensive dataset of over 2,000 legal tasks—including instructions, source documents, and grading rubrics—and an execution harness. Developers use the lab-core Python package to run their agents against these tasks, which then evaluates the agent's output against the provided rubrics to generate performance reports and comparison dashboards.

Who it’s for

It is designed for researchers and developers building AI agents for the legal industry who need a rigorous, repeatable framework to benchmark their models' performance on realistic legal workflows.

Highlights

  • Includes over 2,000 legal tasks across 24 practice areas.
  • Provides an execution harness for running and scoring agent performance.
  • Supports detailed evaluation through rubrics and LLM-based judging.
  • Offers a Python library (lab-core) for integration into custom agent development workflows.

Related

  • Project
    scaleapi/SWE-bench_Pro-os

    A challenging benchmark for evaluating LLMs and AI agents on long-horizon software engineering tasks by requiring them to generate patches for real-world codebase issues.

  • Project
    Simreal-AI/Simreal-MLBench

    An open benchmark for evaluating AI research agents across 60 ML tasks, with transparent scoring rules but a commercially operated evaluation runtime. Agents get two scored submissions, and all results are externally verifiable via public scoring code and provenance records.

  • Project
    Ayanami0730/deep_research_bench

    A comprehensive benchmark for evaluating Deep Research Agents using 100 PhD-level tasks and two specialized frameworks for assessing report quality and citation accuracy.

  • Project
    suyoumo/ClawProBench

    A transparent live-first benchmark harness for evaluating AI agent capabilities and performance within the OpenClaw runtime using deterministic grading.

  • Project
    InternLM/WildClawBench

    WildClawBench is an end-to-end agent benchmark that tests AI agents' ability to perform complex, real-world tasks in live environments using real tools like browsers and bash terminals.