UKGovernmentBEIS/inspect_evals

What it solves

Inspect Evals provides a centralized, community-contributed repository of evaluation benchmarks for Large Language Models (LLMs). It solves the problem of fragmented evaluation tools by offering a standardized way to run diverse tests—ranging from complex coding and scientific research to web-browsing and tool-calling—using the Inspect AI framework.

How it works

The project acts as a library of evaluation tasks that can be executed via the inspect command-line tool or imported as Python objects. Users can install the package via PyPI or clone the repository and use uv for dependency management. The framework supports a wide array of model providers (including OpenAI, Anthropic, and Google) and allows users to run single evaluations or sets of evaluations simultaneously, with logs viewable through a dedicated viewer or VS Code extension.

Who it’s for

It is designed for AI researchers, developers, and safety institutes who need to rigorously benchmark LLM capabilities, specifically in areas like software engineering, scientific computation, and agentic behavior.

Highlights

• Diverse Benchmark Suite: Includes a vast array of evals such as SWE-bench for real-world GitHub issues, HumanEval for Python functions, and AgentBench for agentic capabilities.
• Specialized Domains: Features benchmarks for high-stakes technical fields, including CUDA code generation (ComputeEval), single-cell RNA-seq analysis (scBench), and scientific research replication (PaperBench).
• Extensible Architecture: Provides a template for developers to build and share their own evaluations, which are then registered in a community-driven system.
• Broad Provider Support: Compatible with most major LLM providers and local inference engines like vLLM and Ollama.