UKGovernmentBEIS/inspect_evals

What it solves

It provides a standardized library of evaluations for frontier AI models, allowing developers and researchers to measure model capabilities and assess risks and mitigations in a consistent way.

How it works

Built on top of the Inspect AI framework, the library contains a collection of pre-defined evaluation tasks (such as ARC Easy and ARC Challenge). Users can run these evaluations via a command-line interface or by importing them as Python objects, specifying a model provider (e.g., OpenAI, Anthropic, Google) to test.

Who it’s for

AI researchers, safety engineers, and developers who need to benchmark frontier AI models against established datasets and risk assessments.

Highlights

• Broad Model Support: Compatible with numerous providers including OpenAI, Anthropic, Google, Mistral, Azure AI, AWS Bedrock, Together AI, Groq, Hugging Face, vLLM, and Ollama.
• Integrated Tooling: Includes a log viewer for analyzing evaluation results and a VS Code extension for easier log inspection.
• Flexible Execution: Supports running single evaluations, sets of multiple tasks simultaneously, or programmatic execution via Python.
• Extensible Register: Allows users to add their own evaluations to the Inspect Evals Register to share with the community.