braintrustdata/autoevals

AutoEvals is a tool for quickly and easily evaluating AI model outputs using best practices.

What it solves

Autoevals provides a unified interface for evaluating AI model outputs, eliminating the need to manually implement common evaluation metrics. It simplifies the process of normalizing scores (scaling them between 0 and 1) and parsing model-graded outputs, which are often difficult to debug and tweak in practice.

How it works

The library bundles several categories of evaluation methods that take an input, an output, and an expected value to produce a score:

  • LLM-as-a-judge: Uses a model to grade subjective tasks like factuality, safety, and summarization.
  • RAG evaluations: Specific metrics for retrieval-augmented generation, such as context precision, recall, and faithfulness.
  • Heuristic and Statistical: Traditional metrics like Levenshtein distance, BLEU, and exact match.
  • Custom Scorers: Users can define their own LLM-based classifiers with custom prompts or write non-LLM scoring functions.

It supports multiple AI providers via OpenAI-compatible APIs or the Braintrust Gateway for caching and observability.

Who it’s for

Developers building AI applications who need to quantitatively measure the performance of their models and iterate on prompts or RAG pipelines.

Highlights

  • Multi-language support: Available in both Python and TypeScript.
  • Unified Interface: A single consistent API for applying various scorers across different evaluation methods.
  • Extensible: Easy to create custom model-graded evaluations or simple heuristic scorers.
  • Broad Metric Suite: Includes a wide range of pre-built scorers for RAG, general LLM tasks, and embeddings similarity.

Related

  • Project
    langchain-ai/openevals

    A Python and TypeScript library for creating evaluations for LLM applications, featuring LLM-as-judge capabilities and prebuilt prompts for quality, safety, and RAG.

  • Project
    confident-ai/deepeval

    An open-source LLM evaluation framework that provides unit-testing capabilities for AI agents, RAG pipelines, and chatbots using LLM-as-a-judge metrics.

  • Project
    agentscope-ai/OpenJudge

    An open-source evaluation framework for AI applications and agents that provides a library of ready-to-use graders and tools to generate custom evaluation rubrics.

  • Project
    modelscope/evalscope

    A one-stop LLM evaluation framework that provides model capability benchmarks, inference performance stress testing, and result visualization.

  • Project
    UKGovernmentBEIS/inspect_evals

    A community-contributed repository of LLM evaluation benchmarks for the Inspect AI framework, covering coding, scientific research, and agentic capabilities.