langchain-ai/openevals

What it solves

OpenEvals provides a standardized starting point for writing evaluations (evals) for LLM applications. It helps developers move beyond manual testing by providing tools to programmatically judge the quality, correctness, and safety of model outputs before they go into production.

How it works

The library primarily uses an "LLM-as-judge" approach, where a highly capable model is used to evaluate the outputs of another model based on a specific rubric. It provides a create_llm_as_judge function that handles the formatting of inputs and outputs into a prompt and parses the judge's response into a structured score and comment. It also supports non-LLM evaluators such as exact match, Levenshtein distance, and embedding similarity.

Who it’s for

It is designed for developers building LLM-powered applications in Python or TypeScript who need a reliable way to measure performance and regression during development.

Highlights

• Prebuilt Prompts: Includes ready-to-use prompts for quality, safety, security, and RAG (correctness, helpfulness, groundedness, and retrieval relevance).
• Flexible Judging: Supports binary scores, continuous float scores, or a specific set of custom score choices.
• Broad Evaluation Scope: Beyond text, it includes evaluators for structured output (exact match), code (type-checking via Pyright/Mypy), and agent trajectories.
• Multimodal Support: Capable of evaluating outputs that include attachments or images.
• Simulation Tools: Includes features for simulating users to test multi-turn interactions.