confident-ai/deepeval

The LLM Evaluation Framework

What DeepEval Is

DeepEval is an open‑source Python framework for testing and benchmarking large‑language‑model (LLM) applications. Think of it as a pytest for LLM‑powered agents, chatbots, RAG pipelines, or any system that calls a language model. It lets you write test cases, run them against your code, and automatically score the results with a growing library of evaluation metrics.

Core Features (as described in the README)

Feature What It Means for You
Ready‑made metrics Over 30 built‑in metrics cover classic concerns – correctness, hallucination, bias, toxicity – as well as agent‑specific checks (task completion, tool correctness, plan adherence) and RAG‑specific checks (answer relevance, faithfulness, contextual recall).
LLM‑as‑a‑judge Metrics such as G‑Eval, JevEval, and DAG use a language model (any model you choose) to act as a judge, giving human‑like scoring without needing a separate dataset.
Local execution All metrics can run with locally‑hosted models or statistical methods, so you don’t have to send data to a third‑party service unless you want to.
Component‑level and end‑to‑end testing Evaluate a whole chatbot as a black box, or drill down to individual steps (LLM calls, tool use, retrieval, sub‑agent hand‑offs).
Custom metric creation You can plug in your own scoring function and DeepEval will treat it like any other metric.
Synthetic dataset generation The framework can generate single‑turn or multi‑turn synthetic test sets to stress‑test your system.
CI/CD ready A simple CLI (deepeval test run …) integrates with any CI pipeline; results can be exported as reports.
Prompt optimisation Built‑in helpers can automatically suggest prompt tweaks based on metric outcomes.
Benchmarking in <10 lines One‑liner helpers let you benchmark any LLM on standard suites (MMLU, HellaSwag, BIG‑Bench Hard, etc.).

How You Use It (quick‑start flow)

  1. Install – pip install -U deepeval (Python ≥ 3.9).
  2. Create an account (optional) – deepeval login stores an API key for cloud reporting.
  3. Write a test file – Define a LLMTestCase with input, actual_output, expected_output, and optionally retrieval_context.
  4. Pick a metric – e.g., GEval for correctness, TaskCompletionMetric for agents, AnswerRelevancy for RAG.
  5. Run – deepeval test run test_chatbot.py executes the test and prints a pass/fail based on the metric’s threshold.
  6. Trace – Decorate your functions with @observe() (or use the provided OpenAI/AI‑SDK wrappers) to capture the full decision trajectory; then evaluate with trajectory‑aware metrics.

Ecosystem Integrations

DeepEval ships adapters for the most popular LLM toolkits, so you can add evaluation with almost no code changes:

  • LangChain / LangGraph – callback handlers capture each chain step.
  • CrewAI, Pydantic AI, Mastra – native tracing for multi‑agent workflows.
  • Anthropic, OpenAI, AWS AgentCore, Google ADK – wrappers let you evaluate Claude, GPT‑4, Amazon agents, etc.
  • LlamaIndex – evaluate retrieval‑augmented generation pipelines.
  • AI SDK (TypeScript) – client‑side tracing for JavaScript/TS projects.

For enterprises, the commercial Confident AI platform adds a UI, dataset management, and production‑time observability while re‑using the same DeepEval metric definitions.

Who Might Benefit

  • LLM developers building agents, chatbots, or RAG services who need systematic regression testing.
  • ML engineers wanting to benchmark model upgrades or prompt changes automatically.
  • Product teams that must enforce quality standards (bias, toxicity, hallucination) before release.
  • Researchers who need a reproducible way to apply LLM‑as‑a‑judge metrics to new tasks.

Where to Learn More

Bottom line: DeepEval is a purpose‑built testing framework for LLM‑centric software, offering a rich set of ready‑made and extensible metrics, tight integration with the major LLM toolkits, and optional enterprise‑grade observability via Confident AI. It lets you treat LLM applications with the same rigor you’d apply to traditional code bases.

Related

  • Project
    braintrustdata/autoevals

    A library for evaluating AI model outputs using LLM-as-a-judge, RAG-specific metrics, and heuristic methods through a unified interface.

  • Project
    confident-ai/deepteam

    An open-source red teaming framework for LLM systems that simulates adversarial attacks to uncover vulnerabilities and provides guardrails for production security.

  • Project
    huggingface/lighteval

    A comprehensive toolkit from Hugging Face for evaluating LLMs across multiple backends with support for over 1,000 benchmarks and custom metrics.

  • Project
    langchain-ai/openevals

    A Python and TypeScript library for creating evaluations for LLM applications, featuring LLM-as-judge capabilities and prebuilt prompts for quality, safety, and RAG.

  • Project
    canwhite/AgentEval

    A transparent HTTP proxy that captures Agent-LLM traffic to provide automated grading, behavioral diagnosis, and source-code probing for AI agent evaluation.