evidently: an open-source framework to evaluate, test, and monitor ML and LLM-powered systems

What it solves

Evidently provides a unified framework to evaluate, test, and monitor the quality of machine learning (ML) and Large Language Model (LLM) systems. It addresses the challenge of maintaining performance and reliability from the experimental phase through to production, specifically targeting issues like data drift, model degradation, and the quality of generative AI outputs.

How it works

The library operates through three primary components:

1. Reports: These compute and summarize quality evaluations using built-in metrics or custom ones. They are used for exploratory analysis and debugging and can be exported as JSON, HTML, or Python dictionaries.
2. Test Suites: By adding pass/fail conditions to Reports, users can create automated tests for regression testing, CI/CD checks, and data validation.
3. Monitoring Dashboard: A UI service (available via self-hosting or a managed cloud version) that visualizes these metrics and test results over time to track system health.

Who it’s for

It is designed for ML engineers, data scientists, and AI developers who need to ensure their predictive models (classification, regression) or generative systems (RAG, LLM applications) remain accurate and stable over time.

Highlights

• Broad Support: Works with both tabular and text data.
• Extensive Metric Library: Includes over 100 built-in metrics covering data drift, LLM-as-a-judge, and traditional ML performance.
• Versatile Evaluation: Supports predictive tasks (accuracy, precision) and generative tasks (semantic similarity, retrieval relevance).
• Flexible Deployment: Offers both offline evaluations for experiments and live monitoring for production systems.