relari-ai/continuous-eval

Data-Driven Evaluation for LLM-Powered Applications

continuous‑eval – Data‑driven evaluation for LLM‑powered apps

What it is – continuous-eval is a Python library (available on PyPI) that lets you measure the performance of each component in a generative‑AI pipeline. It ships with a catalog of ready‑made metrics for common LLM use‑cases (retrieval, RAG, code generation, tool‑using agents, classification, etc.) and a small framework for wiring those metrics to a dataset and running automated tests.

Why it matters – Building a production‑grade LLM application usually involves several stages (retriever → reranker → generator, classifier, etc.). Traditional evaluation tools focus on a single end‑to‑end score, making it hard to spot which module is under‑performing. continuous-eval treats the pipeline as a graph of modules, lets you attach bespoke metrics to each node, and even supports probabilistic or LLM‑as‑a‑judge metrics.

Core concepts

Concept Description
Metric library Ready‑to‑use classes such as PrecisionRecallF1, RankedRetrievalMetrics, AnswerCorrectness. They cover deterministic, semantic, and LLM‑based scoring.
EvaluationRunner Orchestrates the execution of all metrics on a dataset, aggregates results, and can run user‑defined tests (e.g., “recall must be ≥ 0.8”).
Pipeline / Module You declare each step of your system as a Module (name, input, output type) and attach the metrics that should be applied to its output. A Pipeline is just a list of modules plus the source Dataset.
CustomMetric A helper for building “LLM‑as‑a‑judge” metrics. You supply a prompt (criteria + rubric) and a schema for the expected response, then call it like any other metric.
Telemetry Optional anonymous usage tracking that can be disabled with CONTINUOUS_EVAL_DO_NOT_TRACK=true.

Typical workflow (illustrated in the README)

  1. Install pip install continuous-eval (or clone the repo and use Poetry).
  2. Prepare data – either download an example with example_data_downloader or supply your own JSON‑lines where each record contains fields like question, retrieved_context, ground_truth_context, answer, etc.
  3. Define a pipeline – create Module objects for each stage and attach the appropriate metric objects via .use(...).
  4. Run evaluation – instantiate EvaluationRunner(pipeline) and call evaluate(). The runner returns a PipelineResults object that can be aggregated and printed.
  5. Run tests – supply a list of Test objects (e.g., GreaterOrEqualThan) to automatically flag regressions.
  6. Extend – write a subclass of CustomMetric or implement a new metric class if the built‑in catalog doesn’t cover your need.

Example snippet (single metric)

from continuous_eval.metrics.retrieval import PrecisionRecallF1

datum = {
    "question": "What is the capital of France?",
    "retrieved_context": [
        "Paris is the capital of France and its largest city.",
        "Lyon is a major city in France."
    ],
    "ground_truth_context": ["Paris is the capital of France."],
    "answer": "Paris",
    "ground_truths": ["Paris"],
}

metric = PrecisionRecallF1()
print(metric(**datum))   # → {'context_precision': ..., 'context_recall': ..., 'context_f1': ...}

When to use it

  • RAG system developers who need separate scores for retrieval quality, reranking effectiveness, and generation correctness.
  • LLM product teams that want automated regression checks (tests) before shipping a new model or prompt.
  • Researchers evaluating multiple LLM‑based modules and requiring a reproducible, modular benchmark suite.
  • Companies that want to plug in custom LLM‑as‑a‑judge metrics (e.g., PII detection, factuality) without writing a full evaluation harness.

Getting started quickly

# Install the library
python3 -m pip install continuous-eval

# Clone the repo for examples and docs
git clone https://github.com/relari-ai/continuous-eval.git && cd continuous-eval
poetry install --all-extras   # optional, pulls in extra deps for all metrics

# Run the built‑in example (retrieval dataset)
python -m examples.run_retrieval_eval   # (see the examples repo linked in the README)

Make sure you have at least one LLM API key in a .env file if you plan to use LLM‑based metrics.

Extensibility & community

  • Custom metrics – use the CustomMetric class or subclass BaseMetric to add any scoring logic, including calls to your own LLM endpoint.
  • Open source – Apache 2.0 license, contribution guide provided, and a Discord community for support.
  • Documentation – Full API reference and tutorials at https://continuous-eval.docs.relari.ai/.

License

Apache 2.0 (per the repository’s LICENSE file).

Related

  • Project
    alexzhang13/rlm

    An inference engine and training environment for Recursive Language Models (RLMs) that enable LLMs to handle near-infinite context by programmatically decomposing tasks via recursive sub-calls in a REPL environment.

  • Project
    EvolvingLMMs-Lab/lmms-eval

    lmms‑eval is a Python toolkit that provides a single, reproducible pipeline for benchmarking multimodal LLMs (vision, video, audio). It ships with 100+ tasks, 30+ model wrappers, statistical reporting, async serving, an HTTP evaluation server, and an optional web UI, all installable via `uv` or pip.

  • Project
    nvk/llm-wiki

    llm‑wiki is a cross‑runtime tool that lets Claude, Codex, OpenCode, Pi and other LLM agents build and query a structured, Obsidian‑compatible wiki. It provides ingestion, parallel research, session memory, private adapters, and exportable knowledge checkpoints, with install scripts for each supported agent.

  • Project
    confident-ai/deepeval

    An open-source LLM evaluation framework that provides unit-testing capabilities for AI agents, RAG pipelines, and chatbots using LLM-as-a-judge metrics.

  • Project
    llm-as-a-verifier/llm-as-a-verifier

    llm-as-a-verifier is a Python framework that turns LLMs into fine‑grained, probabilistic verifiers for AI agent outputs. It scores candidate trajectories, selects the best‑of‑N via an efficient Probabilistic Pivot Tournament, tracks progress step‑by‑step, and supports multimodal (image) inputs. The library achieves state‑of‑the‑art results on several agentic benchmarks and includes scripts for reproducible evaluation, a cache‑optimised token usage tracker, and a Claude Code plugin (TurboAgent) for seamless integration.