truera/trulens

Evaluation and Tracking for LLM Experiments and AI Agents

What it solves

TruLens helps developers identify where AI agents fail and find opportunities to reduce costs without sacrificing quality. It moves evaluation from "vibes" to data by providing traceable causes for bad answers and visible tradeoffs between different app versions.

How it works

TruLens uses OpenTelemetry-native instrumentation to record every step of an application—including function calls, LLM generations, and tool invocations—as structured spans. It then applies "LLM judges" (feedback functions) to score these steps based on specific metrics. These evaluations can be run inline as the app operates or in batch mode over a pre-collected dataset.

Who it’s for

AI engineers and teams building RAG (Retrieval-Augmented Generation) systems and agentic applications who need to monitor performance, debug failures, and optimize costs.

Highlights

  • OpenTelemetry-native: Traces are portable to any OTLP backend like Jaeger, Grafana Tempo, or Datadog.
  • Agentic Evaluators: Seven specialized evaluators for measuring reasoning coherence, execution efficiency, plan adherence, and tool usage.
  • RAG Triad: Built-in support for core RAG metrics including groundedness and context relevance.
  • Flexible Evaluation: Supports both real-time inline scoring and offline batch evaluation via a Run API.
  • Broad Integration: Compatible with major LLM providers (OpenAI, Gemini, Bedrock, etc.) and frameworks like LangChain and LlamaIndex.

Related

  • Project
    johnmarktaylor91/torchlens

    TorchLens is a Python library for capturing, visualising, and intervening on every activation and gradient of a PyTorch model. It builds a reproducible computational graph, offers rich per‑operation metadata, supports selective saving, backward‑pass capture, receptive/projective field analysis, and provides an API for on‑the‑fly interventions. Works with static, dynamic, and recurrent architectures and integrates with HuggingFace models.

  • Project
    traceloop/openllmetry

    An open-source observability framework built on OpenTelemetry that provides complete tracing and monitoring for LLM applications, providers, and vector databases.

  • Project
    JudgmentLabs/judgeval

    Judgeval is an open-source Python SDK that adds continuous-improvement observability to LLM agents: it traces calls via OpenTelemetry, evaluates traces with prompt-based judges, and lets you validate fixes against production data before shipping.

  • Project
    openlit/openlit

    An open-source agent harness engineering platform that uses OpenTelemetry to trace, evaluate, and guard AI agents and coding agents.

  • Project
    confident-ai/deepeval

    DeepEval is an open‑source Python framework that lets you write pytest‑style tests for any LLM‑based app—chatbots, agents, RAG pipelines, or multimodal generators. It ships with dozens of research‑backed metrics (e.g., G‑Eval, JevEval, task‑completion, hallucination, bias) that can run on any model locally, and you can add custom metrics. The library integrates with popular toolkits (LangChain, OpenAI, Anthropic, LlamaIndex, etc.) to automatically trace model calls, enabling both end‑to‑end and step‑level evaluation. Tests run from the CLI, fit into CI/CD, and can be linked to the commercial Confident AI platform for production‑time monitoring. Ideal for developers and teams who need quantitative quality gates for LLM applications.