truera/trulens

What it solves

TruLens solves the problem of "vibe-based" evaluation of AI agents and RAG applications. It provides a structured way to find where an agent fails, identify cost-saving opportunities without sacrificing quality, and replace guesswork with traceable, measurable metrics.

How it works

TruLens uses OpenTelemetry-native tracing to capture every step of an AI application—including function calls, LLM generations, and tool invocations—as structured spans. It then applies "LLM judges" (feedback functions) to score these traces. These evaluations can be run inline as the application operates or in batch mode over a dataset.

Who it’s for

It is designed for AI teams and developers building agentic systems and RAG applications who need to move from qualitative feel to quantitative evaluation and version comparison.

Highlights

• OpenTelemetry-native: Traces are portable to any OTLP-compatible backend like Jaeger or Grafana Tempo.
• Agentic Evaluations: Includes seven specialized evaluators for measuring reasoning coherence, plan adherence, tool selection, and execution efficiency.
• RAG Triad: Supports core RAG evaluation concepts like groundedness and context relevance.
• Flexible Instrumentation: Provides decorators for easy app instrumentation and a Selector API to target specific span attributes for evaluation.
• Broad Provider Support: Integrates with major LLM providers including OpenAI, Google Gemini, AWS Bedrock, and Snowflake Cortex.