trulens: a systematic evaluation and observability framework for tracking LLM experiments and agentic behavior

What it solves

TruLens eliminates the need for "vibe-checking" LLM applications by providing a systematic way to evaluate and track experiments. It helps developers identify failure modes in prompts, models, retrievers, and knowledge sources, allowing for data-driven iteration to improve application performance.

How it works

TruLens uses OpenTelemetry-based instrumentation to capture every function call, LLM generation, and tool invocation as a structured span. It then applies "Feedback Functions" and specific evaluators (such as the RAG Triad) to these spans. These evaluations can be run inline as the app operates or in offline batch mode on pre-collected datasets, with results viewable in a user interface.

Who it’s for

It is designed for developers building LLM-powered applications, specifically those using RAG (Retrieval-Augmented Generation) or agentic systems, who need to move beyond anecdotal testing to rigorous evaluation.

Highlights

• OpenTelemetry Integration: Fully interoperable with observability tools like Jaeger, Grafana Tempo, and Datadog.
• Agentic Evaluators: Seven specialized metrics to measure reasoning coherence, plan adherence, tool selection, and execution efficiency.
• Flexible Evaluation: Supports both real-time inline evaluation and high-throughput batch processing.
• Broad Compatibility: Integrates with major frameworks like LangChain and LlamaIndex, and supports numerous LLM providers including OpenAI, Anthropic, Gemini, and Bedrock.