JudgmentLabs/judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

What it solves

Judgeval addresses the difficulty of improving LLM-powered agents by providing a way to detect failures, triage root causes, and validate fixes using real production data rather than relying on static test sets.

How it works

It uses a Python SDK to instrument agent functions via OpenTelemetry-based tracing, capturing inputs, outputs, and token usage. Users can then define "agent judges"—prompt-based scorers that evaluate agent behaviors at scale. These judges can be run against live production traffic (server-side with no latency impact) or replayed on historical traces to verify that changes actually fix identified issues.

Who it’s for

Developers building LLM agents and AI applications who need production-grade observability, automated evaluation, and a continuous improvement loop for their agent's behavior.

Highlights

  • OpenTelemetry Tracing: Full compatibility with existing observability stacks using @Tracer.observe().
  • Agent Judges: Structured, prompt-based scoring to label and track agent behaviors over time.
  • Online Monitoring: Server-side scoring of live traffic with Slack alerts for regressions.
  • Broad Integration: Native support for OpenAI, Anthropic, Google GenAI, Together AI, LangGraph, OpenLit, and Claude Agent SDK.
  • JQL: A query language for retrieving specific traces and spans.

Related

  • Project
    canwhite/AgentEval

    A transparent HTTP proxy that captures Agent-LLM traffic to provide automated grading, behavioral diagnosis, and source-code probing for AI agent evaluation.

  • Project
    langchain-ai/langsmith-sdk

    Client SDKs for the LangSmith platform, enabling developers to debug, evaluate, and monitor LLM applications and intelligent agents.

  • Project
    truera/trulens

    An OpenTelemetry-native evaluation framework for AI agents and RAG apps that uses LLM judges to trace and score application performance, cost, and quality.

  • Project
    MagnivOrg/prompt-layer-library

    A Python SDK for PromptLayer that enables prompt versioning, tracing, and monitoring for LLM applications and agents across multiple AI providers.

  • Project
    comet-ml/opik

    An open-source LLM observability and evaluation platform for tracing AI agents, managing prompts, and monitoring production LLM applications.