JudgmentLabs/judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

What it solves

Judgeval tackles a core pain point for teams building LLM-powered agents: knowing when they fail, why they fail, and whether a fix actually solves the problem. Instead of blind-shipping changes, it gives you production-grade observability and a way to validate improvements against real traces before they go live.

How it works

The SDK adds observability with two lines of setup. You wrap your OpenAI (or other supported) client and annotate functions with @Tracer.observe() to automatically capture inputs, outputs, and token usage via OpenTelemetry. You then define prompt-based "judges" that score traces at scale — a binary judge applies a label when true, and a classification judge picks from given options. These labels accumulate into a searchable record of agent behavior. You can run judges against live production traffic with no latency impact (online monitoring) or replay them on historical traces to validate a fix. Optional SQL access lets you query the underlying telemetry data directly, with read-only security enforced server-side.

Who it's for

Engineers and teams building LLM agents who need to move beyond ad-hoc testing. It's especially useful for those already using OpenAI, Anthropic, Google GenAI, or Together AI, and for anyone working with LangGraph, OpenLit, or the Claude Agent SDK. The CLI and MCP server make it accessible from the terminal and inside AI assistants/IDEs.

Highlights

  • OpenTelemetry-based tracing that automatically captures inputs, outputs, and token usage
  • Prompt-based judges (binary and classification) that label traces and build a searchable behavior record
  • Online monitoring that scores live production traffic server-side with zero latency impact and Slack alerting
  • Broad auto-instrumentation for major LLM providers and integration with popular agent frameworks
  • A SQL interface for querying telemetry data, plus CLI and MCP server for flexible workflows

Related

  • Project
    langfuse/langfuse-python

    A Python SDK for the Langfuse platform that provides observability, tracing, prompt management, and evaluation tools for LLM applications.

  • Project
    MagnivOrg/prompt-layer-library

    A Python SDK for PromptLayer that enables prompt versioning, tracing, and monitoring for LLM applications and agents across multiple AI providers.

  • Project
    truera/trulens

    An OpenTelemetry-native evaluation and observability framework that uses LLM judges to trace agent failures and optimize the cost-quality tradeoff in AI applications.

  • Project
    comet-ml/opik

    An open-source LLM observability and evaluation platform for AI agent tracing, prompt management, and production monitoring.

  • Project
    darkrishabh/agent-skills-eval

    agent‑skills‑eval is a TypeScript/Node.js CLI + SDK that runs an Agent Skill both with and without its `SKILL.md` context, grades the two outputs with a separate LLM judge, and produces JSON artifacts plus a static HTML report. It supports OpenAI‑compatible APIs, custom providers, tool‑call assertions, and full agentskills.io spec validation, making it suitable for skill development, CI testing, and comparative analysis.