JudgmentLabs/judgeval
The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.
What it solves
Judgeval provides a continuous-improvement stack for LLM-powered agents. It helps developers detect failures, triage root causes, and validate fixes using production data, ensuring that agent behaviors are monitored and improved over time.
How it works
The system uses an OpenTelemetry-based tracing system to capture inputs, outputs, and token usage of agent functions. It employs "agent judges"—prompt-based scorers that evaluate agent behaviors at scale. These judges can be run against live production traffic or replayed on historical traces to validate fixes. Additionally, it offers online monitoring with server-side scoring to alert developers via Slack when regressions occur.
Who it’s for
Developers building LLM-powered applications and agents who need a way to trace their agent's execution paths and systematically evaluate their performance using production data.
Highlights
- OpenTelemetry-based tracing: Instrument functions with a simple decorator to capture execution data.
- Agent judges: Use prompt-based scorers to create structured records of agent behavior.
- Online monitoring: Server-side scoring of live traffic with zero latency impact and Slack alerts for regressions.
- Broad integrations: Supports major LLM providers (OpenAI, Anthropic, Google GenAI, Together AI) and frameworks like LangGraph and Claude Agent SDK.
- MCP Server: Connects to MCP-compatible AI tools to surface failures directly in an IDE or AI assistant.