agentscope-ai/OpenJudge

What it solves

OpenJudge is an open-source evaluation framework designed to help developers improve AI applications, such as chatbots and AI agents, through a systematic evaluation workflow. It addresses the difficulty of defining quality metrics and running evaluations at scale to identify weaknesses and iterate on application performance.

How it works

OpenJudge provides a library of over 50 production-ready graders across general, agent-specific, and multimodal domains. It allows users to create graders using four different methods:

• Customization: Defining rules via Python interfaces or prompt templates.
• Zero-shot Rubrics Generation: Using an LLM to automatically create evaluation criteria from a task description.
• Data-driven Rubrics Generation: Summarizing evaluation rubrics from annotated data using a GraderGenerator.
• Training Judge Models: Training dedicated models for complex scenarios where prompts are insufficient.

Results can be aggregated into overall scores and integrated into observability platforms like LangSmith and Langfuse, or used as reward signals for RL training via frameworks like VERL.

Who it’s for

It is intended for developers and AI researchers who are building and optimizing LLM-powered applications and agents, particularly those who need a professional, scalable way to evaluate quality and generate reward signals for fine-tuning.

Highlights

• Comprehensive Grader Library: Includes 50+ validated graders for semantic quality, agent trajectories, tool calling, and image-text coherence.
• Flexible Rubric Generation: Supports both zero-shot and data-driven generation of evaluation rubrics.
• Integrated Ecosystem: Seamlessly connects with LangSmith, Langfuse, and VERL.
• Online Playground: A browser-based interface for testing graders and building rubrics without installation.
• Agent Lifecycle Evaluation: Evaluates not just the final output, but the entire process including memory, reflection, and tool use.