alphadl/AdaRubrics

What it solves

AdaRubric addresses the failure of static evaluation rubrics (like "Helpfulness" or "Safety") when judging LLM agents. Because different agent tasks require different success criteria—such as "Correctness" for code debugging versus "Action Efficiency" for web navigation—a one-size-fits-all rubric often mismeasures quality. AdaRubric provides a way to generate task-specific rubrics and dense reward signals to more accurately evaluate agent trajectories.

How it works

The project implements a three-stage pipeline:

1. Rubric Generator: Uses an LLM to create a set of orthogonal evaluation dimensions and 5-point scoring criteria based on the specific task description.
2. Trajectory Evaluator: Scores each step of an agent's trajectory (Thought $\rightarrow$ Action $\rightarrow$ Observation) across every generated dimension, assigning both a score and a confidence weight.
3. Data Filter: Uses various strategies to curate high-quality data for training. Notably, the DimensionAwareFilter prevents a high average score from masking a catastrophic failure in a single critical dimension.

Who it’s for

It is designed for researchers and developers training LLM agents who need reliable evaluation metrics and high-quality preference pairs for reward learning (e.g., DPO or PPO).

Highlights

• Task-Adaptive: Generates custom rubrics on the fly instead of relying on static benchmarks.
• Dense Feedback: Provides per-step, per-dimension scoring rather than a single final grade.
• Flexible Aggregation: Supports Weighted Mean, Geometric Mean, and Min Score aggregators depending on the task's criticality.
• Proven Gains: Demonstrated improvements in DPO task success across WebArena, ToolBench, and AgentBench, as well as better correlation with human judgment.