Hugging Face Judge Arena: Benchmarking LLMs as Evaluators

Hugging Face has launched Judge Arena, a crowdsourced platform designed to identify the most effective LLMs for use as evaluators. This tool addresses the critical need to determine which models are best suited for the "LLM-as-a-Judge" approach, where AI models grade the natural language outputs of other AI applications.

Crowdsourced Evaluation Framework

Judge Arena utilizes a randomized, side-by-side battle system to benchmark AI judges. This methodology is based on the success of crowdsourced benchmarking seen in platforms like LMSys's Chatbot Arena. The process follows a specific workflow to ensure objective results:

  1. Sample Selection: Users choose a sample for evaluation, either by letting the system randomly generate a User Input/AI Response pair or by providing a custom sample.
  2. Dual Evaluation: Two anonymous LLM judges score the response and provide detailed reasoning for their scores.
  3. Human Voting: Users review the scores and critiques and vote for the judge whose evaluation best aligns with their own judgment.

To prevent bias and abuse, model identities remain hidden until after the vote is submitted.

Model Selection Criteria

Judge Arena focuses exclusively on generative models, excluding classifier models that only output a score. To be included in the arena, a model must meet two primary criteria:

  • Scoring and Critique Ability: The model must be able to score and provide a critique of other models' outputs effectively.
  • Prompt-ability: The model must be able to be prompted to evaluate using different scoring formats and criteria.

The platform currently includes 18 state-of-the-art LLMs from both proprietary and open-source providers, including:

  • OpenAI: GPT-4o, GPT-4 Turbo, GPT-3.5 Turbo
  • Anthropic: Claude 3.5 Sonnet / Haiku, Claude 3 Opus / Sonnet / Haiku
  • Meta: Llama 3.1 Instruct Turbo (405B, 70B, 8B)
  • Alibaba: Qwen 2.5 Instruct Turbo (7B, 72B), Qwen 2 Instruct 72B
  • Google: Gemma 2 (9B, 27B)
  • Mistral: Instruct v0.3 7B, Instruct v0.1 7B

Performance Metrics and Early Insights

Judge Arena uses an Elo rating system to calculate scores for each model, with the leaderboard updated hourly. Early observations from the platform indicate several key trends:

  • Competitive Open-Source Performance: While GPT-4 Turbo currently leads by a narrow margin, Llama and Qwen models are highly competitive and surpass the majority of proprietary models.
  • Efficiency of Smaller Models: Qwen 2.5 7B and Llama 3.1 8B have shown impressive performance, competing effectively with much larger models.
  • Validation of Llama as a Base: Preliminary results show Llama 3.1 70B and 405B ranking 2nd and 3rd, respectively. This aligns with existing research (such as Lynx, Auto-J, and SFR-LLaMA-3.1-Judge) suggesting that Llama models are strong base models for evaluation tasks.

Community Contribution and Data Sharing

To support the development of more aligned evaluators, Hugging Face and Atla have committed to sharing 20% of the anonymized voting data in the coming months. The platform is open for community feedback and suggestions for new models to be added to the leaderboard.

Sources