AraGen Benchmark and Leaderboard: Introducing 3C3H Evaluation for Arabic LLMs

TL;DR

Hugging Face announced AraGen, a generative tasks benchmark and leaderboard for Arabic large language models, built around the new 3C3H evaluation measure that scores model responses across correctness, completeness, conciseness, helpfulness, honesty, and harmlessness using an LLM‑as‑judge approach.

Overview of the 3C3H Measure

The 3C3H measure is a holistic assessment that combines factuality and usability. It evaluates six dimensions:

  • Correctness (binary): factual accuracy relative to ground truth.
  • Completeness (binary): whether the answer addresses all parts of the question.
  • Conciseness (1‑5): appropriateness of brevity while retaining needed information.
  • Helpfulness (1‑5): degree to which the answer assists the user.
  • Honesty (1‑5): accuracy of all information, penalizing hallucinations.
  • Harmlessness (1‑5): absence of offensive or biased content. Binary scores are computed first; if correctness or completeness is zero, all other dimensions are set to zero. Scaled scores are normalized from [1,5] to [0,1]. The overall 3C3H score aggregates the six dimensions per sample and averages across the dataset.

AraGen Benchmark and Leaderboard

AraGen provides a leaderboard that ranks both open and proprietary Arabic LLMs using the 3C3H measure. The benchmark consists of 279 human‑verified questions covering four tasks: question answering, orthographic and grammatical analysis, reasoning, and safety.

Evaluation Pipeline

  1. Model Submission – users submit a model for evaluation.
  2. Response Generation – the model generates answers for the fixed AraGen Benchmark set.
  3. LLM as Judge – a selected LLM evaluates each model answer against the verified ground truth, following the 3C3H guidelines and outputting scores in JSON.
  4. Scoring and Normalization – binary scores are derived first; scaled scores are normalized to [0,1].
  5. Leaderboard Reporting – results appear on two leaderboards: a general 3C3H leaderboard (showing overall and per‑dimension scores) and a tasks leaderboard (showing scores for each of the four tasks).

Dynamic Leaderboard for Robustness

To mitigate data contamination, AraGen employs a dynamic evaluation strategy:

  • Blind Test Sets – each dataset remains private for a three‑month evaluation period, preventing leakage into model training.
  • Periodic Updates – after three months, the blind set is replaced by a new human‑verified question‑answer pair set, preserving structure, complexity, and domain balance.
  • Open‑Sourcing – following the blind period, the dataset and evaluation code are publicly released, enabling independent verification and reproducibility.

Dataset Design

The AraGen Benchmark includes 279 examples distributed across the four tasks. Interaction types are:

  • Single Interaction – simple question‑answer.
  • Conversational Interaction – multi‑turn exchanges; evaluation focuses on the final turn while considering flow.
  • Follow‑Up Interaction – two‑turn sequences where the second answer depends on the first; the first answer is weighted twice as heavily to emphasize factual continuity. For orthographic and grammatical analysis, the data splits evenly between Arabic grammar and Arabic dictation grammar (50% each). Safety examples belong exclusively to the safety category.

Judge Evaluation and Selection

The authors experimented with several candidate LLMs as judges, including GPT‑4o, GPT‑4o‑mini, Claude‑3.5‑sonnet, Claude‑3‑haiku, Llama 3.1‑405b, and a jury system. Evaluation criteria were agreement with human judges (Cohen’s Kappa), score consistency (standard deviation), self‑bias, and hallucination tendency.

  • Agreement: GPT‑4o‑mini achieved the highest Kappa with humans (0.46), closely followed by Claude‑3.5‑sonnet.
  • Consistency: The jury system showed the lowest average standard deviation (0.0049); among single judges, Claude‑3.5‑sonnet was most stable (0.0063).
  • Self‑Bias: Claude‑3.5‑sonnet displayed minimal self‑preferential scoring, whereas GPT‑4o and GPT‑4o‑mini scored their own outputs higher than other judges.
  • Hallucination: Both Claude‑3.5‑sonnet and GPT‑4o‑mini showed high adherence to guidelines; any discrepancy for Claude‑3.5‑sonnet was traced to API errors, not hallucinations.

Based on these results, Claude‑3.5‑sonnet was selected as the primary judge for the AraGen Leaderboard due to its low variability, low self‑bias, and strong human agreement. The jury system was excluded because it offered no ranking improvement, risked bias amplification, and incurred high computational cost.

Conclusion

AraGen introduces a rigorous, dynamic evaluation framework for Arabic LLMs that unifies factuality and usability through the 3C3H measure. By keeping evaluation data private for three‑month cycles and releasing it afterward, the leaderboard reduces data leakage risks while remaining transparent and reproducible. The authors plan to extend the benchmark with new tasks, semi‑automate dataset creation, and adapt the framework to other under‑resourced languages.

Sources