The Open Arabic LLM Leaderboard 2

Overview

Hugging Face and its partners have launched the Open Arabic LLM Leaderboard 2 (OALL v2), a refined benchmarking platform designed to provide a more accurate and transparent evaluation of Large Language Models (LLMs) supporting Arabic. The update shifts the focus from machine-translated tasks to native Arabic benchmarks to better capture the language's complex morphology, syntax, and cultural nuances.

Addressing Limitations of Arabic Benchmarking

Previous Arabic LLM leaderboards often suffered from resource limitations, where users had to rely on model-maker documentation rather than direct comparisons, and integrity issues due to a lack of centralized verification for submitted results.

OALL v2 specifically addresses shortcomings identified in the first version of the leaderboard and other community initiatives:

  • Reduction of Translation Bias: Many v1 tasks were direct translations from English, which introduced linguistic mismatches and failed to reflect real-world Arabic use cases.
  • Benchmark Saturation: Several v1 benchmarks became less effective as models achieved near-perfect scores, making it impossible to differentiate incremental improvements.
  • Technical Corrections: A bug in the AlGhafa task was fixed; previously, the system evaluated responses based on the choice text rather than indices, which disproportionately penalized smaller or weaker models by up to 20 points.

New and Retained Benchmarks in OALL v2

OALL v2 follows two guiding principles: the removal of saturated and machine-translated tasks and the addition of high-quality native or human-curated benchmarks.

Retained Datasets

From the original OALL, the leaderboard keeps native Arabic datasets from the AlGhafa benchmark (including human-curated versions of Facts-Balanced, SOCAL, XGLUE, Sentiment, and Sentiment-Rating), Belebele (Arabic-MSA and Arabic-Dialects), and the Arabic EXAMS benchmarks.

New Datasets

  • Native Arabic MMLU: 40 tasks with approximately 15,000 multiple-choice questions in Modern Standard Arabic (MSA) sourced from school exams.
  • Human Translated MMLU (MMLU-HT): A 57-task human translation of the original English MMLU, curated by Inception as part of the JAIS project.
  • MedinaQA: A dataset focusing on general Arabic language and grammar aspects.
  • AraTrust: 522 human-written multiple-choice questions evaluating safety and truthfulness.

The ALRAGE Benchmark

OALL v2 introduces ALRAGE (Arabic Language Retrieval Augmented Generation Evaluation), a framework for evaluating RAG capabilities in Arabic.

  • Dataset: Sourced from 40 Arabic books across diverse topics, using Meta-Llama-3.1-70B for synthetic generation and validated by native speakers via a community sprint with Argilla.
  • Methodology: It utilizes an "LLM-as-judge" metric via the lighteval framework. Qwen2.5-72B-Instruct serves as the judge, scoring responses on a 0-10 rubric based on accuracy, relevance, and quality, which is then normalized to a 0-1 range.

Performance Analysis and Results

Statistical comparisons between OALL v1 and v2 reveal that performance on new benchmarks like AraTrust, ALRAGE, and the updated AlGhafa is more dispersed across different model sizes, whereas older tasks like ACVA and Toxigen showed significant saturation.

Model Rankings

  • Top Performers: Llama-3.3-70B-Instruct has emerged as the leader across all categories in OALL v2.
  • Pretrained Models: Qwen models continue to be strong baselines; specifically, Qwen/Qwen2-72B surpasses Qwen/Qwen2.5-72B as the top pretrained/continually pretrained model.
  • Cross-Leaderboard Correlation: There is a notable correlation between OALL v2 and other leaderboards such as AraGen and SEAL Arabic, with Llama-3.3-70B-Instruct ranking first on both OALL v2 and AraGen.

Model Family Trends

Analysis of the AceGPT and Jais families shows that larger models generally achieve higher average scores. However, a notable exception is the inceptionai/jais-family-30b-8k model, which surpasses the larger inceptionai/jais-adapted-70b model on OALL v2.

Operational Changes

To ensure fair access to computational resources and encourage diversity among participating organizations, OALL v2 now limits submissions to five models per organization per week. Additionally, the leaderboard now supports chat templates; if a template is found in the model configuration, it is used for evaluation.

Sources