Arabic Leaderboards: Arabic Instruction Following and AraGen Updates

Hugging Face and Inception, in collaboration with the Mohammed bin Zayed University of Artificial Intelligence (MBZUAI), have launched the Arabic-Leaderboards Space. This unified platform centralizes Arabic AI evaluations, introducing the first publicly available benchmark for Arabic instruction following and updating the AraGen generative leaderboard to improve the rigor of Arabic LLM assessments.

Arabic-Leaderboards Space

The Arabic-Leaderboards Space serves as a central hub for Arabic AI evaluations across various modalities. Currently, the platform hosts two primary live leaderboards: AraGen-03-25 and Arabic Instruction Following. The developers intend to expand this space to include additional leaderboards and tasks as their research progresses.

AraGen-03-25 Release and Evaluation

AraGen is a generative Arabic leaderboard designed to benchmark LLMs on generative tasks. The latest release, AraGen-03-25, introduces several key updates to the evaluation framework.

Dataset Expansion and Distribution

The AraGen-03-25 dataset has been expanded to 340 question-and-answer pairs, increasing from 279 in the previous version. The task distribution is as follows:

  • Question Answering: ~200 pairs
  • Reasoning: 70 pairs
  • Safety Questions: 40 pairs
  • Orthographic and Grammatical Analysis: 30 pairs

Dynamic Evaluation and Ranking Stability

To ensure reliability, Inception utilizes a dynamic evaluation cycle where datasets remain private for three months (blind testing) before public release. The AraGen-12-24 benchmark has now been publicly released, including model responses evaluated by Claude-3.5-Sonnet using 3C3H guidelines.

Analysis of ranking variations between the previous system prompt (SP1) and the current system prompt (SP2) shows that rankings are generally stable. The model o1-2024-12-17 consistently maintains the top position, although its absolute score dropped from 82.67% (AraGen-12-24) to 70.25% (AraGen-03-25), indicating that the updated benchmark is significantly more challenging.

The 3C3H Evaluation Measure

The 3C3H metric assesses the chat capabilities of models based on factuality and usability. A key finding from the recent analysis is the high correlation between correctness, helpfulness, and harmlessness, while conciseness remains an outlier. Most models are rewarded for verbosity, though the analysis noted that silma-ai/SILMA-9B-Instruct-v1.0 achieved higher conciseness than larger open-weight models, albeit at a cost to helpfulness compared to its base model, google/gemma-2-9b-it.

Arabic Instruction Following (Arabic IFEval)

Instruction following is the ability of an LLM to adhere to specific, objectively measurable constraints. Inception has introduced Arabic IFEval, the first public benchmark of its kind for the Arabic language.

Dataset Construction

Inspired by Google's English IFEval, the Arabic IFEval dataset consists of approximately 300 prompts. Rather than simple translation, the team adapted prompts to reflect Arabic linguistic nuances and cultural contexts. Key features include:

  • Linguistic Challenges: Prompts focusing on Arabic phonetics, orthography, and morphology (e.g., the use of diacritical marks/tashkīl).
  • Constraint Types: Verifiable commands such as avoiding specific letters (e.g., Alef), using specific word frequencies, or adhering to strict length and punctuation constraints.
  • Expert Validation: All prompts were validated by Arabic linguists to ensure grammatical accuracy and clarity.

Evaluation Methodology

Evaluation is conducted via automated Python scripts to ensure reproducibility and remove AI-judge bias. The benchmark uses two levels of accuracy:

  • Prompt-level strict accuracy: A rigorous metric where a prompt is marked successful only if all instructions within that prompt are followed.
  • Instruction-level score: A more lenient metric evaluating partial compliance.

Performance Results

Preliminary results comparing prompt-level accuracy on Arabic vs. English IFEval show that models generally perform better in English. For example, claude-3.5-sonnet leads the Arabic rankings with 72.5% accuracy, compared to 84.7% in English. Other top performers in Arabic include gpt-4o-2024-08-06 (70.8%) and gpt-4o-mini-2024-07-18 (68.1%).

Future Roadmap

Inception and MBZUAI plan to continue updating the Arabic-Leaderboards Space. Upcoming additions include a leaderboard for visual question-answering (VQA) powered by camel-bench and kitab.

Sources