Introducing the Open Arabic LLM Leaderboard

Hugging Face and the Technology Innovation Institute (TII) have launched the Open Arabic LLM Leaderboard (OALL), a specialized platform designed to evaluate and compare the performance of Arabic Large Language Models (LLMs). This initiative addresses the resource gap in Natural Language Processing (NLP), where evaluation tools have historically been skewed toward English, providing a standardized way to improve models for over 380 million Arabic speakers.

Comprehensive Benchmark Datasets

The OALL utilizes a diverse collection of robust datasets to ensure a comprehensive evaluation of model capabilities across various linguistic tasks:

  • AlGhafa Benchmark: Developed by the TII LLM team, this benchmark evaluates reading comprehension, sentiment analysis, and question answering. It consists of 11 native Arabic datasets and 11 translated versions of widely adopted English NLP benchmarks.
  • ACVA and AceGPT Benchmarks: These include 58 datasets from the "AceGPT, Localizing Large Language Models in Arabic" paper, as well as translated versions of the MMLU and EXAMS benchmarks to cover a broad spectrum of linguistic complexities and subtleties.

Evaluation Metrics and Technical Infrastructure

The leaderboard employs normalized log likelihood accuracy as its primary metric. This choice is specifically suited for the multiple-choice and yes/no question formats used in the benchmark datasets, providing a fair measurement of performance across different task types.

Technically, the OALL is built using the following stack:

  • Framework: The lighteval library is used to run evaluations. The team integrated Arabic benchmarks into lighteval via PR #44 and PR #95 to allow the community to perform out-of-the-box evaluations.
  • Architecture: The front-end and back-end are inspired by the demo-leaderboard, with the back-end hosted locally on the TII cluster.

Model Submission Guidelines

To maintain the integrity of the leaderboard, participants must follow specific technical requirements for model submission:

  • Precision Alignment: Model precision must align with the original model to ensure correct display and evaluation.
  • Compatibility: Models must be loadable via AutoClasses (AutoConfig, AutoModel, AutoTokenizer) and set to public visibility.
  • Weight Format: Weights must be converted to safetensors to enable faster loading and the display of parameter counts in the Extended Viewer.
  • Licensing: Only openly licensed models are accepted to promote accessibility.
  • Documentation: A complete model card is required, as this data is automatically extracted for the leaderboard display.

Future Roadmap and Community Goals

The OALL team plans to expand the platform's scope through several upcoming initiatives:

  • Specialized Leaderboards: Development of new categories, including a leaderboard for Retrieval Augmented Generation (RAG) scenarios and a chatbot arena using ELO scores based on user preferences.
  • OpenDolphin Benchmark: An open replication of the "Dolphin: A Challenging and Diverse Benchmark for Arabic NLG" paper, which will include approximately 50 datasets.
  • Cross-Language Application: The project aims to serve as a blueprint for creating similar large-scale, language-specific leaderboards for other underrepresented languages to bridge the global NLP resource gap.

Sources