QIMMA: A Quality-First Arabic LLM Leaderboard
QIMMA: A Quality-First Arabic LLM Leaderboard
QIMMA is a new Arabic Large Language Model (LLM) leaderboard that validates benchmarks before evaluating models to ensure reported scores reflect genuine Arabic language capability. It addresses the fragmentation and quality issues prevalent in existing Arabic NLP evaluations by applying a multi-stage validation pipeline to its dataset before model testing.
Solving Fragmentation in Arabic NLP Evaluation
Arabic NLP evaluation has historically been fragmented and plagued by quality issues. QIMMA identifies four primary pain points in the current landscape:
- Translation Artifacts: Many benchmarks are translations from English, leading to distributional shifts and culturally misaligned questions.
- Lack of Validation: Native Arabic benchmarks often lack rigorous quality checks, resulting in annotation inconsistencies and incorrect gold answers.
- Reproducibility Gaps: A lack of public evaluation scripts and per-sample outputs makes auditing results difficult.
- Coverage Gaps: Existing leaderboards often focus on narrow domains or isolated tasks.
QIMMA distinguishes itself by being open source, utilizing 99% native Arabic content, implementing systematic quality validation, including code evaluation, and providing public per-sample inference outputs.
QIMMA Dataset and Domain Coverage
QIMMA consolidates 109 subsets from 14 source benchmarks into a unified suite of over 52,000 samples. The evaluation spans seven distinct domains and three task types:
| Domain | Benchmarks | Task Types |
|---|---|---|
| Cultural | AraDiCE-Culture, ArabCulture, PalmX | MCQ |
| STEM | ArabicMMLU, GAT, 3LM STEM | MCQ |
| Legal | ArabLegalQA, MizanQA | MCQ, QA |
| Medical | MedArabiQ, MedAraBench | MCQ, QA |
| Safety | AraTrust | MCQ |
| Poetry & Literature | FannOrFlop | QA |
| Coding | 3LM HumanEval+, 3LM MBPP+ | Code |
Notably, QIMMA is the first Arabic leaderboard to integrate code evaluation via Arabic-adapted versions of HumanEval+ and MBPP+, allowing for the assessment of coding capabilities using Arabic-language problem statements.
The Quality Validation Pipeline
QIMMA employs a multi-stage validation pipeline to clean benchmarks before any model evaluation occurs.
Stage 1: Multi-Model Automated Assessment
Two state-of-the-art LLMs—Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B—independently score each sample against a 10-point rubric. A sample is eliminated if either model scores it below 7/10. If only one model flags a sample, it proceeds to human review.
Stage 2: Human Annotation and Review
Native Arabic speakers with cultural and dialectal familiarity review flagged samples. These annotators make final decisions on dialectal nuance, cultural context, and subjective interpretations to ensure the ground truth is accurate and culturally sensitive.
Systematic Quality Issues in Arabic Benchmarks
The validation pipeline revealed that quality issues are systematic rather than isolated. The discard rates varied by benchmark, with ArabicMMLU seeing the highest discard rate at 3.1% (436 samples) and MizanQA at 2.3% (41 samples).
Issues were categorized into a taxonomy of four types:
- Answer Quality: Factually wrong answers or mismatched gold indices.
- Text & Formatting Quality: Spelling/grammar errors, corrupt text, or duplicate samples.
- Cultural Sensitivity: Monolithic generalizations or stereotype reinforcement.
- Gold Answer Compliance: Misalignment between gold answers and evaluation protocols.
Refinement of Code Benchmarks
Because code benchmarks cannot be simply discarded, the team refined the Arabic problem statements in 3LM's adaptations of HumanEval+ and MBPP+. This resulted in significant modification rates: 88% of 3LM HumanEval+ prompts and 81% of 3LM MBPP+ prompts were modified for linguistic refinement, clarity, and structural corrections.
Evaluation Framework and Metrics
QIMMA utilizes LightEval, EvalPlus, and FannOrFlop for its evaluation framework to ensure reproducibility. Metrics are assigned based on task type:
- MCQ: Normalized Log-Likelihood Accuracy.
- Multi-select MCQ: Probability Mass on Gold Choices.
- Generative QA: F1 BERTScore (using AraBERT v02).
- Code: Pass@1.
Prompting is standardized across six template types (MCQ, MCQ-C, MCQ-I, QA, QA-C, and QA-F), all written in Arabic.
Leaderboard Results and Insights
As of April 2026, the top-performing models include:
- Overall Leader: Qwen/Qwen3.5-397B-A17B-FP8 (68.06 average score).
- Cultural and Linguistic Specialists: Jais-2-70B-Chat leads in ArabicMMLU and ArabCulture.
- Domain Specialists: Karnak leads in 3LM STEM and ArabLegalQA.
- Coding Performance: Multilingual models, specifically Qwen3.5-397B, continue to lead in coding tasks.
Key findings from the results include that model scale does not strictly guarantee the best performance in all domains, and Arabic-specialized models frequently outperform size-matched multilingual models on cultural and linguistic tasks.