Alyah: Emirati Dialect Benchmark for Arabic LLMs
TL;DR
Hugging Face and collaborators have released Alyah, a specialized benchmark designed to evaluate how Large Language Models (LLMs) handle the Emirati dialect. This release addresses a critical gap in Arabic NLP, as most existing benchmarks focus on Modern Standard Arabic (MSA), leaving the culturally rich and primarily spoken Emirati dialect under-evaluated.
The Alyah Benchmark: Scope and Motivation
Alyah (meaning "North Star" in Emirati) is designed to move beyond surface-level lexical knowledge to test a model's ability to interpret culturally embedded meanings, pragmatic usage, and dialect-specific nuances. Because the Emirati dialect is deeply tied to local heritage, poetry, and oral storytelling, it often contains expressions that cannot be inferred through literal translation from MSA.
Dataset Composition
The benchmark consists of 1,173 manually collected samples from native Emirati speakers to ensure linguistic authenticity. Each sample is a multiple-choice question with four candidate answers (one correct, three synthetic distractors reviewed for plausibility).
The dataset is divided into seven categories with varying difficulty levels:
| Category | Number of Samples | Difficulty |
|---|---|---|
| Language & Dialect | 619 | Difficult |
| Etiquette & Values | 173 | Medium |
| Imagery & Figurative Meaning | 121 | Medium |
| Historical & Heritage Knowledge | 89 | Difficult |
| Religious & Social Sensitivity | 78 | Medium |
| Greetings & Daily Expressions | 61 | Easy |
| Poetry & Creative Expression | 32 | Difficult |
Model Evaluation and Methodology
Researchers evaluated 54 language models, consisting of 23 base models and 31 instruction-tuned models. The evaluation pool included Arabic-native LLMs (e.g., Jais, Allam), multilingual models with Arabic support (e.g., Qwen, Llama), and regionally specialized models (e.g., Fanar, AceGPT).
Scoring Protocol
Models were assessed based on semantic correctness and appropriateness regarding Emirati usage rather than literal text overlap. Difficulty levels for each category were determined empirically based on the observed performance of the models across the board.
Key Findings and Performance Trends
Evaluation results show that while some models perform well, there is a significant gap between surface-level familiarity and deep cultural understanding.
Instruction Tuning Impact
Instruction-tuned models generally outperformed base models, particularly in categories involving conversational norms, etiquette, and figurative meaning. This suggests that alignment and instruction tuning help models better leverage their existing knowledge of MSA-based imagery to navigate dialectal patterns.
Performance Bottlenecks
- Hardest Categories: "Language and Dialect" and "Greetings and Daily Expressions" were consistently the most difficult across all model sizes. This is attributed to the fact that the Emirati dialect is primarily spoken and rarely written, limiting the training data available to LLMs.
- Multilingual Limitations: Even high-performing multilingual models showed notable degradation on the most challenging Alyah questions, indicating that dialect-specific semantic knowledge is not easily acquired through generic multilingual training.
- Arabic-Native Advantage: Arabic-native models generally performed more robustly on culturally grounded content, though performance remained non-uniform across categories.
Top Performing Models
- Base Models:
google/gemma-3-27b-ptled with 74.68% accuracy, followed bytiiuae/Falcon-H1-34B-Base(73.66%). - Instruction-Tuned Models:
falcon-h1-arabic-7b-instructachieved the highest accuracy at 82.18%, followed byhumain-ai/ALLaM-7B-Instruct-preview(77.24%). - Overall: The highest scoring large model was identified as
Jais-2-70B, withjais-2-8BandALLaM-7B-instructleading among smaller models.
Implications for Arabic LLM Development
Alyah serves as a diagnostic tool to highlight that dialectal competence is multi-dimensional. A model's ability to handle figurative language does not guarantee proficiency in poetry or heritage-related queries. The benchmark aims to guide future data collection and training efforts to create models that better serve the specific linguistic and cultural needs of users in the UAE.