Falcon-Arabic: A Breakthrough in Arabic Language Models

The Technology Innovation Institute (TII) has introduced Falcon-Arabic, a 7B parameter language model that establishes a new performance benchmark for Arabic Natural Language Processing (NLP). By adapting the Falcon 3 architecture, the model provides state-of-the-art capabilities in Arabic grammar, mathematical reasoning, and dialect understanding, outperforming models up to four times its size.

Technical Architecture and Training Recipe

Falcon-Arabic was developed by adapting the Falcon 3-7B multilingual foundation rather than training from scratch. This approach was chosen because adapted and multilingual models have consistently shown higher efficiency and capability on the Open Arabic LLM Leaderboard.

Tokenizer and Embedding Adaptation

Because Falcon 3-7B lacked native Arabic support at the tokenizer and embedding levels, TII implemented two primary modifications:

  • Vocabulary Expansion: The tokenizer was extended with 32,000 Arabic-specific tokens.
  • Similarity-Based Initialization: A novel embedding initialization strategy based on textual similarity was used to map new Arabic tokens to semantically related embeddings in the existing vocabulary. This allowed the model to inherit prior knowledge regarding reasoning patterns and abstract concepts.

Pretraining Curriculum

The model underwent continuous pretraining using 100% native Arabic datasets to avoid the cultural bias often introduced by machine-translated content. The training followed a multi-stage curriculum:

  1. Initial Stage: Focused on general knowledge and dialect-rich Arabic content to stabilize the model.
  2. Later Stages: Emphasized mathematical reasoning, coding, and complex problem solving.

Performance Benchmarks

Falcon-Arabic was evaluated using the OALL v2 (Open Arabic LLM Leaderboard), which consists of six multiple-choice tasks (Arabic MMLU native and translated, Arabic Exams, Alghafa, MadinahQA, and Aratrust) and one generative benchmark (Alrage).

Base Model Performance

Falcon-Arabic-7B-Base leads in several key benchmarks, including Arabic MMLU, Exams, MadinahQA, and Aratrust. It outperforms all existing Arabic LLMs within its size category and surpasses models up to four times larger.

Instruct Model Performance

Following pretraining, the model underwent post-training alignment to create Falcon-Arabic-7B-Instruct. This process involved:

  • Supervised Fine-Tuning (SFT): Utilizing high-quality public datasets and internally collected native Arabic instruction data.
  • Direct Preference Optimization (DPO): A reinforcement learning method used to align outputs with human preferences for safety, relevance, and helpfulness.

Falcon-Arabic Instruct outperforms all other Instruct-aligned Arabic LLMs in its size class and exceeds the performance of significantly larger models in instruction following and open-ended dialogue.

Capabilities and Applications

Falcon-Arabic supports a context length of 32,000 tokens, making it suitable for knowledge-intensive tasks and long-document processing. Key capabilities include:

  • Retrieval-Augmented Generation (RAG): Handling long contexts for grounded answers.
  • Multilingualism: Support for Arabic, English, and several other languages.
  • Dialectal Understanding: Fluency in both Modern Standard Arabic (MSA) and various regional dialects.
  • Zero-Shot Translation: The model demonstrates strong machine translation capabilities despite not being specifically fine-tuned for the task.

Limitations

Falcon-Arabic is subject to common LLM limitations, including:

  • Hallucinations: The potential to produce plausible but incorrect information.
  • Prompt Sensitivity: Performance variations based on how prompts are phrased.
  • Context Length: Varying performance when processing very long contexts.

Sources