3LM: A Benchmark for Arabic LLMs in STEM and Code

3LM (علم) is a multi-component benchmark designed to evaluate Arabic Large Language Models (LLMs) on STEM (Science, Technology, Engineering, and Mathematics) subjects and code generation. It addresses a critical gap in Arabic NLP, where previous evaluations focused primarily on general-purpose tasks like sentiment analysis and summarization rather than structured reasoning and formal logic.

Benchmark Components

3LM consists of three distinct datasets targeting different evaluation axes to provide a comprehensive view of model capabilities in technical domains.

1. Native STEM

Native STEM consists of 865 multiple-choice questions (MCQs) sourced from authentic Arabic educational materials, including textbooks, worksheets, and exam banks for grades 8 through 12. The dataset covers five core subjects: Physics, Chemistry, Biology, Mathematics, and Geography. Each question includes metadata for domain and difficulty (rated 1–10). The creation pipeline utilized OCR, LaTeX math parsing via Pix2Tex, LLM-assisted extraction, and manual review.

2. Synthetic STEM

To test higher-difficulty reasoning and reduce answer bias, the Synthetic STEM component contains 1,744 MCQs created using the YourBench pipeline. This process involves chunking and summarizing Arabic textbook text, which is then used as input for an LLM-driven question generation system. The resulting questions focus on mid-to-high difficulty conceptual, analytical, and application-based problems and were validated through manual review.

3. Arabic Code Benchmarks

3LM introduces the first code datasets for testing Arabic LLMs on natural language programming prompts by translating and adapting HumanEval+ and MBPP+ benchmarks. GPT-4o was used for prompt translation, with quality ensured via a backtranslation pipeline (rejecting samples with ROUGE-L F1 < 0.8) and human filtering. To maintain scoring fidelity, the original code and test suites remain unchanged, and evaluations use the EvalPlus framework for pass@1 and pass@1+ metrics.

Key Evaluation Results

Evaluation of over 40 LLMs revealed varying strengths across different task formats and models.

  • Multiple-Choice Questions: Qwen2.5-72B-Instruct achieved the highest performance, scoring 71.8% on native STEM and 67.0% on synthetic STEM.
  • Generative Completion: Gemma-3-27B demonstrated the strongest results with 43.2% accuracy on STEM answers.
  • Code Generation: GPT-4o led the benchmarks with 83.5% pass@1+ on HumanEval-ar and 63.6% pass@1+ on MBPP-ar.

Analysis showed a strong correlation (~0.97) between Arabic and English pass@1 scores, indicating that the quality of language-specific prompts significantly influences model outcomes. Additionally, instruction-tuned models proved to be significantly more stable than base models when subjected to distractor perturbation.

Implementation and Tooling

3LM is designed for reproducibility and integrates with standard evaluation tools:

  • lighteval: Used for both multiple-choice and open-ended question evaluation for STEM datasets.
  • evalplus: Used for robust pass@1 and pass@1+ code scoring via function-level testing.

All evaluation scripts, configurations, and pipelines are available on GitHub, supporting any model compatible with OpenAI APIs or HuggingFace Transformers.

Dataset Access

All 3LM datasets are open-source and available on HuggingFace Datasets:

  • 3LM-native-stem
  • 3LM-synthetic-stem
  • 3LM-code-arabic

Sources