Hugging Face Open Leaderboard for Hebrew LLMs

Hugging Face has introduced an open leaderboard for Hebrew Large Language Models (LLMs) to provide a standardized evaluation framework for a language often underserved by general LLM benchmarks. This initiative aims to catalyze the development of models that are linguistically and culturally accurate by addressing the specific morphological complexities of Hebrew.

Addressing Hebrew Linguistic Challenges

Hebrew is a morphologically rich language characterized by a complex system of roots and patterns, where prefixes, suffixes, and infixes modify meaning, tense, and plurality. Because traditional tokenization strategies are often designed for morphologically simpler languages, they frequently prove ineffective for Hebrew. This creates a gap where existing LLMs may struggle to process the nuances of the language, necessitating dedicated benchmarks that reflect these unique linguistic properties.

Evaluation Metrics and Benchmarks

The leaderboard utilizes four key datasets to test the understanding and generation of Hebrew. These benchmarks employ a few-shot prompt format to ensure models can adapt and respond correctly with limited context.

Hebrew Question Answering

This task evaluates the model's ability to comprehend Hebrew syntax and semantics and accurately retrieve answers based on provided context. It utilizes the test subset of the HeQ dataset.

Sentiment Accuracy

This benchmark assesses the model's capability to classify Hebrew statements as positive, negative, or neutral based on linguistic cues. It uses the Hebrew Sentiment dataset.

Winograd Schema Challenge

This task measures the model's ability to resolve pronouns and handle contextual ambiguity using logical reasoning and general world knowledge. It uses a translation of the Winograd Schema Challenge to Hebrew provided by Dr. Vered Schwartz.

Translation

This benchmark evaluates the proficiency of models in translating between English and Hebrew, focusing on linguistic accuracy, fluency, and the preservation of meaning. It utilizes the NeuLabs-TedTalks aligned translation corpus.

Technical Implementation

The leaderboard is built using the Demo Leaderboard template and is inspired by the Open LLM Leaderboard. The technical pipeline operates as follows:

• Deployment: Submitted models are automatically deployed via Hugging Face Inference Endpoints.
• Evaluation: Models are evaluated through API requests managed by the lighteval library.

Sponsorship and Collaboration

The project is sponsored by DDR&D IMOD / The Israeli National Program for NLP in Hebrew and Arabic, in collaboration with DICTA: The Israel Center for Text Analysis and Webiks. Scientific consultation and guidance were provided by Prof. Reut Tsarfaty from Bar-Ilan University.