BenCzechMark: A Comprehensive Evaluation Suite for Czech LLMs

Hugging Face, in collaboration with BUT FIT, FI MUNI, and CIIRC CTU, has introduced BenCzechMark, the first comprehensive evaluation suite designed to assess the capabilities of Large Language Models (LLMs) in the Czech language. The benchmark provides a standardized framework to measure reasoning, grammatical correctness, factual knowledge of Czech culture, and language modeling probability.

Benchmark Composition and Task Categories

BenCzechMark consists of 50 tasks spanning 9 categories, with 90% of the content being native Czech and 10% translated (primarily via CUBBITT and DeepL). The suite evaluates models across the following dimensions:

  • Reading Comprehension: Uses Belebele (Accuracy) and SQAD3.2 (Exact Match) to test information extraction from context.
  • Factual Knowledge: Tests stored knowledge via Umimeto (Accuracy), TriviaQA (Exact Match), and NaturalQuestions (Exact Match).
  • Czech Language Understanding: Focuses on syntax and nuance using CERMAT (EM/AUROC/Acc), Grammar Error Detection (AUC), and Agree (Accuracy).
  • Language Modeling: Measures text sampling likelihood via the Czech National Corpus (Perplexity) and HellaSwag (Accuracy).
  • Math Reasoning in Czech: Evaluates problem-solving using Klokan QA (Accuracy), CERMAT (Exact Match/Accuracy), and Umimeto Math (Accuracy).
  • Natural Language Inference: Tests entailment and support using Czech SNLI (AUROC), CSFever (AUROC), CTKFacts (AUROC), and Propaganda (AUROC).
  • Named Entity Recognition (NER): Identifies entity types using CNEC2.0 (Exact Match) and Court Decisions (Exact Match).
  • Sentiment Analysis: Quantifies sentiment via Subjectivity (AUROC) and CzechSentiment (AUROC).
  • Document Retrieval: Identifies relevant passages using Historical IR (Accuracy).

Evaluation Metrics

To handle diverse task types, the benchmark employs four primary metrics:

  1. Accuracy (Acc): For multiple-choice tasks.
  2. Exact Match (EM): For open short-answer generation.
  3. Area Under the Receiver Operating Characteristic Curve (AUROC): Used for classification to eliminate the need for threshold calibration and ensure fairer comparisons across models.
  4. Word-level Perplexity (Ppl): For language modeling tasks, normalized per word in the corpus.

Novel Duel Scoring Mechanism

Because the benchmark uses multiple metrics with varying scales, a simple average is not feasible. BenCzechMark introduces a Duel Win Score (DWS) to determine final rankings.

For every task, models are compared using statistical significance tests at α=0.05. The tests used include one-tailed paired t-tests (for ACC and EM), a Bayesian test (for AUROC), and bootstrapping (for Ppl). A model's DWS is the proportion of these "duels" it wins against all other models on a specific task. The final aggregate score is the Macro-averaged model win-rate, calculated as the average of category DWSs.

Model Performance and Leaderboard Results

An evaluation of 26 open-weight models (using 3-shot examples, a 2048 token input length, and average-pooling for log-probability aggregation) revealed the following key findings:

  • Llama-405B is the overall top-performing model on the leaderboard.
  • Qwen-72B demonstrated superior performance in Math and Information Retrieval but lagged in other categories.
  • Aya-23-35B excelled in Sentiment Analysis and Language Modeling.
  • Gemma-2 9B outperformed significantly larger models in Czech reading comprehension.

Researchers and developers can submit their own models to the BenCzechMark leaderboard via the dedicated Hugging Face Space to further the development of Czech-centric LLMs.

Sources