Open Ko-LLM Leaderboard

TL;DR

Hugging Face and Upstage have introduced the Open Ko-LLM Leaderboard, a specialized evaluation platform designed to foster a transparent and competitive ecosystem for Korean Large Language Models (LLMs). By utilizing private test sets and Korean-specific benchmarks, the leaderboard aims to prevent data contamination and accurately measure the linguistic and cultural nuances of the Korean language.

Fair Benchmarking via Private Test Sets

The Open Ko-LLM Leaderboard distinguishes itself from traditional benchmarks by employing a closed test set environment. While many leaderboards use open test sets, the Open Ko-LLM Leaderboard maintains its datasets as private to prevent test set contamination—a common issue where models are trained on the evaluation data—and to ensure a more equitable comparison between models.

To ensure cultural and linguistic relevance, the platform utilizes translated benchmark datasets, such as Ko-MMLU, that are specifically adapted to reflect the distinctive attributes of the Korean language and culture.

Evaluation Tasks and Metrics

The leaderboard employs five distinct evaluation methods to measure a wide array of LLM capabilities:

  • Ko-ARC (AI2 Reasoning Challenge): A multiple-choice test assessing scientific thinking, complex reasoning, and problem-solving skills. Performance is measured by accuracy rates.
  • Ko-HellaSwag: Evaluates situational comprehension and the ability to predict the most likely next scenario, using either generative or multiple-choice formats. Accuracy is the primary metric.
  • Ko-MMLU (Massive Multitask Language Understanding): A broad multiple-choice test across various topics and fields to measure general language comprehension and versatility. Metrics include overall accuracy and domain-specific performance.
  • Ko-Truthful QA: A multiple-choice benchmark that tests a model's ability to discern factual accuracy and truthfulness from a set of options, measured by selection accuracy.
  • Ko-CommonGEN V2: A custom benchmark developed specifically for the Open Ko-LLM Leaderboard to assess whether models can generate outputs aligned with Korean common sense and cultural relevance.

Adoption and Model Performance Trends

Within five months of launch, the Open Ko-LLM Leaderboard has seen over 1,000 model submissions, reaching a quarter of the volume of the original English Open LLM Leaderboard (which has over 4,000 models). Participants include individual researchers, academic institutions (such as KAIST and Korea University), and corporations (including KT, Lotte Information & Communication, and Yanolja).

Key performance observations include:

  • Top Performers: Models that underwent cross-lingual transfer or Korean-specific fine-tuning (e.g., Upstage's SOLAR) and models fine-tuned from strong foundational models like LLaMa2, Yi, and Mistral demonstrate the strongest results.
  • Notable Milestones: KT's Mi:dm 7B model topped the rankings for models with 7B parameters or fewer and has been made available for public use.

Infrastructure Challenges and Future Vision

The leaderboard currently operates on 16 A100 80GB GPUs. This infrastructure creates a bottleneck for models larger than 30 billion parameters, which often result in prolonged pending states due to high compute requirements.

Looking forward, the organizers aim to evolve the leaderboard to address three primary limitations of current benchmarking:

  1. Data Obsolescence: Moving beyond fixed datasets (like SQUAD and KLEU) to incorporate data that reflects the current moment.
  2. Real-World Application: Developing ways to measure a model's ability to handle edge cases and outliers found in B2B and B2C services.
  3. Overfitting: Reducing the "leaderboard-centric" approach where models are tuned specifically for test sets rather than real-world utility.

The ultimate goal is to bridge the gap between academic research and practical application by incorporating benchmarks that correlate strongly with real-world use cases.

Sources