Hugging Face and LLM-jp Launch Open Japanese LLM Leaderboard

Hugging Face and the LLM-jp research consortium have launched the Open Japanese LLM Leaderboard to provide a centralized, transparent system for evaluating the performance of large language models (LLMs) in Japanese. This initiative addresses the lack of an open benchmarking standard for a language that presents unique linguistic challenges, including a complex writing system and the absence of word boundaries.

Comprehensive Evaluation Suite

The Open Japanese LLM Leaderboard utilizes the llm-jp-eval library to assess models across 16 distinct tasks, ranging from classical NLP to modern reasoning. These tasks are executed using 4-shot prompting. The evaluation suite includes datasets developed from scratch by linguists and experts, as well as automatically translated datasets adjusted for Japanese cultural and linguistic specificities.

Key Evaluation Datasets

  • Jamp: A temporal inference benchmark for Natural Language Inference (NLI), testing entailment, neutrality, and contradiction.
  • JEMHopQA: A multi-hop question-answering dataset designed to evaluate internal reasoning and derivation generation.
  • jcommonsenseqa: A Japanese version of CommonsenseQA used to evaluate general commonsense reasoning.
  • chABSA: An Aspect-Based Sentiment Analysis dataset based on 2016 fiscal year financial reports from Japanese listed companies, following the Financial Service Agency (FSA) taxonomy.
  • mbpp-ja: A programming dataset translated from the Mostly Basic Python Problems (MBPP) dataset via DeepL.
  • mawps: A mathematical evaluation dataset focusing on step-by-step Chain-of-Thought (CoT) reasoning for basic arithmetic and equations, adjusted for Japanese contexts.
  • JMMLU: A knowledge dataset based on a portion of MMLU, adjusted for Japanese civics, geography, and idioms.
  • XL-Sum: An abstractive summarization dataset utilizing Japanese translations of BBC News articles.

Technical Implementation

The leaderboard's architecture is inspired by the Open LLM Leaderboard. The technical stack includes:

  • Deployment: Automatic deployment via Hugging Face Inference Endpoints.
  • Evaluation Framework: The llm-jp-eval library (version 1.14.1).
  • Inference Engine: vLLM (version v0.6.3) for memory-efficient serving.
  • Compute Backend: The mdx premium research computer platform in Japan.

Performance Observations

Analysis of the leaderboard reveals several trends in the Japanese LLM ecosystem:

  • Architecture Preferences: Meta's Llama architecture is widely favored by Japanese AI labs, though Mistral and Qwen architectures have also achieved top scores.
  • Open vs. Closed Source: Open-source architectures are narrowing the gap with closed-source models. For example, the llm-jp-3-13b-instruct model, developed by LLM-jp, shows performance comparable to closed-source alternatives.
  • Remaining Challenges: Most LLMs continue to struggle with domain-specific tasks, including finance (chABSA), linguistic annotations (Wikipedia Annotated Corpus), code generation (mbpp-ja), and summarization (XL-Sum).
  • Cultural Alignment: Models developed by Japanese companies or laboratories perform better on the JCommonsenseMorality dataset, which tests the ability to make ethical choices aligned with Japanese values.

Future Roadmap

The evaluation framework will evolve alongside the llm-jp-eval tool. Planned enhancements include:

  • Expanded Datasets: Integration of JHumanEval (a Japanese version of HumanEval) and MMLU.
  • Advanced Evaluation: Implementation of Chain-of-Thought (CoT) evaluation to compare performance against basic prompts.
  • New Metrics: Introduction of the "Out-of-Choice rate" to measure how frequently models predict tokens outside the provided labels, serving as a proxy for instruction-following capability.

Sources