FilBench: Evaluating LLM Capabilities in Philippine Languages

Hugging Face has introduced FilBench, a comprehensive evaluation suite designed to assess the fluency, linguistic abilities, and cultural knowledge of LLMs in Tagalog, Filipino, and Cebuano. This benchmark provides a systematic way to move beyond anecdotal evidence of LLM fluency in Philippine languages and establish a data-driven understanding of model performance.

FilBench Framework and Methodology

FilBench is a structured evaluation suite consisting of 12 tasks divided into four primary categories. To ensure the benchmark reflects actual NLP research and usage trends, the categories were curated based on a historical survey of Philippine language NLP research spanning from 2006 to early 2024. Most categories utilize non-translated content to maintain faithfulness to the natural use of the languages.

Evaluation Categories

  • Cultural Knowledge: Tests the ability to recall factual and culturally specific information. This includes regional knowledge (Global-MMLU), Filipino-centric values (KALAHI), and word sense disambiguation (StingrayBench).
  • Classical NLP: Covers information extraction and linguistic tasks traditionally handled by specialized models, including named entity recognition (CebuaNER, TLUnified-NER, and Universal NER), text categorization, and sentiment analysis (subsets of SIB-200 and BalitaNLP).
  • Reading Comprehension: Evaluates the interpretation of Filipino text through readability, comprehension, and natural language inference (Cebuano Readability Corpus, Belebele, and NewsPH NLI).
  • Generation: Focuses on translation faithfulness, specifically from English to Filipino and Cebuano to English, using datasets such as NTREX-128, Tatoeba, and TICO-19.

Implementation and Scoring

FilBench is built on top of the Lighteval framework. The researchers defined specific translation pairs for common evaluation terms (e.g., "yes" to oo, "no" to hindi) and implemented custom tasks using provided templates. The final result is expressed as the FilBench Score, a weighted average based on the number of examples in each category.

Key Findings from LLM Evaluations

Evaluating over 20 state-of-the-art LLMs revealed three primary insights regarding their performance in Philippine languages.

Region-Specific Models vs. General Models

Southeast Asian (SEA)-specific LLMs, such as SEA-LION and SeaLLM, are the most parameter-efficient models for Philippine languages, achieving higher FilBench scores than other models of similar size. However, they are still outperformed by closed-source models like GPT-4o.

Despite this gap, the research indicates that continuous fine-tuning of base LLMs with SEA-specific instruction-tuning data yields performance gains of 2-3%, suggesting that the curation of Filipino and SEA-specific training data remains a highly effective direction for model improvement.

Challenges in Generation and Translation

Generation is the most difficult category across the evaluated models. Common failure modes in translation tasks include:

  • Failure to follow translation instructions.
  • Generation of overly verbose text.
  • Hallucination of languages other than Tagalog or Cebuano.

Cost-Efficiency of Open-Weight Models

Open-weight LLMs available on Hugging Face provide a cost-effective alternative to commercial models without significant performance sacrifices. For users seeking an alternative to GPT-4o for Filipino language tasks, the researchers highlight Llama 4 Maverick as a strong option on the efficiency Pareto frontier.

Resources and Access

FilBench is available as a set of community tasks in the official Lighteval repository. Developers can access the full research paper via arXiv (2508.03523) and the evaluation code via GitHub.

Sources