Huggy Lingo: Improving Hugging Face Hub Language Metadata with ML

Hugging Face is employing machine learning to automatically detect the language of datasets on the Hugging Face Hub that lack language metadata, using Librarian-Bots to submit pull requests for these updates. This initiative aims to improve dataset discoverability and help the community identify gaps in language representation across the Hub.

The Challenge of Missing Language Metadata

Language metadata is critical for users to filter and find relevant datasets for specific use cases, such as training open-source Large Language Models (LLMs) for under-represented languages. However, a significant portion of the Hub's datasets lack this information.

  • Current State: Out of approximately 50,000 public datasets, only about 13% specify language metadata in their YAML headers. Roughly 87% of datasets do not provide this information.
  • Language Distribution: English (en) is the most common language, representing about 19% of datasets that have specified their language. When excluding English, there is a grouping of a few dominant languages followed by a smooth decline in frequency.
  • Metadata Inconsistency: Existing language tags are often inconsistent, with some datasets using variations like en, eng, english, or English to describe the same language.

Machine Learning Workflow for Language Prediction

To address the missing metadata, Hugging Face implemented a pipeline to predict the language of datasets using a combination of API access and machine learning models.

Data Acquisition via Dataset Viewer API

To avoid downloading entire datasets locally, Hugging Face utilizes the dataset viewer API. This allows the system to sample text data without full downloads. The process involves:

  1. Column Filtering: The system identifies columns likely to contain text (e.g., columns named text or prompt with string features) and ignores non-textual columns (e.g., image).
  2. Sampling: The system retrieves 20 rows of text data from these identified columns to be passed to the prediction model.

Language Identification Model

For the prediction task, Hugging Face uses the facebook/fasttext-language-identification model developed by Meta as part of the No Language Left Behind project. This model is capable of detecting 217 languages.

Prediction Filtering and Validation

To ensure the accuracy of the metadata suggestions, the system applies several filters to the 20 individual row predictions:

  • Frequency Filter: If a language is predicted for less than 20% of the sampled rows, that prediction is discarded.
  • Confidence Filter: Predictions are only accepted if the mean score for a language is 80% or higher.
  • Language Code Mapping: The model returns ISO 639-3 codes (e.g., kor_Hang for Korean). The system strips the script information and converts these to ISO 639-1 codes (which have better Hub UI support) wherever possible. In cases where no ISO 639-1 equivalent exists, a manual mapping is used (e.g., Standard Arabic arb to Arabic ar). If no mapping is possible, no metadata is suggested.

Automated Metadata Updates via Librarian-Bot

Once a language is predicted and validated, the information is integrated back into the Hub via Librarian-Bot.

Librarian-Bot automatically opens pull requests to add the predicted language metadata to the dataset cards. This allows dataset owners to review and approve the merge, ensuring that the metadata is updated efficiently without requiring manual human labor for every dataset. Progress can be tracked via the Librarian-Bot activity feed.

Sources