Hugging Face Data Is Better Together Initiative

Overview

Hugging Face and Argilla, with support from the open-source ML community, established the Data Is Better Together (DIBT) initiative to enable the collective creation of impactful datasets. The project aims to move beyond English-centric data by fostering community-driven data curation and providing the tools necessary for developers and domain experts to build high-quality datasets independently.

Community-Driven Dataset Achievements

The DIBT initiative has focused on two primary community projects to improve prompt quality and multilingual evaluation:

Prompt Ranking Project

The initiative successfully created a dataset of 10,000 prompts, comprising both synthetic and human-generated entries, ranked by quality. This effort involved over 385 participants and resulted in the release of the DIBT/10k_prompts_ranked dataset. This dataset is designed for prompt ranking tasks and synthetic data generation and has already been used to develop new models, including SPIN.

Multilingual Prompt Evaluation Project (MPEP)

To address the lack of language-specific benchmarks for open Large Language Models (LLMs), the MPEP was created to develop a multilingual leaderboard. The project selected a subset of 500 high-quality prompts from the DIBT/10k_prompts_ranked dataset for translation into various languages. This effort has seen the following progress:

  • More than 18 language leaders have established translation spaces.
  • Translations have been completed for languages including Dutch, Russian, and Spanish, with ongoing work for additional languages.
  • A dedicated community of dataset builders has been formed on Discord.

Dataset Creation Guides and Tools

As part of the "cookbook" efforts, DIBT provides guides and tools to help the community build specialized datasets independently. These resources focus on three key areas:

  • Domain-Specific Datasets: Resources to help engineers and domain experts collaborate to bootstrap the creation of datasets tailored to specific fields.
  • DPO/ORPO Datasets: Guidance for building Direct Preference Optimization (DPO) style datasets across different languages, domains, and tasks.
  • KTO Datasets: Tools and instructions to help the community create Kahneman-Tversky Optimization (KTO) datasets.

Key Insights and Lessons Learned

Through the DIBT initiative, Hugging Face and Argilla have identified several critical findings regarding open-source data collaboration:

  • Community Demand: There is significant excitement and a strong willingness among the community to participate in collective dataset construction.
  • Data Inequality: Existing inequalities persist, meaning certain languages, domains, and tasks remain underrepresented in open-source benchmarks, necessitating a more inclusive approach to ensure comprehensive evaluation.
  • Tooling Availability: The necessary tools for effective community collaboration on dataset building already exist and can be leveraged to the community's community's scale.

How to Contribute

Contributions to the DIBT initiative are ongoing. Users can participate by following the README instructions for specific cookbook projects, sharing their datasets and results, or contributing new guides and tools. Coordination for these efforts takes place in the #data-is-better-together channel of the Hugging Face Discord server.

Sources