Hugging Face and Argilla Enable Collective Community Dataset Building

Hugging Face and Argilla have launched an initiative to enable open-source communities to collectively build high-quality datasets, reducing the barrier to entry for non-technical contributors. This effort aims to address the lack of high-quality training, evaluation, and benchmarking data for many languages, domains, and specific tasks.

Collective Dataset Building via Argilla and Hugging Face Spaces

Collective dataset building allows individuals without machine learning or programming skills to contribute directly to the development of Open Source ML. By leveraging a streamlined technical workflow, communities can collaborate on various data types, including:

  • Language-specific chat datasets: Creating data for underrepresented languages.
  • Domain-specific benchmarks: Developing evaluation sets for specialized fields.
  • Preference datasets: Gathering rankings from a diverse range of participants.
  • Task-specific datasets: Building data tailored to particular machine learning tasks.

Technical Implementation and Workflow

The primary technical barrier to collective data annotation has historically been the difficulty of setting up efficient annotation tasks. This is solved by combining Argilla—an open-source tool for creating LLM and task-specific datasets—with Hugging Face Spaces, a platform for hosting ML demos and applications.

Argilla now supports authentication via Hugging Face accounts for instances hosted on Spaces. This integration allows users to begin contributing to annotation tasks in seconds, significantly lowering the friction for community participation.

Proof of Concept: The 10k_prompts_ranked Dataset

To test this workflow, Hugging Face and Argilla conducted an experiment called "Data is Better Together," which focused on building a preference dataset of prompt rankings. The results demonstrated the efficiency of the community-driven approach:

  • Contributor Volume: 350 community contributors participated in labeling data.
  • Data Volume: Over 11,000 prompt ratings were collected in a few days.
  • Outcome: The release of the 10k_prompts_ranked dataset, containing 10,000 prompts with user ratings for prompt quality.

Community Cohort Support

Hugging Face and Argilla are recruiting an initial cohort of community dataset builders to expand this model. Support for participants in this cohort includes:

  • Infrastructure: Assistance in creating an Argilla Space with Hugging Face authentication, including free persistent storage and improved CPU spaces provided by Hugging Face.
  • Visibility: Amplification of communications and promotional efforts by both Argilla and Hugging Face.
  • Collaboration: Access to a dedicated cohort community channel.

Project Scope and Limitations

The initiative is primarily focused on text-based datasets, particularly those targeting languages, domains, and tasks that are currently underrepresented in the open-source ecosystem. While the program is open to multimodal dataset ideas, support for them may be limited during the first cohort. Annotation tasks can be configured as either fully open to the public or restricted to members of a specific Hugging Face Hub organization.

Sources