Hugging Face Dataset Hub Search Features Update

Hugging Face has launched four new search filters for the Dataset Hub to improve how researchers and engineers discover and explore the more than 180,000 public datasets available on the platform. These updates address the challenges of dataset discoverability and visualization, enabling users to find data that fits specific technical requirements for training and evaluating AI models.

Filter by Modality

Users can now filter datasets based on the type of data they contain. Modalities are automatically detected by the Hub based on file extensions and contents. The supported modality filters include:

  • Text
  • Image
  • Audio
  • Tabular
  • Time-Series
  • 3D
  • Video
  • Geospatial

These filters support multi-selection, allowing users to find datasets that contain multiple modalities, such as combined text and image data.

Filter by Dataset Size

The Dataset Hub now allows users to search for datasets by specifying a minimum and maximum number of rows. This functionality enables the discovery of everything from small-scale datasets to massive datasets used for LLM pretraining.

Row counts are available for all datasets in supported formats. For datasets where the number of rows is not explicitly included in the metadata, Hugging Face estimates the total row count based on the content of the first 5GB of the dataset.

Filter by Data Format

Users can now filter datasets by their storage format, which is critical for determining whether data needs to be reformatted for a specific use case. The Hub highlights that different formats offer different trade-offs:

  • Parquet: Provides efficient filtering, analytics, and a high compression ratio, though accessing a single row requires decoding a full row group.
  • WebDataset: Optimized for high data streaming speeds, though it lacks certain metadata like rows per file, which can impact multi-node training distribution.
  • Other formats: Common formats include JSON Lines, CSV, and raw text files.

Filter by Library Compatibility

The Dataset Hub now includes filters for libraries and tools used to load and prepare data, such as Pandas, Dask, and the ‚‚ Datasets library. Compatibility is determined by the dataset's format and size (e.g., Dask is listed for large JSON Lines datasets that would exceed Pandas' memory limits).

To streamline integration, Hugging Face provides code snippets for loading datasets using these supported tools.

Combining Search Filters

All four new filters—modality, size, format, and library—can be combined with each other and with existing filters such as Language, Tasks, and Licenses. When used alongside the text search bar, these combined filters allow for highly specific dataset discovery.

Sources