Hugging Face Hub and DuckDB Integration for Dataset Analysis

Hugging Face has integrated DuckDB to enable users to execute SQL queries directly on public datasets stored on the Hugging Face Hub. This integration allows for fast, analytical querying of massive datasets without the need to download them locally, leveraging the Hub's automatic conversion of datasets to the Parquet columnar format.

Automatic Parquet Conversion for Public Datasets

The Hugging Face dataset viewer automatically converts all public datasets on the Hub into Parquet files. Parquet is a columnar storage format that is significantly more efficient for storing, loading, and analyzing data, which is critical for the large-scale datasets used in the current era of Large Language Models (LLMs).

Users can identify these files by clicking the "Auto-converted to Parquet" button on a dataset page. The specific URLs for these Parquet files can be retrieved programmatically via the /parquet endpoint of the datasets-server.

Analytical Querying with DuckDB

DuckDB is utilized as the database management system (DBMS) for this integration because of its high performance in executing complex analytical queries. A key technical capability of this setup is DuckDB's ability to execute SQL queries directly on remote Parquet files without overhead.

By using the httpfs extension, DuckDB can read and write remote files using the URLs provided by the Hugging Face /parquet endpoint. This is particularly useful for large datasets, as the dataset viewer shards big datasets into smaller 500MB chunks, and DuckDB supports querying across multiple Parquet files simultaneously.

Implementation Example

To analyze a dataset, users can follow these steps:

  1. Retrieve Parquet URLs: Use an HTTP call to the /parquet endpoint to get the list of Parquet file URLs for a specific dataset.
  2. Initialize DuckDB: Create a connection and load the httpfs extension to enable remote file access.
  3. Execute SQL: Run standard SQL queries directly against the remote URL.
import duckdb

# Example URL retrieved from the /parquet endpoint
url = "https://huggingface.co/datasets/blog_authorship_corpus/resolve/refs%2Fconvert%2Fparquet/blog_authorship_corpus/blog_authorship_corpus-train-00000-of-00002.parquet"

con = duckdb.connect()
con.execute("INSTALL httpfs;")
con.execute("LOAD httpfs;")

# Querying the remote Parquet file
con.sql(f"""SELECT horoscope, 
    count(*), 
    AVG(LENGTH(text)) AS avg_blog_length 
    FROM '{url}' 
    GROUP BY horoscope 
    ORDER BY avg_blog_length 
    DESC LIMIT(5)""")

Implications for Dataset Transparency

Providing SQL access to Hub datasets enables researchers and developers to better understand the contents of the datasets used to train models. Because dataset composition directly impacts model quality, this tool provides a mechanism for open access and increased awareness of dataset contents, allowing users to uncover insights through structured querying.

Sources