Scaling AI Data Processing with Hugging Face and Dask

Hugging Face and Dask provide a scalable framework for processing massive AI datasets that exceed local memory limits. By combining Dask's distributed computing capabilities with Hugging Face's transformers and datasets, users can scale AI tasks—such as model inference and data filtering—from a few rows on a laptop to hundreds of millions of rows across a multi-GPU cloud cluster.

Distributed Data Processing with Dask

Dask allows for out-of-core computing, enabling the processing of datasets that are too large to fit in system memory by breaking them into manageable chunks. For users familiar with pandas, Dask DataFrame provides a similar API, which simplifies the transition from local prototyping to large-scale production.

Key benefits of using Dask for AI data processing include:

  • Efficient Loading: Dask works natively with Parquet, the default format for Hugging Face datasets, allowing for efficient columnar filtering and compression.
  • Parallel Execution: The map_partitions function allows users to apply custom functions (such as model inference) in parallel across each pandas DataFrame partition within a larger Dask DataFrame.
  • Distributed Writing: Dask supports writing results back to Parquet format in parallel, which can be integrated with Hugging Face dataset repositories.

Scaling Model Inference: From Pandas to Dask

To demonstrate scaling, Hugging Face used the FineWeb dataset—consisting of 15 trillion tokens of English web data—and the FineWeb-Edu classifier to identify high-educational-value web pages.

Local Prototyping with Pandas

On a small scale (e.g., 100 rows), pandas can be used to run the FineWeb-Edu classifier. On an M1 Mac with a GPU, this process takes approximately 10 seconds. The workflow involves using a Hugging Face pipeline for text classification, where the hardware device (CUDA, MPS, or CPU) is dynamically selected within the function to ensure compatibility when the code is later distributed.

Scaling to 211 Million Rows

When scaling to a dataset of 211 million rows (part of a 432 GB on-disk crawl), serial processing becomes prohibitively slow. By switching to a Dask DataFrame and loading the data lazily from Hugging Face, the task is parallelized.

In this scaled workflow, the compute_scores function is applied via map_partitions. To optimize performance, the batch_size in the Hugging Face pipeline is increased (e.g., to 768) to better utilize GPU hardware.

Multi-GPU Parallel Inference in the Cloud

For maximum throughput, Dask can be deployed on cloud infrastructure. In the provided example, Coiled was used to provision a cluster of 100 workers using AWS g5.xlarge instances (NVIDIA A10 Tensor Core GPUs).

Infrastructure Automation

Coiled automates several critical deployment steps:

  • VM Provisioning: Automatically spins up GPU-enabled cloud VMs.
  • Environment Setup: Handles the installation of NVIDIA drivers and the CUDA runtime.
  • Package Synchronization: Syncs local Python packages and files to the cloud workers to ensure environment consistency.

Performance and Utilization

The processing of 211 million rows took approximately 5 hours. Monitoring showed that the hardware was utilized efficiently, with median GPU utilization at 100% and median memory usage at 21.5 GB out of the available 24 GB on the GPUs.

Potential AI Use Cases

This distributed processing pattern can be applied to various large-scale AI tasks beyond text classification, including:

  • Genomic Data Filtering: Selecting specific genes of interest from massive genomic datasets.
  • Structured Data Extraction: Converting unstructured text into structured datasets using LLMs.
  • Web Data Cleaning: Cleaning and filtering large-scale scrapes from Common Crawl.
  • Multimodal Inference: Analyzing large-scale audio, image, or video datasets using multimodal models.

Sources