Databricks and Hugging Face Integrate Apache Spark for Faster LLM Training

Databricks and Hugging Face have integrated Apache Spark directly into the Hugging Face codebase, allowing users to create Hugging Face Datasets from Apache Spark dataframes. This integration reduces the time required to move data from Spark to Hugging Face datasets by up to 40%, streamlining the pipeline for training and fine-tuning Large Language Models (LLMs).

Direct Spark Integration via Dataset.from_spark()

The previous workflow for moving data from an Apache Spark dataframe to a Hugging Face dataset required writing data to Parquet files on disk and then reloading those files into the dataset. This process was inefficient because it required writing to disk, reading back from disk, and rematerializing the data, which increased resource consumption, time, and cost.

With the new update, users can now use the Dataset.from_spark() function. This allows for a direct mapping of a Spark dataframe or Delta table into a Hugging Face dataset, bypassing the need for intermediate Parquet files.

Performance Benchmarks

In a test using a 16GB dataset, the processing time was reduced from 22 minutes (using the Parquet method) to 12 minutes (using Dataset.from_spark()), representing a reduction in processing time of more than 40%.

Strategic Importance for Domain-Specific AI

Integrating Spark with Hugging Face is critical for organizations that need to perform complex data transformations on massive datasets to augment AI models for specific domains. By combining the cost-effectiveness and performance of Spark's data processing engine with the memory-mapping and smart caching optimizations of Hugging Face datasets, organizations can more efficiently prepare and utilize their proprietary data for model tuning.

Broader Open-Source AI Ecosystem Contributions

This integration is the first of several planned contributions from Databricks to the Hugging Face codebase, with future plans to add streaming support through Spark to further accelerate dataset loading.

Beyond the Hugging Face integration, Databricks has implemented several other AI-focused open-source improvements:

  • MLflow: Added support for the transformers library, OpenAI integration, and Langchain support.
  • Databricks SQL: Introduced AI Functions to integrate OpenAI and other deployed models into SQL queries.
  • PyTorch Distributor: Released a PyTorch distributor for Spark to simplify distributed PyTorch training on the Databricks platform.

Sources