Parquet Content-Defined Chunking for Efficient Data Deduplication

Parquet Content-Defined Chunking (CDC) is now available in PyArrow and Pandas, enabling efficient deduplication of Parquet files on content-addressable storage systems like Hugging Face's Xet storage layer. By uploading or downloading only the changed data chunks, CDC dramatically reduces data transfer and storage costs for large-scale datasets.

Optimizing Parquet Storage with Xet and CDC

Hugging Face hosts over 4 PB of Parquet files. To optimize this storage, Hugging Face introduced the Xet storage layer, which uses content-defined chunking to deduplicate data chunks and improve transfer speeds.

While Xet is format-agnostic, the standard Apache Parquet layout and column-chunk compression often create entirely different byte-level representations even when minor changes are made to the data. This leads to suboptimal deduplication. Parquet CDC solves this by ensuring that columns are chunked into data pages based on their logical content before serialization or compression occurs, minimizing byte-level differences between similar datasets.

Implementation and Usage

Users can enable Parquet CDC by passing the use_content_defined_chunking=True argument in PyArrow or Pandas:

PyArrow:

import pyarrow.parquet as pq
pq.write_table(table, "hf://datasets/{user}/{repo}/path.parquet", use_content_defined_chunking=True)

Pandas:

import pandas as pd
df.to_parquet("hf://datasets/{user}/{repo}/path.parquet", use_content_defined_chunking=True)

Performance Across Common Data Workflows

Parquet CDC provides significant deduplication benefits across several common data engineering scenarios when used with the Xet storage layer:

Exact Copies and Column Modifications

  • Exact Copies: Uploading identical files to different repositories is instantaneous, as the system recognizes the content is already present.
  • Adding/Removing Columns: Only the new columns and updated footer metadata are transferred. For example, adding new columns to a dataset resulted in only 575kB of new data being uploaded for a 96.6MB file.
  • Changing Column Types: Casting a column (e.g., int64 to int32) only requires uploading the modified column and updated metadata.

Row-Level Operations

  • Appending Rows: Concatenating new rows to a table only transfers the new data, as the original rows remain unchanged in the storage layer.
  • Inserting and Deleting Rows: In vanilla Parquet, inserting or deleting a row shifts all subsequent rows, changing the byte-level representation of all following data pages and forcing a full re-upload. Parquet CDC prevents this by chunking based on content, allowing the system to identify and transfer only the specific changes. In tests, inserting rows into a dataset reduced the transfer from nearly 90MB (vanilla) to 6MB (CDC).

Structural and Sharding Changes

  • Row-Group Sizes: Changing row-group sizes (e.g., from 1M rows to 128k or 256k) typically shifts values between data pages. Parquet CDC maintains deduplication efficiency despite these structural changes.
  • File-Level Splits: When a dataset is split into different numbers of shards (e.g., 5, 10, or 20 shards), Parquet CDC combined with Xet ensures that the overall upload size remains barely larger than the original dataset, regardless of the sharding boundaries.

Limitations and Considerations

Because Parquet CDC operates at the data page (column chunk) level, its effectiveness depends on the selectivity of the changes. If a filter or modification affects the majority of data pages across the dataset, the deduplication ratio will decrease as more chunks are identified as unique.

Sources