Improving Parquet Deduplication on Hugging Face Hub

Hugging Face is optimizing its storage architecture to improve the efficiency of Parquet file deduplication, reducing the storage overhead required when users update large datasets. This effort is critical because Parquet files account for over 2.2PB of the nearly 11PB of datasets hosted on the Hugging Face Hub.

The Challenge of Parquet Deduplication

To minimize storage costs, Hugging Face uses byte-level Content-Defined Chunking (CDC) to deduplicate data. While CDC is effective for insertions and deletions in many file types, the specific layout of Parquet files creates challenges for efficient deduplication during incremental updates.

Parquet tables are organized into row groups, where each column within a row group is compressed and stored. This structure means that changes to the data can have disproportionate effects on the resulting file bytes, impacting how much of the file can be deduplicated against previous versions.

Impact of Data Operations on Deduplication

Hugging Face conducted experiments using a 2GB Parquet file with 1,092,000 rows from the FineWeb dataset to analyze how different operations affect deduplication rates:

Data Appends

Appending 10,000 new rows results in high deduplication efficiency. Because appends only affect the end of the file, the new version is 99.1% deduped, requiring only 20MB of additional storage.

Data Modifications

Modifying a single row (e.g., row 10,000) significantly degrades deduplication. The new file is only 89% deduped, requiring 230MB of additional storage. This occurs because the Parquet file format includes absolute file offsets in the column headers (specifically in ColumnChunk and ColumnMetaData structures), meaning any modification triggers a rewrite of all column headers.

Data Deletions

Deleting a row from the middle of a file typically causes the remaining file to consist of entirely new blocks, breaking deduplication for the second half of the file. This is primarily due to aggressive column compression. While turning off compression improves deduplication, it doubles the file size, creating a trade-off between storage efficiency and deduplication capability.

Proposed Solution: Content-Defined Row Groups

To maintain both compression and deduplication efficiency during deletions or insertions, Hugging Face proposes using Content-Defined Chunking at the row level rather than just the byte level.

Instead of splitting row groups based on a fixed row count (e.g., every 1,000 rows), row groups would be split based on a hash of a provided "Key" column. A row group is split whenever the hash of the key column modulo a target row count equals zero (with minimum and maximum size constraints).

Experimental results show that this method allows for efficient deduplication across compressed Parquet files even when rows are deleted, as only the affected row group and the column headers are rewritten.

Future Optimizations for Parquet Storage

Hugging Face identifies two primary paths for improving Parquet dedupe-ability:

1. Relative Offsets: Replacing absolute offsets with relative offsets in the file structure data to make Parquet structures position-independent. However, this is noted as a complex change to the file format.
2. Content-Defined Row Groups: Implementing support for content-defined chunking on row groups. Since the Parquet format does not require uniformly sized row groups, this can be implemented by updating Parquet format writers with minimal impact on the existing format.

Additionally, Hugging Face is exploring the possibility of optionally rewriting Parquet files before upload to strip absolute file offsets and restoring them upon download.