Hugging Face Introduces Content-Defined Chunking to Improve Storage Efficiency

Content-Defined Chunking Foundations

Content-defined chunking (CDC) breaks files into variable-sized chunks based on the data itself, enabling deduplication and fine-grained updates. A rolling hash algorithm scans the byte sequence; when a hash meets a predefined condition, a chunk boundary is placed. Identical chunks produce the same hash, so only one copy is stored in a content-addressed store, providing built-in deduplication.

Handling Insertions and Deletions

When a file changes, CDC allows only the modified chunks to be uploaded and stored, because unchanged chunks already exist in the store. For example, inserting the string "super" into a repeated "transformers" sequence results in just one new chunk needing to be saved, while the two existing "transformers" chunks are reused.

Real-World Benchmark (CORD-19 Dataset)

Using CDC-backed storage (Xet) reduced storage and transfer costs compared to Git LFS for the CORD-19 dataset with 50 incremental updates. The average download time dropped from 51 minutes to 19 minutes, the average upload time dropped from 47 minutes to 24 minutes, and storage used fell from 8.9 GB to 3.52 GB, representing a consistent ~50% improvement across metrics.

Implications for the Hugging Face Hub

Applying CDC to Hub files shows significant storage savings. For two versions of the model.safetensors file in the openai-community/gpt2 repository, Git LFS would require 1.2 GB total, while Xet-backed storage would require 645 MB, a 53% reduction, with an estimated additional 10% saving from compression. Across the Hub, fine-tuned models and model checkpoints exhibit deduplication ratios between 30% and 85%; PyTorch model checkpoints alone account for about 200 TB, so a 50% deduplication could save up to 100 TB immediately and roughly 7-8 TB per month thereafter. Chunk-level deduplication also improves upload/download speeds because only modified chunks are transferred, reducing user and machine waiting time.

Future Work

The Xet team is completing a proof‑of‑concept of Xet-backed storage for the Hub and plans to roll out some Xet-backed repositories in early 2025. Future work will focus on scaling CDC across globally distributed repositories, balancing network performance, managing privacy boundaries, and parallelizing the chunking algorithm.