Hugging Face Xet-backed Repositories: Accelerating Hub Transfers with Block-Level Aggregation

Hugging Face is implementing a new chunk-based deduplication system, powered by xet-core and hf_xet, to accelerate file uploads and downloads on the Hub by 2-3x in some cases. This system moves away from traditional file-centric transfers to a content-addressed storage (CAS) model that optimizes how data is moved and stored to support rapid iteration for AI builders.

Scaling Deduplication via Block-Level Aggregation

Purely chunk-based deduplication is not viable at the scale of the Hugging Face Hub—which hosts nearly 45PB across 2 million repositories—because it would create billions of individual entries in the CAS. A naive approach using only chunks (averaging ~64KB each) would result in approximately 690 billion chunks, leading to unsustainable network overhead from millions of individual requests and prohibitive infrastructure costs for metadata management in services like S3 or DynamoDB.

To solve this, Hugging Face employs aggregation, ensuring that communication and storage strategies do not scale 1:1 with the number of chunks:

  • Blocks: After deduplication, data is bundled into blocks of up to 64MB. These blocks remain content-addressed, reducing the total number of CAS entries by a factor of 1,000.
  • Shards: Shards map files to chunks and reference the blocks containing them. This allows the system to identify specific changed parts of a file and skip transfers for chunks already present in the CAS.

Optimizing Transfers with Key Chunks and Spatial Locality

To avoid performing a network query for every single chunk during upload, Hugging Face uses key chunks. A key chunk is a 0.1% subset of all chunks, selected via a modulo condition based on the chunk hash.

By maintaining a global index of these key chunks and the shards they reside in, the system can retrieve a related shard when a key chunk is queried. This leverages spatial locality, as chunks referenced in the same shard are likely to be similar. This mechanism significantly reduces the number of network and database requests required to verify if a chunk has already been uploaded.

Performance Impact on Quantized Models

Quantized models (such as .gguf files) are primary beneficiaries of this system due to the nature of quantization, which restricts weight matrix values to smaller integer ranges, creating high repetition and overlap across different quantization variants (e.g., Q4_K, Q3_K, Q5_K).

In a practical test using the bartowski/gemma-2-9b-it-GGUF repository, which contains 29 quantizations totaling 191GB, the Xet-backed approach demonstrated significant efficiency gains:

  • Storage Reduction: The repository was reduced from 191GB to approximately 97GB (a savings of ~94GB) by storing only 1,515 unique blocks.
  • Upload Speed: At a transfer rate of 50MB/s, upload time dropped from 509 minutes to 258 minutes, representing a nearly 2x speedup.
  • Download Efficiency: Local chunk caching ensures that only unchanged chunks are downloaded when a file is updated or a new quantization is added, unlike file-based approaches that require downloading the entire file.

Availability

Hugging Face is rolling out the first Xet-backed repositories in the coming weeks and months, with the goal of making file transfers "invisible" for all builders on the Hub.

Sources