Hugging Face Xet Storage Integration

Hugging Face has integrated Xet storage into the Hub, migrating the first set of Model and Dataset repositories away from Git LFS. This transition introduces content-defined chunking (CDC) to enable byte-level deduplication, drastically reducing the time and bandwidth required to update large files.

The Xet Storage Architecture

Xet storage replaces file-level deduplication with byte-level deduplication using content-defined chunking (CDC). While Git LFS treats any change to a large file as a requirement to upload the entire file again, Xet splits data into ~64KB chunks. Only the chunks that have changed are transmitted over the wire.

For example, appending 1MB of data to a 5GB SQLite database requires a full 5GB re-upload under LFS, but with Xet, only the new data is pushed. In internal testing, this reduced upload times from 13 minutes to a tenth of a second at 50Mb/s.

System Components

  • Xet-aware Client: Handles the splitting of data into ~64KB chunks, deduplicates identical chunks locally, and aggregates them into ~64MB blocks before uploading.
  • Hugging Face Hub: Manages routing, authentication, and security guarantees.
  • Content Addressed Store (CAS): Enforces chunk-based deduplication for transfers. It includes an LFS Bridge to ensure backward compatibility for non-Xet clients by acting as a traditional LFS server.
  • Amazon S3: Serves as the final persistence layer, storing blocks (file contents) and shards (reconstruction metadata).

Production Migration and Validation

On February 20, 2025, Hugging Face migrated 4.5 TB of data across target repositories to Xet storage. This migration shifted approximately 6% of the Hub’s total download traffic to the Xet infrastructure, providing a real-world proving ground for the system’s reliability and performance.

Technical Challenges and Optimizations

Post-migration analysis revealed two primary technical bottlenecks that were resolved through architectural updates:

Download Overhead and Block Format

Initial metrics showed the Content Addressed Store (CAS) was downloading four times more data from S3 than it was returning to clients. This was caused by hf_transfer requests for 10MB ranges that did not align with Xet’s block boundaries. Because the block format lacked uncompressed chunk lengths, CAS had to stream entire blocks from the beginning to find the requested data.

Solution: Hugging Face updated the block format to store chunk-length metadata. This allowed CAS to download only the specific data required for each request, resulting in a ~35% reduction in GET latency and a balanced download-to-sent data ratio.

Pod Load Imbalance

The team observed unexpected load spikes where single pods in the CAS cluster would handle hundreds of active uploads while others remained idle. This was traced to the OS page cache buffering writes to temporary files during validation without calling fsync. High upload volume led to memory pressure and throughput throttling from AWS EBS block storage, creating a cycle of latency and backlogs.

Solution: The team implemented a limit on the number of concurrent uploads each machine accepts. Once a pod reaches its limit, requests are pushed to other pods in the cluster, triggering autoscaling policies if all pods are saturated.

Implementation for Users

Users can access Xet-backed storage by joining a waitlist. Once accepted, new repositories will automatically use Xet, and existing repositories will be migrated.

  • Tooling: An hf_xet Python package is being integrated into huggingface_hub. Users of transformers or datasets can install hf_xet in their environment to leverage the benefits.
  • Compatibility: Legacy clients remain compatible via the LFS Bridge, though upgrading to hf_xet is required to achieve full performance gains.

Sources