Hugging Face Hub Migration from Git LFS to Xet

Hugging Face has migrated 500,000 repositories and 20 PB of data from Git LFS to Xet, a new storage backend designed to scale with the massive data requirements of AI builders. This transition enables faster transfers and better scalability without requiring users to change their existing workflows.

Seamless Backward Compatibility via the Git LFS Bridge

To avoid a "hard cut-over" and minimize user disruption, Hugging Face implemented a system where Xet-enabled repositories can contain both Xet and LFS files. The core of this compatibility is the Git LFS Bridge, which ensures that users with older clients can still interact with Xet-backed files.

How the Git LFS Bridge Works

When a non-Xet-aware client (such as older versions of huggingface_hub or huggingface.js) requests a file via the resolve endpoint, the Git LFS Bridge:

  1. Constructs and returns a single presigned URL that mimics the LFS protocol.
  2. Reconstructs the file from content held in S3.
  3. Returns the reconstructed file to the requester.

Xet-Aware Client Workflow

Clients that are Xet-aware (e.g., hf-xet or the Xet integration in huggingface_hub) bypass the bridge and use the full Xet stack:

  • Uploads: Files are broken into chunks using content-defined chunking, passed to the Content Addressed Store (CAS), and stored in S3.
  • Downloads: The client requests file reconstruction information and retrieves specific chunk ranges from S3 via the CAS.

Background Migration Process

Hugging Face utilizes a background migration process to move data from LFS to Xet without requiring repository locks or disrupting active uploads and downloads.

When a file is migrated, a webhook triggers a distributed queue processed by an orchestrator. The orchestrator enables Xet on the repository, fetches LFS revisions, and batches files into jobs (either 1,000 files or 500MB). Migration worker pods then download the LFS files and upload them to the Xet CAS using xet-core.

Scaling Challenges and Performance Gains

During large-scale migrations of power users—including those with up to 6.1 PB of data across 42,000 repositories—Hugging Face identified and resolved several technical bottlenecks:

  • Disk Space: Fixed issues where temporary shard files for global deduplication were written to /tmp on a different mount point than the Xet cache, leading to No space left on device errors.
  • Resource Allocation: Resized the CAS to handle bursty uploads from migration workers, which differed from the typical bursty download patterns seen during major model releases like Llama 4.
  • I/O Bottlenecks: Upgraded worker node specifications to resolve network and EBS I/O bottlenecks.

CAS Throughput Improvements

As a result of these optimizations, Content Addressed Store (CAS) throughput increased significantly:

  • Initial migrations: Sustained ~35 Gb/s (with ~5 Gb/s of regular Hub traffic).
  • Latest migrations: Peaked at ~300 Gb/s while serving ~40 Gb/s of baseline load.

Future Roadmap and Availability

As of July 2025, Xet is the default for new users and organizations. Hugging Face is now expanding Xet access to all users, which includes the automatic migration of all existing repositories from LFS to Xet.

Upcoming developments include:

  • Chunk-based support for browser-based uploads/downloads and Git.
  • Open-sourcing the Xet protocol and the entire infrastructure stack.

Sources