Hugging Face acquires XetHub to upgrade Hub storage and collaboration

TL;DR

Hugging Face announced the acquisition of XetHub, a Seattle‑based startup that built Git‑scale storage and versioning for terabyte‑size AI datasets and models, enabling more efficient collaboration and paving the way for trillion‑parameter model support on the Hub.

Our common goal at Hugging Face

"The XetHub team will help us unlock the next 5 years of growth of HF datasets and models by switching to our own, better version of LFS as storage backend for the Hub's repos." – Julien Chaumond, HF CTO

Hugging Face built the original Hub on Git LFS in 2020 because it was a pragmatic choice for early growth. The company now plans to replace Git LFS with XetHub’s optimized storage and versioning system, which is designed for the very large files typical in modern AI workflows.

Example future use cases 🔥

  • Incremental updates to large Parquet files – Adding a single row to a 10 GB Parquet file currently requires re‑uploading the entire file. XetHub’s chunked storage and deduplication will allow only the affected chunks to be uploaded, dramatically reducing bandwidth and latency.
  • Fine‑grained updates to GGUF model headers – Updating a single metadata field in a Llama 3.1 405B GGUF repository will later require uploading only a few kilobytes instead of the full multi‑gigabyte model file.
  • Scaling to trillion‑parameter models – As models like the newly released BigLlama‑3.1‑1T emerge, XetHub’s technology is expected to enable both community and enterprise users to store, version, and collaborate on datasets and models at unprecedented scale.
  • Improved collaboration on large assets – XetHub will provide tooling to help teams understand dataset and model evolution, supporting reproducibility and coordinated experimentation.

Current Hub statistics 🤯

  • Repositories: 1.3 M models, 450 k datasets, 680 k spaces
  • Storage: 12 PB in LFS (≈280 M files) and 7.3 TB in Git (non‑LFS)
  • Traffic: 1 B daily requests and 6 PB daily CloudFront bandwidth

These numbers illustrate the scale at which the Hub currently operates and the pressure on storage back‑ends.

Founder’s perspective

"I have been part of the AI/ML world for over 15 years… XetHub’s goal is to enable ML teams to operate like software teams, by scaling Git file storage to TBs, seamlessly enabling experimentation and reproducibility, and providing the visualization capabilities to understand how datasets and models evolve." – Yucheng Low (@ylow), XetHub co‑founder

Low recounts his experience scaling data pipelines at Apple (over 100 PB) and founding XetHub to bring those lessons to the broader AI community. The acquisition aligns XetHub’s mission with Hugging Face’s goal of simplifying AI collaboration at massive scale.

Hiring notice

The Hugging Face Infrastructure team is actively recruiting engineers who want to build and scale the open‑source AI collaboration platform. Interested candidates are encouraged to apply.

Follow the XetHub team on Hugging Face at hf.co/xet-team.

Sources