Hugging Face Hub Storage Rearchitecture

Hugging Face is rearchitecting the Hub’s storage backend to replace opaque blob transfers with a content-addressed store (CAS). This shift allows the platform to analyze files at the byte level, enabling deduplication and compression that significantly improve the transfer of massive model weights and datasets.

Transition to Content-Addressed Storage (CAS)

Hugging Face is implementing a custom protocol based on the philosophy of "dumb reads and smart writes" to overcome the 50GB file size limit of AWS CloudFront and the inefficiencies of treating files as opaque blobs.

The Read Path (Dumb Reads)

To maintain high throughput and low latency, the read path is designed for simplicity. Requests for files are routed to a CAS server, which provides the necessary reconstruction information. The actual data remains stored in an S3 bucket in us-east-1 and continues to be served via AWS CloudFront as the CDN.

The Write Path (Smart Writes)

The write path is optimized for speed and security by operating on chunks rather than whole files. When a user uploads data, the CAS server identifies existing chunks and instructs the client (such as huggingface_hub) to transfer only the new, unique chunks. These chunks are validated by the CAS before being uploaded to the S3 backing store.

Technical Optimizations and Capabilities

By managing files at the byte level, Hugging Face can apply format-specific optimizations to reduce upload times:

  • Tensor Files: The team is investigating compression for tensor files (e.g., Safetensors), which could potentially reduce upload speeds by 10-25%.
  • Parquet Files: The architecture enables improved deduplication for Parquet files.
  • Enterprise Security: The insertion of a control plane for file transfers allows the platform to prevent the upload of malicious or invalid data and provides detailed audit trails and telemetry for the infrastructure team.

Global Infrastructure Design

To minimize latency and balance costs, Hugging Face is deploying CAS nodes across three strategic AWS regions based on an analysis of upload traffic from 88 countries:

  • us-east-1 (North and South America): 4 nodes
  • eu-west-3 (Europe, Middle East, and Africa): 4 nodes
  • ap-southeast-1 (Asia and Oceania): 2 nodes

This distribution targets the regions responsible for the vast majority of upload volume, with the US and Europe accounting for 78.4% of uploaded bytes and Asia accounting for 21.6%.

Implementation Roadmap

Hugging Face expects to move the new infrastructure into production by the end of 2024, starting with a single CAS in us-east-1. The rollout will follow a phased approach:

  1. Internal Benchmarking: Duplicating internal repositories to the new system to test transfer performance.
  2. Regional Expansion: Replicating the CAS to additional points of presence (PoPs).
  3. Full Integration: Completing the storage backend transition in 2025.

Sources