Hugging Face Storage Buckets Release

Hugging Face has launched Storage Buckets, a mutable, S3-like object storage system designed for intermediate machine learning artifacts. This new storage layer allows developers to manage high-throughput data—such as checkpoints, optimizer states, and processed shards—without the overhead of Git version control.

Optimized Storage for ML Artifacts via Xet

Storage Buckets are powered by Xet, a chunk-based storage backend that deduplicates content across files. Unlike traditional object storage that treats files as monolithic blobs, Xet breaks content into chunks, allowing the system to skip bytes that already exist on the Hub.

This architecture is specifically beneficial for ML workloads where related artifacts often share significant overlap, such as:

  • Successive checkpoints: Where large portions of the model remain frozen.
  • Processed datasets: Which often remain mostly similar to their raw counterparts.
  • Agent traces: And their derived summaries.

For Enterprise customers, this efficiency extends to cost, as billing is based on the deduplicated storage footprint, reducing both bandwidth usage and total billed storage.

Pre-warming for High-Throughput Compute

To minimize latency in distributed training and large-scale pipelines, Hugging Face introduced "pre-warming." This feature allows users to declare the required data location, moving hot data closer to the cloud provider and region where compute is running.

By reducing the need to pull data across regions during every read, pre-warming increases throughput for training clusters and multi-region pipeline setups. Hugging Face is currently partnering with AWS and GCP to provide this capability, with more providers planned for the future.

Integration and Programmatic Access

Storage Buckets are integrated into the Hugging Face ecosystem through multiple interfaces, allowing them to be managed as non-versioned containers under user or organization namespaces.

CLI and Python API

Users can manage buckets using the hf CLI or the huggingface_hub Python library (available since v1.5.0). Key operations include:

  • Creation: hf buckets create [name] --private
  • Synchronization: hf buckets sync [local_dir] [hf_path]
  • Inspection: hf buckets list [bucket_name] -h

Filesystem Integration via fsspec

Through HfFileSystem, Buckets are compatible with the fsspec standard. This enables direct integration with data libraries like pandas, Polars, and Dask using hf:// paths. For example, a CSV can be read directly into a pandas DataFrame without local downloading: pd.read_csv("hf://buckets/username/my-training-bucket/results.csv").

JavaScript Support

Bucket support is also available for Node.js services and web applications via the @huggingface/hub library (since v2.10.5).

Workflow: From Mutable Storage to Versioned Repos

Storage Buckets serve as the working layer for artifacts in motion. Once an artifact becomes a stable deliverable, the intended workflow is to move it from a Bucket to a versioned model or dataset repository. Hugging Face plans to implement direct transfers between Buckets and repositories to streamline the promotion of final checkpoint weights or processed shards into official Hub releases.

Availability and Pricing

Storage Buckets are included in existing Hub storage plans. Free accounts include initial storage, while PRO and Enterprise plans provide higher limits.

Sources