Hugging Face and SkyPilot Integration for Zero-Egress AI Storage

Hugging Face and SkyPilot have integrated to enable AI developers to run compute workloads across any cloud provider while storing their models and datasets on the Hugging Face Hub. This integration eliminates the "cross-cloud transfer tax" by providing zero-egress storage, allowing users to mount Hub repositories and Buckets directly into SkyPilot jobs regardless of where the GPUs are located.

Zero-Egress Storage Across Multiple Clouds

Cloud GPU capacity is often fragmented across multiple vendors (hyperscalers, neoclouds, and on-prem). Traditionally, moving data from a regional object store (like AWS S3) to a GPU in a different cloud incurs significant egress fees (approximately $0.09/GB on AWS).

By using Hugging Face Storage, which charges no egress or CDN fees, teams can read their models and datasets from a single source across 20+ clouds, Kubernetes, and on-prem clusters. This removes the need to maintain redundant copies of data across every vendor's bucket and prevents the "pinning" of workloads to a specific cloud based on where the data resides.

Technical Implementation: store: hf and hf://

SkyPilot now supports Hugging Face Storage as a first-class backend via the store: hf configuration and the hf:// URI scheme. This allows for the mounting of three types of Hub assets:

• Hugging Face Buckets: Read-write access for checkpoints, logs, and processed data.
• Model Repositories: Read-only access to model weights.
• Dataset Repositories: Read-only access to datasets, often pinned to specific revisions.

Mounting Modes: MOUNT vs. COPY

• MOUNT: Utilizes the hf-mount FUSE backend. It performs lazy reads, pulling only the bytes required by the application. This allows GPUs to begin training or inference almost immediately without waiting for a full download. MOUNT also maintains an on-disk cache for repeat reads.
• COPY: Downloads the assets via the huggingface_hub client upfront, requiring no special FUSE requirements.

Authentication is handled via a single HF_TOKEN passed as a secret to the SkyPilot run, which works across all cloud providers.

Xet-Backed Deduplication for Efficiency

Hugging Face Buckets are built on Xet, which employs content-defined chunking to split files into ~64 KB chunks. This architecture provides several efficiency gains:

• Incremental Checkpoints: Only chunks that have changed between saves are uploaded, reducing transfer time for adapter checkpoints or frozen layers.
• Model Variants: Shared chunks between a base model and its fine-tunes or quantizations are stored only once.
• Dataset Appends: Appending rows to large Parquet files only transfers the new data. In tests, appending 10K rows to a 100K-row table transferred ~10 MB instead of ~106 MB.
• Fast Re-uploads: Re-uploading a blob already present in the bucket is significantly faster (e.g., 8 seconds vs 24 seconds for an 8.43 GB file) because only chunk hashes are transferred.

Performance Benchmarks

In a fine-tuning test using Qwen/Qwen3.5-4B and the Multilingual-Thinking dataset, the following results were observed:

• Model Loading: The model was ready for training in approximately 30 seconds with speeds up to ~500 MB/s. Due to the zero-egress policy, this cost nothing across AWS, GCP, and Lambda.
• Checkpoint Writing: Checkpoints (8.43 GB each) were streamed to the bucket with the following throughput:

Cloud	GPU	Checkpoint Write Speed
AWS (us-east-2)	L40S	~168 MB/s
GCP (us-central1)	L4	~123 MB/s
Lambda (us-west-3)	H100	~112 MB/s

Getting Started

To use this integration, users can install the required package:

pip install "skypilot[huggingface]"

Users must then authenticate using hf auth login or by exporting the HF_TOKEN environment variable. For MOUNT mode, the base image must have /dev/fuse and glibc 2.34+.