huggingface/hf-mount

What it solves

hf-mount allows users to access Hugging Face Buckets and repositories as if they were local filesystems. This eliminates the need to download or clone entire repositories before using them, which is especially useful for large ML models and datasets where disk space is limited or waiting for downloads is impractical.

How it works

The tool exposes Hugging Face storage as a local filesystem using either FUSE (Filesystem in Userspace) or NFS (Network File System) backends. It employs lazy loading, meaning files are only fetched from the network when they are actually read by the code.

It supports two write modes: a default "streaming" mode for simple append-only writes and an "advanced writes" mode that uses local staging files to support random writes and seeks. It also features an Overlay mode, where the mount point acts as a writable local layer on top of a read-only remote source.

Who it’s for

• ML Engineers and Researchers: Those loading large models and datasets for training, inference, or evaluation without wanting to full-repo downloads.
• Developers: Those who need to browse repository contents using standard UNIX tools (ls, cat, find, grep) without cloning.
• Users with limited disk space: Those operating in environments where local storage is constrained.

Highlights

• Lazy Loading: Only the bytes actually touched by the application are downloaded.
• Dual Backends: Supports both NFS (no root required) and FUSE (tighter kernel integration).
• Subfolder Mounting: Ability to mount specific subdirectories of a repo or bucket.
• Remote Sync: Uses a live-follow event stream and background polling to keep the local view updated with remote changes.
• Overlay Mode: Allows local writes to persist on disk while keeping the remote source read-only, useful for shared compilation caches.
• POSIX Metadata: Supports basic file permissions, timestamps, and symlinks in-memory.