huggingface/hf-mount

What it solves

hf-mount allows users to access Hugging Face Buckets and repositories as if they were local folders on their computer. This eliminates the need to download entire large models or datasets before using them, saving disk space and reducing the time spent waiting for downloads.

How it works

The tool acts as a bridge between the application and the Hugging Face Hub, using either FUSE (Filesystem in Userspace) or NFS (Network File System) to present a virtual filesystem. It fetches files lazily—meaning it only downloads the specific bytes of a file that your code actually reads—and uses an adaptive prefetch buffer to optimize sequential access. It is built on xet-core for content-addressed storage and fuser for FUSE implementation.

Who it’s for

• ML Engineers and Researchers: Those performing read-heavy workloads like training, inference, or evaluation on large models and datasets.
• Developers with Limited Disk Space: Users who cannot store full repositories locally.
• AI Agents: Agents that can interact with data using standard UNIX tools like ls, cat, and grep without needing specialized APIs.

Highlights

• Lazy Loading: Only fetches data on demand, avoiding full repository clones.
• Dual Backends: Supports both NFS (no root required) and FUSE (tighter kernel integration).
• Subfolder Mounting: Ability to mount specific subdirectories of a repo or bucket.
• Overlay Mode: Allows a read-only remote view to be layered with a writable local disk, useful for shared compilation caches.
• Remote Sync: Background polling automatically detects and updates the local view based on remote changes.
• Advanced Write Support: Supports random writes, seeks, and overwrites via local staging files.