huggingface/xet-core

What it solves

Large machine-learning files (models, datasets) often share identical chunks—for example, common code, repeated parameters, or duplicated data. Without deduplication, every upload or download re-sends and re-stores these redundant chunks, wasting bandwidth and storage. xet-core eliminates that waste by splitting files into chunks and identifying duplicates, so only unique chunks travel and are stored.

How it works

xet-core breaks files into chunks and hashes each one with a MerkleHash. It talks to a content-addressable storage (CAS) backend on Hugging Face Hub to see which chunks already exist, then uploads only the new ones. On download, it reconstructs files from chunks, using a local disk cache to avoid fetching the same chunk twice. It is written in Rust for speed and exposes Python bindings so huggingface_hub can use it seamlessly. The library also includes a Git LFS–compatible command-line tool and several Rust crates for different layers (client, data pipeline, core structures, runtime).

Who it’s for

End users of huggingface_hub who upload or download large models and datasets. They don’t interact with xet-core directly—huggingface_hub uses it under the hood. Developers of huggingface_hub or similar storage integrations can also use the Rust crates directly.

Highlights

• Chunk-based deduplication avoids transferring and storing redundant data across files.
• Local disk caching reuses previously downloaded chunks to speed up repeated operations.
• Python bindings allow drop-in use within huggingface_hub (via the hf-xet PyPI package).
• Concurrent network communication to Hugging Face Hub’s CAS servers for fast transfers.
• Git LFS compatibility via the git-xet CLI tool.
• Privacy-conscious telemetry that reports only byte counts and performance metrics—no file names, paths, hashes, or user IDs—and can be disabled.