huggingface/xet-core

What it solves

xet-core provides a high-performance storage system designed for the massive binary files typically found in AI models and datasets. It solves the problem of inefficient storage and transfer of these files by implementing chunk-based deduplication, which ensures that only unique parts of files are uploaded or downloaded, reducing bandwidth and storage overhead.

How it works

The project implements a Content Addressable Storage (CAS) system. It breaks files into chunks and uses a data processing pipeline for chunking, deduplication, and reconstruction. It utilizes a local disk cache to speed up retrieval and provides Python bindings so it can be integrated into the huggingface_hub library. It also includes a Git LFS compatible CLI tool (git-xet) for users who prefer command-line interaction.

Who it’s for

This library is primarily intended for the developers of huggingface_hub and users who upload or download large-scale AI assets from the Hugging Face Hub, as it is not meant to be used as a standalone library by end-users.

Highlights

• Chunk-based deduplication: Avoids transferring and storing redundant data across different binary files.
• Local disk caching: Implements a chunk-based cache to optimize storage and retrieval.
• Git LFS compatibility: Maintains backwards compatibility with Git LFS via a dedicated CLI tool.
• Cross-platform support: Provides Rust crates for core logic and Python bindings for accessibility.