earth-mover/icechunk

What it solves

Icechunk provides a transactional storage engine for tensor and ND-array data, specifically designed for cloud object storage. It solves the consistency and safety issues that arise when multiple uncoordinated processes read and write Zarr data in parallel, allowing Zarr stores to function more like a database with serializable isolation.

How it works

Icechunk adds a layer of indirection between Zarr keys and the actual on-disk storage. Instead of mapping Zarr keys directly to filenames, it manages a system of immutable metadata and data files, including a "chunk manifest" that explicitly tracks every chunk. This allows it to implement transactions, where updates are committed atomically and are not visible to readers until the transaction is completed.

Who it’s for

It is designed for data scientists, AI/ML engineers, and researchers in scientific computing who work with large-scale multidimensional arrays (tensors) and require reliable, collaborative cloud storage using the Zarr specification.

Highlights

• Serializable Isolation: Reads are isolated from concurrent writes, ensuring users always see a committed snapshot.
• Data Version Control: Supports branches (mutable references) and tags (immutable references) for managing different versions of a dataset.
• Time Travel: Previous snapshots of the repository remain accessible after new updates.
• Chunk Sharding: Decouples chunk storage from filenames, allowing multiple chunks to be packed into a single object for better performance.
• Virtual Datasets: Can reference chunks existing in other formats like HDF5 or NetCDF without modifying the original files.