google/tensorstore

What it solves

TensorStore addresses the challenge of managing and manipulating massive multi-dimensional arrays that are too large to fit in memory, especially when stored across diverse storage systems like cloud object stores or network filesystems.

How it works

It provides a C++ and Python library that creates a uniform API for reading and writing various array formats (such as Zarr and N5). It uses an asynchronous API to maintain high throughput even when dealing with high-latency remote storage and implements optimistic concurrency to allow safe, efficient access from multiple processes or machines.

Who it’s for

Researchers and developers working with large-scale numerical data, such as those in AI and scientific computing, who need high-performance access to multi-dimensional arrays across local or cloud storage.

Highlights

• Multi-storage support: Natively supports Google Cloud Storage, Amazon S3, HTTP servers, local filesystems, and in-memory storage.
• Composable Indexing: Offers advanced indexing operations and virtual views.
• ACID Guarantees: Supports read caching and transactions with strong atomicity, isolation, consistency, and durability.
• High Throughput: Uses an asynchronous API to mitigate latency issues with remote storage.