TileDB-Inc/TileDB

What it solves

TileDB addresses the difficulty of storing and accessing complex, high-dimensional data. It provides a way to model data as dense or sparse multi-dimensional arrays, which is the native format for most data science tools, thereby removing the need for complex data management and storage abstractions.

How it works

It is an embeddable C++ library that implements a "universal storage engine" based on chunked (tiled) arrays. It supports both dense and sparse array formats, as well as dataframes and key-value stores. The engine includes built-in support for cloud storage (AWS S3, Google Cloud Storage, Azure Blob Storage), multi-threading, parallel IO, and data versioning for rapid updates and "time traveling."

Who it’s for

It is designed for developers and data scientists working with large-scale complex data in fields such as Genomics, Geospatial analysis, and Finance.

Highlights

• Support for dense and sparse multi-dimensional arrays
• Native cloud storage integration (S3, GCS, Azure)
• Data versioning and time-travel capabilities
• Multi-threaded implementation with parallel IO
• Broad API support including Python, R, Java, Go, and C#
• Integrations with Spark, Dask, MariaDB, and GDAL