NVIDIA/aistore

What it solves

AIStore (AIS) addresses the storage bottlenecks common in AI and machine learning workloads. It provides a high-performance, scalable distributed storage stack that handles massive datasets across multiple cloud backends and local clusters without the performance degradation typically seen when treating remote data as a simple cache.

How it works

AIS operates as an elastic cluster of storage nodes and gateways that can be deployed on bare-metal Linux machines or Kubernetes. It uses a unified namespace to manage data across independent clusters and supports a native HTTP-based API and an S3-compliant API. The system features a "Get-Batch" operation to fetch entire training batches in a single call, reducing overhead for ML pipelines. It also supports ETL offloading, allowing data transformations to occur directly on the storage nodes to reduce data movement.

Who it’s for

It is designed for AI researchers, data scientists, and ML engineers who need to manage petascale datasets and ensure high-throughput data delivery to training clusters.

Highlights

• Multi-Cloud Access: Seamlessly manages content across AWS S3, GCS, Azure, and OCI.
• Linear Scalability: Maintains consistent performance as the number of clustered nodes increases.
• Get-Batch: Optimized for ML pipelines by retrieving multiple objects in one operation as a TAR or other serialization format.
• ETL Offload: Executes I/O intensive transformations close to the data, either inline or as batch processing.
• Flexible Deployment: Runs anywhere from a single Linux VM to petascale Kubernetes clusters.
• Native PyTorch Integration: Includes dedicated datasets, samplers, and dataloaders for PyTorch workflows.