mosaicml/streaming

A Data Streaming Library for Efficient Neural Network Training

What it solves

Training large-scale AI models often requires datasets too large to fit on local disks, leading to bottlenecks when loading data from cloud storage. This project provides a high-performance streaming library that allows models to fetch data just-in-time, reducing the need for massive local storage and eliminating long wait times when resuming training runs.

How it works

StreamingDataset acts as a drop-in replacement for PyTorch's IterableDataset. It converts raw data into a specialized format called Mosaic Data Shard (MDS), which is then uploaded to cloud storage (AWS, GCS, Azure, etc.). During training, the library streams shards of data into a local cache, downloading only what is needed for the current step. It uses a deterministic shuffling algorithm to ensure that data is delivered in the same order regardless of the number of GPUs or nodes used in a cluster.

Who it’s for

ML engineers and researchers training large models (LLMs, Diffusion models, etc.) on multi-node, distributed clusters who need to efficiently handle massive datasets stored in the cloud.

Highlights

  • True Determinism: Ensures identical sample ordering across different hardware configurations, making debugging and reproduction easier.
  • Instant Mid-Epoch Resumption: Allows training to resume in seconds after a failure, avoiding the hours of data-loading wait time typical of large datasets.
  • Seamless Data Mixing: Supports mixing multiple datasets with specific proportions or absolute sample counts just-in-time.
  • High Throughput: Optimized MDS format provides lower sample latency and higher throughput compared to traditional loaders.
  • Random Access: Supports accessing any specific sample by index, even if it hasn't been downloaded yet.
  • Disk Usage Limits: Includes a cache_limit feature to automatically manage local disk space by deleting least recently used shards.