mosaicml/streaming

A Data Streaming Library for Efficient Neural Network Training

What it solves

训练大规模 AI 模型通常需要数据集太大，无法放入本地磁盘，这会导致从云端存储加载数据时出现瓶颈。本项目提供了一个高性能的流式传输库，允许模型即时获取数据，从而减少对大规模本地存储的需求，并消除在恢复训练运行时的漫长等待时间。

How it works

StreamingDataset 可作为 PyTorch 的 IterableDataset 的直接替换方案。它 converts 原始数据转换为一种称为 Mosaic Data Shard (MDS) 的专用格式，然后上传到云端存储（AWS, GCS, Azure, 等）。在训练期间，shuffle-based 库将数据分片 (shards) 流式传输到本地缓存，仅下载当前步骤所需的数据。它使用确定性洗牌算法来确保无论在集群中使用了多少个 GPU 或节点，节点间的数据交付顺序一致。

Who it’s for

需要在多节点、分布式集群上训练大模型（LLMs, Diffusion models, 等）的机器学习工程师和研究人员，他们需要高效地处理存储在云端的超大规模数据集。

Highlights

  • True Determinism: 确保在不同硬件配置下具有相同的样本排序，使调试和复现更加容易。

  • Instant Mid-Epoch Resumption: 允许在故障后几秒钟内恢复训练，避免了大规模数据集常见的漫长数据加载等待时间。

  • Seamless Data Mixing: 支持即时以特定比例或绝对样本数量进行多个数据集的混合。

  • High Throughput: Streaming-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-based loader-s

  • Random Access: 支持通过索引访问任何特定样本，例如即使它尚未被下载。

  • Disk Usage Limits: s

  • High Throughput: Optimized MDS format provides lower sample latency and higher throughput compared to optimized MDS format provides lower sample latency and optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS optimized MDS*

  • Disk Usage Limits: Includes a cache_limit 功能，通过删除最近最少使用 (LRU) 的分片 (shards) 管理自动管理本地磁盘空间。