meta-pytorch/data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.

What it solves

TorchData provides scalable and performant dataloading solutions for PyTorch, addressing the need for better state management and flexible data preprocessing pipelines.

How it works

It enhances the standard PyTorch DataLoader and Dataset through two primary components:

  • StatefulDataLoader: A drop-in replacement for the standard DataLoader that adds state_dict and load_state_dict methods, allowing users to save and resume training from mid-epoch checkpoints, including tracking custom iteration progress and RNG states.
  • torchdata.nodes: A library of composable iterators that allows users to chain together common dataloading and preprocessing operations using a streaming programming model.

Who it’s for

Machine learning engineers and researchers using PyTorch who need to handle large datasets and ensure training can be resumed precisely from where it left off.

Highlights

  • Mid-epoch checkpointing: Enables saving and restoring the exact state of the dataloader.
  • Composable iterators: Provides a streaming model for chaining preprocessing steps.
  • Drop-in replacement: The StatefulDataLoader integrates easily into existing PyTorch workflows.

Related

  • Project
    sktime/pytorch-forecasting

    A PyTorch-based library for time series forecasting that provides a high-level API and state-of-the-art deep learning architectures like Temporal Fusion Transformers and N-BEATS.

  • Project
    facebookresearch/multimodal

    A PyTorch library for training state-of-the-art multimodal models at scale, providing modular building blocks and pretrained weights for models like CLIP and BLIP-2.

  • Project
    nschaetti/EchoTorch

    A PyTorch-based module for implementing and testing Echo State Networks and Reservoir Computing, designed for research purposes and integration into deep architectures.

  • Project
    mosaicml/streaming

    A high-performance data streaming library for PyTorch that enables fast, deterministic loading of massive datasets from cloud storage during distributed AI training.

  • Project
    Lightning-AI/torchmetrics

    A collection of over 100 PyTorch metrics implementations that automate accumulation and synchronization across distributed devices to reduce boilerplate in ML pipelines.