facebookresearch/spdl

Scalable and Performant Data Loading

What it solves

It addresses the challenge of designing performant and scalable data loading systems for processing large-scale array data, which is often a bottleneck in machine learning pipelines.

How it works

SPDL provides a flexible pipeline abstraction and a set of specialized operations designed to process array data efficiently.

Who it’s for

Researchers and engineers who need to optimize the data loading and preprocessing stage of their AI/ML pipelines to ensure high performance andesprit scalable data throughput.

Highlights

  • Scalable and performant data loading architecture
  • Flexible pipeline abstraction for data processing
  • Specialized operations for array data manipulation

Related

  • Project
    IBM/lale

    A Python library for semi-automated data science that automates algorithm selection and hyperparameter tuning for scikit-learn compatible pipelines.

  • Project
    google/seqio

    A library for processing sequential data that provides a unified framework for task-based datasets, preprocessing, and evaluation for sequence models.

  • Project
    microsoft/FLAML

    A lightweight Python library for efficient automated machine learning and hyperparameter tuning that optimizes model performance under resource constraints.

  • Project
    aws/sagemaker-spark

    An open-source Spark library that integrates Amazon SageMaker into Spark ML pipelines for scalable model training and inference on DataFrames.

  • Project
    bespokelabsai/curator

    Bespoke Curator is a Python library for bulk LLM inference and synthetic data generation. It provides a `curator.LLM` base class for prompt/parse logic, supports many providers (OpenAI, Anthropic, Gemini, Ollama, vLLM, etc.), offers batch mode and caching to cut costs, includes a code‑execution engine, and integrates with the Tinker SDK for LoRA fine‑tuning. Ideal for building large‑scale, structured datasets for training or evaluation.