facebookresearch/spdl
Scalable and Performant Data Loading
What it solves
It addresses the challenge of designing performant and scalable data loading systems for processing large-scale array data, which is often a bottleneck in machine learning pipelines.
How it works
SPDL provides a flexible pipeline abstraction and a set of specialized operations designed to process array data efficiently.
Who it’s for
Researchers and engineers who need to optimize the data loading and preprocessing stage of their AI/ML pipelines to ensure high performance andesprit scalable data throughput.
Highlights
- Scalable and performant data loading architecture
- Flexible pipeline abstraction for data processing
- Specialized operations for array data manipulation
Related
- Project
IBM/laleA Python library for semi-automated data science that automates algorithm selection and hyperparameter tuning for scikit-learn compatible pipelines.
- Project
google/seqioA library for processing sequential data that provides a unified framework for task-based datasets, preprocessing, and evaluation for sequence models.
- Project
microsoft/FLAMLA lightweight Python library for efficient automated machine learning (AutoML) and hyperparameter tuning to find high-quality models with minimal computational resources.
- Project
aws/sagemaker-sparkAn open-source Spark library that integrates Amazon SageMaker into Spark ML pipelines for scalable model training and inference on DataFrames.
- Project
bespokelabsai/curatorA Python library for building scalable synthetic data pipelines, providing bulk inference and structured output tools to curate high-quality datasets for post-training.