open-gigaai/giga-datasets

What it solves

GigaDatasets provides a unified, lightweight framework for managing massive datasets used in AI training and inference. It simplifies the complex process of data curation, packaging, loading, and evaluation, ensuring consistency across different data formats and structures.

How it works

The framework operates through a set of specialized tools for different stages of the data pipeline:

• Data Packaging: It uses Writer classes (such as PklWriter, FileWriter, and LmdbWriter) to convert unstructured raw data (like images and JSON annotations) into structured, efficient formats.
• Data Loading: A single load_dataset function allows users to load datasets regardless of the underlying storage format.
• Data Evaluation: The FIDEvaluator provides a streamlined way to evaluate prediction results against the dataset.
• Extensibility: The system is designed to allow users to easily add new data fields (e.g., adding canny maps to an existing dataset) to accommodate different model requirements.

Who it’s for

It is designed for AI researchers and developers who handle large-scale multimodal data, including images, videos, 2D/3D boxes, and 2D/3D points, as well as those using LeRobot datasets.

Highlights

• Unified Workflow: Combines curation, packaging, loading, and evaluation into one system.
• Multi-format Support: Compatible with File, LMDB, Pickle, and LeRobot datasets.
• High Performance: Optimized for speed and memory efficiency during large-scale processing.
• Simple API: Enables dataset loading and evaluation with single lines of code.