open-edge-platform/datumaro

What it solves

Datumaro simplifies the complex process of preparing datasets for AI model training. It addresses the problem of fragmented dataset formats, inconsistent annotations, and the need for rigorous data cleaning and splitting before a model can be trained.

How it works

It acts as a central hub for dataset management through a framework and CLI tool. It can read and write data in a wide variety of industry-standard formats (such as COCO, Pascal VOC, and YOLO), allowing users to convert datasets between these formats. Beyond conversion, it provides tools to merge datasets, filter out unwanted data based on specific criteria, transform annotations (e.g., converting polygons to masks), and split data into training, validation, and test sets while maintaining label distributions.

Who it’s for

Data scientists and ML engineers who need to build, transform, and analyze computer vision datasets for model training and publication.

Highlights

• Multi-format support: Seamlessly converts between numerous formats including COCO, YOLO, Cityscapes, and ImageNet.
• Dataset building: Tools for merging datasets and filtering images or annotations based on custom criteria.
• Intelligent splitting: Supports random splits or task-specific splits that preserve label and attribute distributions.
• Quality control: Includes features for error checking, annotation validation, and comparison with model inference results.
• Dataset statistics: Generates image mean, standard deviation, and annotation statistics.