NVIDIA-NeMo/Curator

What it solves

NeMo Curator addresses the challenge of preparing massive datasets for AI training. It replaces ad-hoc scripts and one-off notebooks with repeatable, GPU-accelerated pipelines that can scale from a single laptop to multi-node GPU clusters, ensuring high-quality data through filtering, deduplication, and transformation.

How it works

The system uses a modular architecture based on stages and executors. Each stage handles a specific task—such as loading, filtering, or deduplicating data—and declares its own resource requirements. These stages are chained into pipelines that stream data concurrently, overlapping CPU and GPU work to maximize hardware utilization. It leverages NVIDIA RAPIDS (cuDF, cuML, cuGraph) and Ray to achieve high throughput across distributed environments.

Who it’s for

ML engineers and data teams who need to process and curate large-scale multimodal datasets (text, image, video, and audio) for training frontier AI models.

Highlights

• Multimodal Support: Ready-made building blocks for text (deduplication, language detection), images (aesthetic filtering, NSFW detection), video (scene detection, motion filtering), and audio (ASR transcription, quality assessment).
• GPU Acceleration: Significant speedups in data processing, such as reducing fuzzy deduplication time from 10.7 hours to 0.65 hours on H100 nodes.
• Distributed Scaling: Native support for Ray and Slurm, allowing pipelines to run across multi-node HPC clusters.
• Synthetic Data Generation: Includes an OpenAI-compatible inference server to integrate LLM-based synthetic data workflows directly into the pipeline.
• Proven at Scale: Used to curate the 8+ trillion token multilingual web dataset for NVIDIA Nemotron models.