NVIDIA-NeMo/Curator

What it solves

NeMo Curator addresses the challenge of preparing massive datasets for AI training. It replaces ad-hoc scripts and notebooks with repeatable, GPU-accelerated pipelines that can handle trillions of tokens or vast amounts of multimodal data (text, image, video, and audio) across multi-node clusters.

How it works

The system uses a modular architecture based on stages and executors. Each stage performs a specific task—such as loading, filtering, deduplicating, or transforming data—and declares its own resource requirements. These stages are chained into pipelines that stream tasks concurrently, overlapping CPU and GPU work to maximize hardware utilization. The production default is the XennaExecutor, though Ray-based backends are also supported. It leverages NVIDIA RAPIDS (cuDF, cuML, cuGraph) to achieve high throughput and scale across GPUs.

Who it’s for

It is designed for ML engineers and data teams who need to build production-grade data curation pipelines for large-scale AI model training, particularly those using NVIDIA training workflows like Nemotron.

Highlights

• Multimodal Support: Ready-made building blocks for text (deduplication, classification), images (aesthetic/NSFW filtering), video (scene detection, clip extraction), and audio (ASR transcription, quality assessment).
• GPU Acceleration: Significant speedups in data-heavy tasks like fuzzy deduplication compared to CPU-based alternatives.
• Distributed Execution: Ability to run the same pipeline on a single laptop or across multi-node Ray or Slurm clusters.
• Integrated Inference: Includes an OpenAI-compatible LLM endpoint for synthetic data generation (SDG) and classification within the pipeline.
• Proven at Scale: Powers the data pipelines for NVIDIA Nemotron models, processing over 8 trillion tokens.