NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
What it solves
NeMo Curator addresses the challenge of preparing massive datasets for AI training. It replaces ad-hoc scripts and notebooks with repeatable, GPU-accelerated pipelines that can scale from a single laptop to multi-node clusters, ensuring high-quality data through filtering, deduplication, and transformation.
How it works
The system uses a modular architecture based on stages (discrete tasks like loading, filtering, or classifying) and executors (the runtime environment).
- Pipeline Composition: Stages are chained together, and the executor automatically scales replicas to maintain throughput.
- Streaming Execution: It overlaps CPU and GPU workloads to keep GPUs highly utilized (often >99%).
- Modality Support: It provides specialized building blocks for text, image, video, and audio data.
- Distributed Scaling: It leverages NVIDIA RAPIDS (cuDF, cuML, cuGraph) and Ray to distribute workloads across multiple GPUs and nodes.
Who it’s for
ML engineers and data teams who need to process trillion-token scale datasets or multimodal data (images, video, audio) for training frontier AI models.
Highlights
- Multimodal Capabilities: Dedicated pipelines for text (deduplication, language detection), images (aesthetic/NSFW filtering), video (scene detection, motion filtering), and audio (ASR transcription, WER filtering).
- GPU Acceleration: Significant speedups over CPU-based alternatives, such as reducing fuzzy deduplication time from 10.7 hours to 0.65 hours.
- Production Proven: Powers the data pipelines for NVIDIA Nemotron models, including the curation of 8+ trillion tokens of web data.
- Synthetic Data Generation: Includes an OpenAI-compatible inference server to facilitate SDG and classification workflows within the pipeline.