datajuicer/data-juicer

What it solves

Data-Juicer is a comprehensive data processing system designed to transform raw, chaotic data into high-quality, AI-ready datasets. It addresses the challenge of preparing large-scale data for foundation models, including cleaning, deduplication, and synthesis across multiple modalities (text, image, audio, video).

How it works

The system treats data processing as composable infrastructure. It uses a modular architecture with over 200 operators that can be chained together into reproducible pipelines defined via YAML recipes. These pipelines can scale from a single laptop to thousand-node clusters using Ray for distributed execution, with optimizations like automatic operator fusion and CUDA acceleration to handle petabyte-scale datasets.

Who it’s for

It is built for AI researchers and engineers who need to curate pre-training corpora, prepare fine-tuning or RL data, clean agent interaction traces, or build domain-specific RAG indices.

Highlights

• Multimodal Support: Operators for text, image, audio, video, and embodied AI (VLA) data.
• Recipe-First Workflow: Reproducible YAML-based pipelines that can be versioned and shared.
• Cloud-Native Scaling: Deep integration with Ray for high-performance distributed processing of billions of samples.
• Extensive Operator Zoo: Over 200 built-in operators for filtering, mapping, and analysis.
• Juicer Model: A locally deployable data-refinement model that follows natural-language instructions for cleaning and labeling.