datajuicer/data-juicer

What it solves

Data-Juicer addresses the "raw data chaos" encountered when preparing datasets for foundation models. It provides a scalable, composable infrastructure to transform messy, unorganized data into high-quality, AI-ready intelligence for pre-training, fine-tuning, and RAG applications.

How it works

The system uses a modular architecture of over 200 operators that can be chained together into reproducible pipelines (recipes) defined in YAML. It supports various data modalities including text, image, audio, video, and multimodal data. For large-scale workloads, it leverages distributed execution frameworks like Ray to process massive datasets across thousands of cores.

Who it’s for

• AI Researchers building foundation models who need high-quality pre-training or fine-tuning corpora.
• Machine Learning Engineers developing agentic systems or RAG pipelines who need to clean interaction traces or structure context.
• Data Scientists working with multimodal data (video, audio, etc.) who require specialized processing like camera calibration or pose estimation.

Highlights

• Modular Architecture: 200+ operators for text, image, audio, video, and multimodal data.
• Scalable Performance: Capable of processing 70 billion samples in 2 hours on a 6400-core cluster.
• Recipe-driven: Reproducible YAML pipelines that can be versioned and shared.
• Multimodal Support: Includes specialized operators for embodied AI, video understanding, and semantic LLM extraction.
• Production-Ready: Integrated with Alibaba Cloud PAI and supports distributed execution via Ray.