datajuicer/data-juicer

Data‑Juicer: A Data‑Operating System for AI

What it is – Data‑Juicer (DJ) is an open‑source library that treats data preparation for foundation models as a composable, cloud‑native workflow. It ships a Python package (py-data-juicer) and a CLI (dj‑process) that let you build pipelines from 200+ reusable operators (filters, mappers, analysers) covering text, image, audio, video and multimodal data.

Why it matters – Modern LLM or multimodal model training needs billions of high‑quality samples. Cleaning, deduplicating, chunking, and annotating that data is a massive engineering effort. DJ abstracts those steps into reusable, version‑able YAML recipes and a scalable execution engine (Ray‑based) that can run on a laptop or a thousand‑node cluster without custom glue code.

Key features

• Modular operators – over 200 built‑in ops; you can also add third‑party ops via pip plugins.
• Recipe‑first pipelines – YAML files describe the operator chain; they are shareable and reproducible.
• Scalable execution – Ray partitioned executor, automatic GPU/CPU resource planning, MinHash deduplication, and CUDA‑accelerated mappers. Reported ability to process 70 B samples in 2 h on 50 nodes.
• Observability – built‑in tracing, stats collection, and a visual analysis UI.
• Domain‑specific extensions – agents (DJ Copilot), a natural‑language “Juicer” model for NL‑driven cleaning, and recipes for RAG, RL‑style curation, and embodied‑AI video pipelines.
• Integrations – works with Hugging Face, ModelScope, Alibaba Cloud PAI, HDFS, S3, Delta Lake, LanceDB, Ray, and many other data platforms.

Typical workflow

1. Load a dataset (local files, HDFS, S3, Hugging Face hub, etc.).
2. Define a pipeline in YAML or directly in Python, e.g.:

   - op: TextLengthFilter   # drop short sentences
     min_len: 10
   - op: WhitespaceNormalizationMapper
   - op: MinHashDeduplicator
   

3. Run with dj-process --config my_pipeline.yaml or via the Python API (NestedDataset.process([...])).
4. Analyze the output with built‑in stats or the web UI; export to JSONL, Parquet, etc.

Getting started – Install with uv pip install py-data-juicer (or pull the Docker image). A JupyterLab playground and a “Ask DJ Copilot” web UI are provided for zero‑install exploration.

Community & adoption – Used internally at Alibaba Cloud, integrated into products, and cited by several industry partners (ByteDance, NVIDIA, JD.com, etc.). The project is actively maintained (monthly releases, recent v1.6.x series) and welcomes contributions via good‑first‑issues, plugins, or new operator recipes.

---

TL;DR – Data‑Juicer is a production‑grade, extensible data‑processing framework for building, scaling, and auditing the massive data pipelines that power modern foundation‑model training and RAG systems.