OpenDCAI/DataFlow

📦 DataFlow – One‑stop platform for building LLM‑training data pipelines

What it is – DataFlow is an open‑source Python framework that lets you turn raw, noisy text (PDFs, web crawls, spreadsheets, etc.) into clean, high‑quality datasets for training or augmenting large language models. It does this with a low‑code, visual pipeline builder and a library of reusable operators (e.g., generators, evaluators, filters). An optional AI‑assistant (DataFlow‑Agent) can even write new operators and stitch them together automatically.

---

🎯 Core ideas

Idea	Why it matters
Operator‑based pipelines	Each step (e.g., extract text, run an LLM to generate answers, filter low‑quality rows) is an isolated, version‑controlled component. Pipelines are reproducible, shareable, and can be swapped out for different models or heuristics.
Visual / low‑code UI	dataflow webui launches a drag‑and‑drop editor (or a natural‑language chat) so you don’t need to write boilerplate Python to wire operators together.
Built for data‑centric AI	Focuses on the data side of LLM development: synthesis, cleaning, evaluation, and filtering for domains like healthcare, finance, legal, or academic research.
Extensible ecosystem	A suite of companion repos – DataFlow‑Skills (tutorials), DataFlow‑WebUI, DataFlow‑Ecosystem (plug‑in operators), and RayOrch (distributed execution).
Agentic assistance	The DataFlow‑Agent can analyze a task description, generate operator code, and compose a pipeline on the fly.

---

⚙️ Main features (as listed in the README)

1. Ready‑made pipelines for common tasks:
   • Text QA mining, reasoning‑enhanced QA, Text‑to‑SQL generation, knowledge‑base cleaning, agentic RAG extraction.
2. 100+ operators covering generation, evaluation, filtering, and domain‑specific processing. You can also create custom operators and publish them via GitHub or PyPI.
3. Reproducibility & sharing – pipelines are just Python/YAML configs that can be version‑controlled like code.
4. Scalable execution – optional RayOrch layer schedules jobs across multiple GPUs or a cluster.
5. Web UI & CLI – dataflow webui for visual editing; dataflow CLI for quick runs and version checks.
6. Community extensions – e.g., a speech‑quality cleaning plug‑in from China Unicom.

---

🚀 Quick start (from the README)

# Install the fast installer "uv" first (optional but recommended)
pip install uv
uv pip install open-dataflow            # core package
uv pip install "open-dataflow[vllm]"   # with GPU‑accelerated inference support

# Verify installation
dataflow -v

You’ll see something like:

open-dataflow codebase version: 1.0.0
Checking for updates…
Local version:  1.0.0
PyPI newest version:  1.0.0
You are using the latest version: 1.0.0.

Run the visual builder

dataflow webui   # opens a browser‑based drag‑and‑drop pipeline editor

Run a pipeline from the CLI (example using the built‑in PromptedGenerator operator):

from dataflow.operators.core_text import PromptedGenerator
from dataflow.utils.storage import FileStorage
from dataflow.serving import APILLMServing_request

storage = FileStorage(first_entry_file_name="./input.json")
llm = APILLMServing_request(api_url="https://api.openai.com/v1/chat/completions")

gen = PromptedGenerator(llm_serving=llm, system_prompt="Please solve this math problem.")
gen.run(storage=storage.step(), input_key="problem", output_key="solution")

The operator reads input.json, calls the LLM, and writes a new column solution.

---

📚 Learning resources

• Video tutorials – Bilibili playlist (link in README)
• Written tutorials – Feishu wiki (link in README)
• Full documentation – https://OpenDCAI.github.io/DataFlow-Doc/
• Technical report – arXiv: 2512.16676 (accepted to ACM SIGMOD 2027)

---

🛠️ When to use DataFlow?

• You need to generate or clean large volumes of text data for fine‑tuning or RLHF of LLMs.
• Your data comes from heterogeneous sources (PDFs, tables, code repositories) and you want a reproducible workflow.
• You prefer a visual, low‑code interface over hand‑crafting ETL scripts.
• You plan to scale to many GPUs or a cluster – RayOrch handles the distribution.
• You want to share pipelines with teammates or the community (publish as a GitHub repo or PyPI package).

---

📦 Ecosystem snapshot

Component	Purpose
DataFlow‑Skills	Tutorials, example operators, best‑practice guides
DataFlow‑WebUI	Drag‑and‑drop pipeline editor + AI‑agent chat
DataFlow‑Ecosystem	Registry for domain‑specific operator packs (e.g., DataFlow‑MM)
RayOrch	Distributed task scheduler built on Ray
Community extensions	Speech‑quality cleaning, other domain‑specific plug‑ins

---

🏁 Bottom line

DataFlow is a genuine, production‑grade framework for the data‑centric side of LLM development. It bundles a library of reusable operators, a visual pipeline builder, and an optional AI‑agent that can auto‑compose pipelines. If you spend time curating or synthesizing training data, DataFlow gives you a reproducible, scalable, and extensible way to do it.