OpenDCAI/DataFlow

What it solves

DataFlow is a data-centric AI system designed to transform noisy, raw data (such as PDFs, plain-text, and low-quality QA pairs) into high-quality training datasets for Large Language Models (LLMs). It addresses the time-consuming and costly nature of manual data cleaning and synthesis, specifically for domains like healthcare, finance, legal, and academic research.

How it works

The system uses an "operator-based" design where individual data processing tasks (generation, evaluation, filtering, and refinement) are encapsulated as operators. These operators are then orchestrated into reproducible and shareable pipelines.

Key components include:

• Operators: Modular units that take structured inputs (JSON, JSONL, CSV) and produce processed outputs.
• Pipelines: Sequences of operators that define a specific data workflow (e.g., Text, Reasoning, Text2SQL, and Knowledge Base Cleaning).
• DataFlow-WebUI: A visual, low-code workspace for building and running these pipelines via drag-and-drop or natural language.
• DataFlow-Agent: An intelligent assistant that can analyze data, write custom operators, and automatically assemble pipelines.

Who it’s for

DataFlow is built for AI researchers and enterprises who need to scale the preparation of high-quality data for Pre-training, Supervised Fine-tuning (SFT), RL training, or RAG systems.

Highlights

• Low-Code Orchestration: Visual pipeline builder via WebUI for faster experimentation.
• Ready-to-Use Pipelines: Pre-built workflows for text, math, code, and structured data extraction from PDFs.
• PyTorch-like Structure: A hierarchical Pipeline $\rightarrow$ Operator $\rightarrow$ Prompt organization for clear workflow control.
• Distributed Scaling: Integration with RayOrch for high-performance orchestration of massive-scale data tasks.