duoan/mega-data-factory

What it solves

Mega Data Factory is a high-throughput, distributed pipeline designed to process web-scale multimodal datasets (hundreds of billions of records). It allows researchers and developers to reproduce state-of-the-art foundation model data pipelines—such as those used for FineWeb, RefinedWeb, and LAION-5B—by providing a standardized way to filter, score, and deduplicate massive amounts of text, image, and video data.

How it works

Built on Ray, the system uses a distributed architecture that separates processing into stages (CPU and GPU pools). It employs a pipeline-parallel design where data batches are passed through a series of "operators" (refiners, filters, and deduplicators). To maximize performance, it uses Rust-accelerated operators for CPU-heavy tasks and GPU-optimized operators for embedding extraction and scoring.

Who it’s for

It is designed for AI researchers and data engineers building foundation models who need to curate high-quality training sets from raw, noisy web-scale data across multiple modalities.

Highlights

• Multimodal Support: Dedicated operators for text, images, and video, including CLIP/SigLIP embedding extraction.
• High Performance: Uses Rust for acceleration and vLLM for high-throughput offline LLM synthesis.
• Reproducible Recipes: Implements specific filtering and scoring logic from major papers like RefinedWeb and Z-Image.
• LLM Synthesis: Integrated support for both online (API-based with account/proxy rotation) and offline (local GPU) data synthesis.
• Comprehensive Reporting: Generates interactive HTML reports with data quality funnels and Sankey diagrams to visualize data flow.