data-prep-kit/data-prep-kit

What it solves

Data-Prep-Kit simplifies the complex process of preparing unstructured data for Large Language Models (LLMs). It provides a scalable way to cleanse, transform, and enrich data used for pre-training, fine-tuning, instruct-tuning, or building Retrieval Augmented Generation (RAG) applications.

How it works

The kit consists of a growing library of modular transforms that can be applied to natural language, code, and images. These transforms are built on Python and Ray, allowing the processing to scale from a single laptop to a full data center. The framework supports various file formats including Parquet, ZIP, NDJSON, and JSONL. For complex workflows, transforms can be deployed as jobs on Kubernetes clusters or sequenced using Tekton pipelines.

Who it’s for

LLM application developers who need to process large volumes of unstructured data for model training or RAG pipelines.

Highlights

• Multi-modal support: Handles natural language, code, and image data.
• Scalable architecture: Supports both Python-only and Ray runtimes for massive scaling.
• ** uma modular transform library**: Includes tools for data ingestion (e.g., HTML to Parquet), deduplication (exact and fuzzy), quality annotation, PII redaction, and language identification.
• Enterprise-ready deployment: Compatible with Kubernetes and Tekton pipelines for automation.