pdpipe/pdpipe

What it solves

pdpipe simplifies the process of creating data preprocessing pipelines for pandas DataFrames. It replaces fragmented, repetitive data cleaning and transformation steps with a structured, chainable sequence of operations.

How it works

The library provides a set of ready-made "stages" (transformation steps) that can be chained together. Users define a pipeline by linking these stages, and then apply the entire pipeline to a DataFrame. It supports common tasks like dropping columns, one-hot encoding, and mapping values, and can integrate with scikit-learn transformers while maintaining DataFrame column context.

Who it’s for

Data scientists and analysts who use pandas for data preparation and want a more organized, reproducible way to handle feature engineering and preprocessing.

Highlights

• Ready-made stages: Includes built-in tools for column dropping, value filtering, one-hot encoding, and feature generation.
• Visual Diagnostics: Can generate Graphviz DOT diagrams of the pipeline for visualization.
• Dry-run Diagnostics: Features a .trace() method for structured per-stage diagnostics.
• Parallel Execution: Supports optional thread-based parallel execution for row and column applications.
• Scikit-learn Integration: Wraps scikit-learn transformers to ensure they work seamlessly with pandas DataFrames.