kedro-org/kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

What it solves

Kedro is designed to move data science and data engineering projects from one-off scripts and Jupyter notebooks into production-ready, maintainable, and reproducible code. It addresses the shortcomings of "glue-code" and helps teams with varying software engineering skills collaborate more effectively by enforcing modularity and separation of concerns.

How it works

Kedro provides a structured framework that applies software engineering best practices to data pipelines. It uses a project template based on Cookiecutter Data Science to standardize project layout. It features a Data Catalog to abstract the load/save operations for various file formats and cloud stores, and a Pipeline abstraction that automatically resolves dependencies between Python functions to create a modular workflow.

Who it’s for

Data scientists and data engineers who need to build scalable, reproducible analytics pipelines and real-world ML applications that handle large amounts of raw data.

Highlights

  • Data Catalog: Lightweight connectors for saving and loading data across local, network, and cloud object stores with built-in versioning.
  • Pipeline Abstraction: Automatic dependency resolution between Python functions and visualization via Kedro-Viz.
  • Flexible Deployment: Support for single or distributed machines, as well as orchestration tools like Argo, Prefect, Kubeflow, AWS Batch, and Databricks.
  • Standardized Coding Standards: Integrated support for pytest, Sphinx, and ruff for testing, documentation, and linting.

Related

  • Project
    kedro-org/kedro-viz

    An interactive visualization tool for Kedro data science pipelines that transforms complex pipeline code into searchable, interactive diagrams.

  • Project
    tidymodels/recipes

    A framework for feature engineering and data preprocessing in R that uses pipeable sequences of steps to prepare data for machine learning models.

  • Project
    drivendataorg/cookiecutter-data-science

    Cookiecutter Data Science (CCDS) is a CLI tool that creates a standardized, best‑practice project skeleton for data‑science work, including folders for raw/processed data, notebooks, a Python package with training/prediction scripts, docs, and a Makefile. Install via pipx, run `ccds`, answer prompts, and you get a ready‑to‑use, reproducible project layout.

  • Project
    kubeflow/kale

    A tool that converts Jupyter Notebooks into production-ready Kubeflow Pipelines by tagging cells, removing the need to rewrite code using the KFP SDK.

  • Project
    quixio/quix-streams

    An open-source Python framework for real-time data engineering and machine learning on Apache Kafka streams, providing a tabular DataFrame API for reliable ETL pipelines.