kubeflow/pipelines

What it solves

Kubeflow Pipelines simplifies the deployment and orchestration of end-to-end machine learning workflows on Kubernetes, making them simple, portable, and scalable. It addresses the challenge of managing complex ML pipelines that need to be repeatable and scalable across different environments.

How it works

It provides a Python SDK that allows users to build reusable components and end-to-end workflows. The service orchestrates these workflows on Kubernetes, using Argo Workflows by default to manage the resources. It is container-runtime agnostic, meaning it can run on any Kubernetes cluster regardless of the specific container runtime used.

Who it’s for

Data scientists and ML engineers who need to build, deploy, and experiment with scalable, portable ML workflows on Kubernetes.

Highlights

• End-to-end orchestration: Simplifies the orchestration of complete ML pipelines.
• Easy experimentation: Facilitates the the trial and management of various ideas and techniques.
• Easy re-use: Allows components and pipelines to be reused to quickly assemble solutions without rebuilding from scratch.
• Kubernetes native: Built specifically for making ML deployments on Kubernetes simple and portable.