argoproj/argo-workflows

Workflow Engine for Kubernetes

What it solves

Argo Workflows addresses the challenge of orchestrating complex, parallel jobs on Kubernetes. It eliminates the overhead of legacy VM and server-based environments by providing a container-native way to manage multi-step compute-intensive tasks, such as those required for machine learning and data processing.

How it works

Implemented as a Kubernetes Custom Resource Definition (CRD), Argo Workflows allows users to define workflows where each step is a container. These workflows can be modeled as a simple sequence of tasks or as a Directed Acyclic Graph (DAG) to capture complex dependencies between steps. It provides a server interface with REST API (HTTP and GRPC) and supports various client libraries in Java, Golang, Python (via Hera), and Typescript (via Juno).

Who it’s for

It is designed for developers and platform engineers who need to run scalable, cloud-agnostic batch processing, machine learning pipelines, or CI/CD automation on any Kubernetes cluster.

Highlights

  • Container-native orchestration: Every step in a workflow is a container, designed specifically for Kubernetes.
  • Flexible workflow modeling: Supports both DAGs and sequential steps.
  • Rich feature set: Includes a UI for visualization, artifact support (S3, GCS, Azure, etc.), scheduled cron workflows, and loop/conditional logic.
  • Robust execution control: Features step-level timeouts, retries, resubmissions, and the ability to suspend and resume workflows.
  • Broad ecosystem: Used by major projects like Kubeflow Pipelines and Metaflow for AI/ML orchestration.

Related

  • Project
    kubeflow/pipelines

    A platform for building and deploying reusable, scalable end-to-end machine learning workflows on Kubernetes.

  • Project
    dagucloud/dagu

    A local-first, self-hosted workflow engine that turns existing scripts and containers into observable DAGs using a single binary and YAML configuration.

  • Project
    kubeflow/kubeflow

    Kubeflow is a Kubernetes-native foundation of tools for building AI platforms, providing a modular ecosystem to manage the entire AI lifecycle.

  • Project
    raids-lab/crater

    A Kubernetes-native control plane for shared AI computing clusters that simplifies GPU resource governance, workload scheduling, and asset management.

  • Project
    kubernetes-sigs/jobset

    A Kubernetes-native API for managing groups of Jobs as a single unit, specifically designed to simplify the deployment of distributed AI/ML training and HPC workloads.