kestra: an event-driven orchestration platform for data and AI workflows using declarative YAML

What it solves

Kestra simplifies the orchestration of complex data, AI, and infrastructure workflows. It eliminates the need to manage fragmented automation tools by providing a single platform to handle both scheduled and real-time, event-driven pipelines while maintaining a "code-first" approach to infrastructure.

How it works

Kestra uses a declarative YAML interface to define "Flows" (workflows) composed of "Tasks" (individual units of work). These flows can be built visually via a drag-and-drop UI or written in a code editor with real-time validation. The platform is language-agnostic, using a vast plugin ecosystem to execute scripts in Python, Node.js, Go, R, and Shell, or to interact with cloud services (AWS, GCP, Azure) and big data tools like Apache Spark.

Who it’s for

It is designed for data engineers, AI practitioners, and DevOps professionals who need to build scalable, resilient pipelines that integrate multiple languages and cloud services while keeping their orchestration logic in version control.

Highlights

• Declarative YAML: Define workflows as code for easy versioning and CI/CD integration.
• Event-Driven Architecture: Trigger flows based on schedules or real-time events like file arrivals or message bus signals (Kafka, Redis, etc.).
• Language Agnostic: Run code in any language via plugins and execute tasks locally, on remote servers, or in Docker/Kubernetes containers.
• Visual Orchestration: Build and visualize workflows as Directed Acyclic Graphs (DAGs) using an intuitive UI with auto-completion and syntax highlighting.