kestra-io/kestra

What it solves

Kestra is an open-source orchestration platform designed to simplify the management of complex data, AI, and infrastructure workflows. It eliminates the chaos of fragmented automation by providing a unified, event-driven system that handles both scheduled and real-time tasks across different environments and languages.

How it works

Kestra uses a declarative YAML interface to define workflows (called "flows"), which are composed of individual units of work (tasks). These flows are managed as code, allowing for version control via Git and CI/CD integration. The platform provides a visual UI with a drag-and-drop editor and a live topology view (DAG) to build and visualize these pipelines.

Who it’s for

It is built for data engineers, AI practitioners, and DevOps engineers who need to automate scripts, manage data pipelines, and orchestrate microservices without being locked into a specific programming language.

Highlights

• Language Agnostic: Supports running scripts in Python, Node.js, R, Go, Shell, and more via a rich plugin ecosystem.
• Event-Driven Architecture: Can trigger workflows based on schedules or real-time events from external systems like Kafka, Redis, and cloud messaging services.
• Infrastructure as Code: Integrates directly with Git and offers a Terraform provider for managing resources.
• Visual Orchestration: Combines a code editor with a real-time visual DAG and drag-and-drop task arrangement.
• Highly Scalable: Designed for high availability and fault tolerance to handle millions of workflows.