kestra-io/kestra

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

What it solves

Kestra is an open-source orchestration platform designed to simplify the management of complex data, AI, and infrastructure workflows. It eliminates the fragmentation between scheduled tasks and real-time event-driven automation by providing a unified, declarative interface for building and managing pipelines.

How it works

Kestra uses a declarative YAML interface to define workflows (flows), which are composed of individual units of work (tasks). These flows can be triggered by schedules or external events. The platform is language-agnostic, allowing users to run scripts in Python, Node.js, R, Go, and Shell via a rich plugin ecosystem.

Architecturally, Kestra 2.0 separates the control plane from the data plane using a gRPC worker-controller, enabling high scalability and the ability to run workers in different regions or restricted networks. It supports multiple ways to build flows: a YAML editor, a No-Code visual editor, and an AI Copilot.

Who it’s for

Kestra is built for data engineers, DevOps professionals, and AI developers who need to orchestrate complex pipelines across diverse environments (local, Docker, Kubernetes, or cloud VMs) and integrate various cloud services and AI models.

Highlights

  • AI-Native Orchestration: Includes first-class support for AI Agents, RAG workflows, and an MCP server to expose flows as tools for AI agents.
  • AI-Assisted Building: A built-in AI Copilot and No-Code editor that stay in sync with the underlying YAML code.
  • Infrastructure as Code: Full Git version control integration and an official Terraform provider for managing resources.
  • Extensible Plugin System: Hundreds of built-in plugins for databases, cloud storage, and APIs, with the ability to develop custom plugins.
  • Resilient Workflow Components: Features like checks (preconditions), quotas (execution limits), retries, and conditional branching to ensure pipeline stability.

Related

  • Project
    deepset-ai/haystack

    An open-source AI orchestration framework for building production-ready RAG systems, autonomous agents, and LLM applications in Python.

  • Project
    mastra-ai/mastra

    A TypeScript framework for building and scaling AI agents and applications, providing tools for model routing, graph-based workflows, and production observability.

  • Project
    dagster-io/dagster

    A cloud-native data pipeline orchestrator for managing the development and maintenance of data assets like machine learning models and tables with integrated lineage and observability.

  • Project
    heymrun/heym

    An AI-native workflow automation platform for building, visualizing, and running multi-agent orchestrations and RAG pipelines via a no-code visual canvas.

  • Project
    dagucloud/dagu

    A local-first, self-hosted workflow engine that turns existing scripts and containers into observable DAGs using a single binary and YAML configuration.