Netflix/maestro

What it solves

Maestro is a general-purpose workflow orchestrator designed to handle data and machine learning workflows at massive scale. It provides a fully managed "workflow-as-a-service" (WAAS) that allows users to schedule and manage millions of jobs daily while maintaining strict service level objectives (SLOs) even during traffic spikes.

How it works

Maestro operates as a server-based orchestrator that manages the execution of workflows (represented as Directed Acyclic Graphs or DAGs). Users can define workflows via JSON or a Python SDK, push them to the Maestro server, and trigger their execution. The system is highly extensible, supporting integrations with AWS (via SQS/SNS) and Kubernetes, and includes an extensions module for specialized functionality like "foreach" step flattening views.

Who it’s for

It is built for technical roles managing large-scale data pipelines, including data scientists, data engineers, machine learning engineers, software engineers, and business analysts.

Highlights

• Massive Scalability: Capable of scheduling hundreds of thousands of workflows and millions of jobs per day.
• Managed Service Model: Operates as a workflow-as-a-service (WAAS) for simplified user access.
• Flexible Deployment: Supports local execution, AWS modules, and Kubernetes.
• Python SDK: Provides a dedicated client for creating and pushing workflows programmatically.