PrefectHQ/prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

What it solves

Prefect simplifies the process of turning Python scripts into production-ready data pipelines. It addresses the challenges of building resilient and dynamic workflows that need to handle scheduling, error recovery, and complex dependencies without requiring a complete rewrite of existing code.

How it works

Users apply @flow and @task decorators to their Python functions to define the structure and logic of their workflow. These flows can then be deployed as scheduled tasks (using cron expressions) or triggered by specific events. Activity is monitored via a self-hosted Prefect server or a managed cloud dashboard, which tracks execution and provides visibility into the pipeline's health.

Who it’s for

It is designed for data teams and engineers who need to automate data processes and build reliable pipelines that can react to changes and recover from unexpected failures.

Highlights

  • Resilient Automation: Built-in support for retries, caching, and complex branching logic.
  • Flexible Execution: Supports both self-hosted server instances and a managed cloud version.
  • Dynamic Triggering: Workflows can be run on a fixed schedule or in response to real-time events.
  • Low Friction: Allows developers to elevate standard Python scripts into orchestrated workflows with minimal code changes.

Related

  • Project
    PrefectHQ/marvin

    Marvin is a Python library that simplifies working with language models by providing tools to extract structured data, classify information, and execute tasks using custom functions.

  • Project
    kedro-org/kedro

    A Python framework for creating production-ready data engineering and data science pipelines using software engineering best practices.

  • Project
    Avaiga/taipy

    A Python framework for data scientists and ML engineers to build and deploy production-ready AI and data-driven web applications without needing frontend expertise.

  • Project
    run-llama/llama-agents

    An open-source framework for building and shipping document-centric agents using event-driven orchestration in Python.

  • Project
    conductor-oss/conductor

    An open-source durable execution platform for microservices and AI agents that turns complex workflow graphs into persisted, inspectable, and recoverable executions.