dagster-io/dagster

What it solves

Dagster is a cloud-native data pipeline orchestrator designed to simplify the development and maintenance of data assets, such as machine learning models, tables, and reports. It addresses the complexity of managing the entire data development lifecycle—from local development and testing to production deployment—by providing integrated lineage and observability.

How it works

Users define the data assets they want to create as Python functions. Dagster then manages the execution of these functions at the optimal time to keep assets up-to-date. It uses a declarative programming model to track dependencies and build a graph of assets, which can be visualized and managed via a web UI.

Who it’s for

This tool is for data practitioners and engineers who need to build robust, scalable data pipelines and maintain control over complex data assets across different environments (local, staging, and production).

Highlights

• Integrated lineage and observability for centralized metadata and diagnostics.
• Declarative approach to defining data assets using Python functions.
• Robust multi-tenant orchestration engine that scales technically and organizationally.
• Extensive library of integrations with popular data tools.
• Strong focus on testability and CI/CD best practices.