apache/hamilton
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
What it solves
Apache Hamilton is a lightweight Python library designed to structure and manage data transformation Directed Acyclic Graphs (DAGs). It addresses the challenge of moving data projects from proof-of-concept to production by providing a standardized way to organize transformations, separating the definition of the dataflow from its execution, and reducing code redundancy.
How it works
Users define their DAG by writing regular Python functions where the function parameters specify the dependencies. Apache Hamilton automatically builds the DAG from these definitions. The library separates the "definition" (the logic) from the "execution" (the driver), allowing the same DAG to be portable across different environments such as scripts, notebooks, Airflow pipelines, or FastAPI servers.
Who it’s for
It is built for data teams—including data scientists, engineers, and ops—who need to build maintainable ETL pipelines, ML workflows, LLM applications, and RAG systems.
Highlights
- Portable Transformations: DAGs are independent of infrastructure or orchestration tools, allowing for local development and reuse across contexts.
- Detailed Observability: Includes a dedicated UI for visualizing, cataloging, and monitoring execution with lineage and tracing.
- Data Validation: Built-in support for output validation via
@check_outputand schema validation for dataframe-like objects. - Modular Design: Encourages the use of multiple Python modules to assemble pipelines, keeping code DRY and unit-testable.