apache/airflow

What is Apache Airflow?

Apache Airflow is a platform for creating, scheduling, and monitoring workflows — but instead of clicking through a GUI, you write your workflows as code. That makes them easier to maintain, version, test, and share with others.

In Airflow, a workflow is called a DAG (Directed Acyclic Graph). A DAG defines a set of tasks and the order they must run in. The Airflow scheduler automatically executes those tasks on a group of worker machines, respecting the dependencies you've defined. You also get:

• Command-line tools for managing DAGs
• A web interface to see pipelines running in production, monitor progress, and troubleshoot problems

What is it good for?

Airflow works best with workflows that are mostly static and slowly changing — where the structure stays similar from one run to the next. It's commonly used for:

• Data processing pipelines (but with a caveat, see below)
• Machine learning workflows — training, retraining, evaluation, and deployment
• AI/LLM pipelines — coordinating steps like data prep, tool calls, model invocation, and evaluation (Airflow orchestrates the steps; it doesn't act as the AI agent itself)

Airflow has some strong opinions:

• Tasks should be idempotent — running a task twice should give the same result and not create duplicate data.
• Tasks should not pass large amounts of data to each other. They can pass small metadata using a feature called XCom, but heavy data should be handled by external services.
• It's not a streaming solution, but it's often used to process real-time data in batches (pulling data off streams periodically).

Similar projects include Luigi, Oozie, and Azkaban.

Core principles

• Dynamic — Pipelines are defined in code, so you can generate DAGs programmatically and parameterize them.
• Extensible — Airflow has many built-in operators (task types) and can be extended to fit your needs.
• Flexible — Uses the Jinja templating engine, allowing rich customization.

Requirements

Airflow is tested with specific versions of:

• Python — e.g., 3.10–3.14 for the stable 3.3.2 release
• Platform — AMD64/ARM64
• Kubernetes — various versions (e.g., 1.30–1.35 for stable)
• Databases — PostgreSQL, MySQL, SQLite (SQLite only for tests, not production)

Important notes:

• MariaDB is not tested or recommended.
• Airflow runs on POSIX-compliant operating systems (Linux, macOS). On Windows, you can use WSL2 or Linux containers.
• The official Docker image uses Debian Bookworm.

How to install it

Airflow is published on PyPI as the apache-airflow package. Installation can be tricky because Airflow is both a library and an application — it keeps dependencies open, which means pip install apache-airflow alone might not always work. The recommended approach is to use constraint files that pin known-good dependency versions:

pip install 'apache-airflow==3.3.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.10.txt"

You can also add extras, e.g., apache-airflow[postgres,google].

Only pip and uv are officially supported for installation. Tools like Poetry or pip-tools are not supported (though you can use them if you convert the constraint files).

Other ways to get Airflow

These are "convenience" packages — not official ASF releases, but built from official sources:

• PyPI releases — via pip
• Docker Images — for Kubernetes, docker-compose, etc.
• GitHub tags — to get the source code

The user interface

The Airflow web UI includes several views:

• DAGs — overview of all workflows
• Assets — view of data assets and their dependencies
• Grid — a time-based grid view of a DAG's runs
• Graph — visualization of a DAG's task dependencies and current status
• Home — summary statistics of your environment
• Backfill — run a DAG for a specific date range
• Code — quickly view a DAG's source code

Versioning

Since Airflow 2.0.0, the project follows strict Semantic Versioning (SemVer). This applies to core Airflow and to provider packages separately.

Project governance

Apache Airflow is an Apache Software Foundation project. Official source code releases:

• Follow the ASF Release Policy
• Are cryptographically signed
• Are voted on by the project's PMC members

There's also a large community — you can join the Slack channel, contribute code, and more (details in the README's contributing sections).

---

This explanation is based solely on the content of the project's README.