table of content

Part 1 Getting started

1 Meet Apache Airflow

1.1 Introducing data pipelines

1.1.1 Drawing a pipeline as a graph

1.1.2 Executing a pipeline graph

1.1.3 Pipeline graphs vs. sequential scripts

1.1.4 Running pipelines using workflow managers

1.2 Introducing Airflow

1.2.1 Defining pipelines flexibly in (Python) code

1.2.2 Integrating with external systems

1.2.3 Scheduling and executing pipelines

1.2.4 Monitoring and handling failures

1.2.5 Incremental loading and backfilling

1.3 When to use Airflow

1.3.1 Reasons to choose Airflow

1.3.2 Reasons not to choose Airflow

1.4 The rest of this book

2 Anatomy of an Airflow DAG

2.1 Collecting data from numerous sources

2.2 Writing your first Airflow DAG

2.2.1 Tasks vs. operators

2.2.2 Running arbitrary Python code

2.3 Running a DAG in Airflow

2.3.1 Running Airflow in a Python environment

2.3.2 Running Airflow with Docker

2.3.3 Inspecting the DAG in Airflow

2.4 Running at regular intervals

2.5 Handling failing tasks

2.6 DAG versioning

3 Time-based scheduling

3.1 Processing user events

3.2 The basic components of an Airflow schedule

3.3 Running regularly using trigger-based schedules

3.3.1 Defining a daily schedule

3.3.2 Using cron expressions

3.3.3 Using shorthand expressions

3.3.4 Using frequency-based timetables

3.3.5 Summarizing trigger timetables

3.4 Incremental processing with data intervals

3.4.1 Processing data incrementally

3.4.2 Defining incremental schedules with data intervals

3.4.3 Defining intervals using frequencies

3.4.4 Summarizing interval-based schedules

3.5 Handling irregular intervals

3.6 Managing backfilling of historical data

3.7 Designing well-behaved tasks

3.7.1 Atomicity

3.7.2 Idempotency

4 Asset-aware scheduling

4.1 Challenges of scaling time-based schedules

4.2 Introducing asset-aware scheduling

4.3 Producing asset events

4.4 Consuming asset events

4.5 Adding extra information to events

4.6 Skipping updates

4.7 Consuming multiple assets

4.8 Combining time- and asset-based schedules

5 Templating tasks using the Airflow context

5.1 Inspecting data for processing with Airflow

5.2 Task context and Jinja templating

5.2.1 Templating operator arguments

5.2.2 Templating the PythonOperator

5.2.3 Passing additional variables to the PythonOperator

5.2.4 Inspecting templated arguments

5.3 What is available for templating

5.4 Bringing it all together

6 Defining dependencies between tasks

6.1 Basic dependencies

6.1.1 Linear dependencies

6.1.2 Fan-in/fan-out dependencies

6.2 Branching

6.2.1 Branching within tasks

6.2.2 Branching within the DAG

6.3 Conditional tasks

6.3.1 Conditions within tasks

6.3.2 Making tasks conditional

6.3.3 Using built-in operators

6.4 Exploring trigger rules

6.4.1 What is a trigger rule?

6.4.2 The effect of failures

6.4.3 Other trigger rules

6.5 Sharing data between tasks

6.5.1 Sharing data using XComs

6.5.2 When and when not to use XComs

6.5.3 Using custom XCom backends

6.5.4 XCom cleanup

6.6 Chaining Python tasks with the Taskflow API

6.6.1 Simplifying Python tasks with the Taskflow API

6.6.2 Using the Taskflow API to define a new DAG

6.6.3 When and when not to use the Taskflow API

Part 2 Beyond the basics

7 Triggering workflows with external input

7.1 Polling conditions with sensors

7.1.1 Polling custom conditions

7.1.2 Working with sensors outside the happy flow

7.2 Starting workflows with the REST API and CLI

7.3 Triggering workflows with messages

8 Communicating with external systems

8.1 Installing additional operators

8.2 Developing a machine learning model

8.2.1 Use case: Classifying handwritten digits

8.2.2 Setting up the pipeline

8.2.3 Developing locally with external systems

8.3 Moving data from between systems

8.3.1 Use case: Analyzing Airbnb listings

8.3.2 Implementing a PostgresToS3Operator

8.3.3 Outsourcing the heavy work

9 Extending Airflow with custom operators and sensors

9.1 Starting with a PythonOperator

9.1.1 Simulating a movie-rating API

9.1.2 Fetching ratings from the API

9.1.3 Building the actual DAG

9.2 Building a custom hook

9.2.1 Designing a custom hook

9.2.2 Building a DAG with the MovielensHook

9.3 Building a custom operator

9.3.1 Defining a custom operator

9.3.2 Building an operator to fetch ratings

9.4 Building custom sensors

9.5 Building a custom deferrable operator

9.5.1 Executing asynchronous tasks using the triggerer

9.5.2 Running the Movielens sensor asynchronously

9.6 Packaging the components

9.6.1 Bootstrapping a Python package

9.6.2 Installing the package

9.6.3 Sharing the package with others

10 Testing

10.1 Getting started with testing

10.1.1 Integrity testing all DAGs

10.1.2 Setting up a CI/CD pipeline

10.1.3 Writing unit tests

10.1.4 Creating the pytest project structure

10.1.5 Testing with files on disk

10.2 Working with external systems

10.3 Using tests for development

10.4 Testing complete DAGs

10.4.1 Using dag.test() to test the whole DAG

10.4.2 Emulating production environments with Whirl

10.4.3 Creating DTAP environments

11 Running tasks in containers

11.1 Challenges of different operators

11.1.1 Operator interfaces and implementations

11.1.2 Complex and conflicting dependencies

11.1.3 Moving toward a generic operator

11.2 Introducing containers

11.2.1 What are containers?

11.2.2 Running a first Docker container

11.2.3 Creating a Docker image

11.2.4 Persisting data using volumes

11.3 Containers and Airflow

11.3.1 Tasks in containers

11.3.2 Why use containers?

11.4 Running tasks in Docker

11.4.1 Introducing the DockerOperator

11.4.2 Creating container images for tasks

11.4.3 Building a DAG with Docker tasks

11.4.4 Docker-based workflow

11.5 Running tasks in Kubernetes

11.5.1 Introducing Kubernetes

11.5.2 Setting up Kubernetes

11.5.3 Using the KubernetesPodOperator

11.5.4 Diagnosing Kubernetes-related issues

11.5.5 Differences between Kubernetes- and Docker-based workflows

Part 3 Airflow in practice

12 Best practices

12.1 Writing clean DAGs

12.1.1 Using style conventions

12.1.2 Managing credentials centrally

12.1.3 Specifying configuration details consistently

12.1.4 Avoiding computation in your DAG definition

12.1.5 Using factories to generate common patterns

12.1.6 Grouping related tasks with task groups

12.1.7 Being explicit when specifying your DAG schedule

12.1.8 Using Dynamic Task Mapping to generate tasks dynamically

12.2 Designing reproducible tasks

12.2.1 Requiring tasks to be idempotent

12.2.2 Ensuring that task results are deterministic

12.2.3 Designing tasks using functional paradigms

12.3 Handling data efficiently

12.3.1 Limiting the amount of data being processed

12.3.2 Loading/processing data incrementally

12.3.3 Caching intermediate data

12.3.4 Avoiding storing data on local filesystems

12.3.5 Offloading work to external/source systems

12.4 Managing concurrency using pools

13 Project: Finding the fastest way to get around NYC

13.1 Use case: Investigating traffic in New York City

13.2 Understanding the data

13.2.1 Yellow Cab file share

13.2.2 Citi Bike REST API

13.2.3 Deciding on a plan of approach

13.3 Extracting the data

13.3.1 Downloading Citi Bike data

13.3.2 Downloading Yellow Cab data

13.4 Applying similar transformations to data

13.5 Structuring a data pipeline

13.6 Developing idempotent data pipelines

14 Project: Keeping family traditions alive with Airflow and generative AI

14.1 Use case: Bringing family recipes to life

14.2 Fine-tuning an existing LLM

14.3 RAG to the rescue

14.4 Uploading recipes to the Recipe Vault UI

14.5 Preprocessing the recipes with DockerOperator

14.6 Creating a collection to store our recipes

14.6.1 Defining how to vectorize our text

14.6.2 Creating a schema for the collection

14.6.3 Preparing our collection of recipes

14.7 Updating and creating new records in the vector database

14.8 Deleting outdated records from the vector database

14.9 Adding recipes to the vector database

14.10 RAG in action

14.10.1 The R is for retrieving

14.10.2 Structuring our questions with prompt templates

14.10.3 Searching for recipes

Part 4 Airflow in production

15 Operating Airflow in production

15.1 Revisiting the Airflow architecture

15.2 Choosing the executor

15.2.1 Overview of executor types

15.2.2 Which executor is right for you?

15.2.3 Installing each executor

15.3 Configuring the metastore

15.4 Configuring the scheduler

15.4.1 Configuring scheduler components

15.4.2 Running multiple schedulers

15.4.3 Configuring system performance

15.4.4 Controlling the maximum number of running tasks

15.5 Configuring the DAG processor manager

15.6 Capturing logs

15.6.1 Capturing API server output

15.6.2 Capturing scheduler output

15.6.3 Capturing task logs

15.6.4 Sending logs to remote storage

15.7 Visualizing and monitoring Airflow metrics

15.7.1 Collecting metrics from Airflow

15.7.2 Configuring Airflow to send metrics

15.7.3 Configuring Prometheus to collect metrics

15.7.4 Creating dashboards with Grafana

15.7.5 What should you monitor?

15.8 Setting up alerts

15.9 Scaling Airflow beyond a single instance

16 Securing Airflow

16.1 Role-based access in the Airflow UI

16.1.1 Adding users

16.1.2 Configuring the RBAC interface

16.2 Encrypting data at rest

16.3 Connecting with a directory service

16.3.1 Understanding LDAP

16.3.2 Fetching users from an LDAP service

16.4 Encrypting traffic to the web server

16.4.1 Understanding HTTPS

16.4.2 Configuring a certificate for HTTPS

16.5 Fetching credentials from secrets-management systems

17 Airflow deployment options

17.1 Managed Airflow

17.1.1 Astronomer

17.1.2 Google Cloud Composer

17.1.3 Amazon Managed Workflows for Apache Airflow

17.2 Airflow on Kubernetes

17.2.1 Preparing the Kubernetes cluster

17.2.2 Connecting to your Kubernetes cluster

17.2.3 Deploying with the Apache Airflow Helm Chart

17.2.4 Changing the default deployment configuration

17.2.5 Changing the apiserver secret key

17.2.6 Using an external database for Airflow metadata

17.2.7 Deploying DAGs

17.2.8 Deploying a Python library

17.2.9 Configuring the executor(s)

17.3 Choosing a deployment strategy

Appendices

Appendix A: Running code samples

A.1 Understanding the code structure

A.2 Running the examples

A.2.1 Starting the Docker environment

A.2.2 Inspecting running services

A.2.3 Tearing down the environment

Appendix B: Prometheus metric mapping

Overview

11 Running tasks in containers

This chapter explains how running tasks in containers can simplify Airflow deployments that otherwise rely on many specialized operators and sprawling, conflicting dependencies. It motivates a move toward a generic, container-based execution model where each task runs inside an isolated image, giving a uniform interface for orchestration while reducing the cognitive load of learning multiple operator APIs. The chapter introduces containers and their lightweight isolation compared to virtual machines, then shows how Airflow can execute containerized tasks via Docker or Kubernetes to improve reliability, portability, and maintainability.

With Docker, the chapter walks through the lifecycle: building images that package task code and dependencies, running containers with commands and arguments, and persisting outputs using mounted volumes or networked storage. It demonstrates converting task logic into small CLI-style scripts, assembling minimal images, and invoking them from the DockerOperator, emphasizing a consistent pattern for inputs, outputs, and logging. A recommended workflow separates image development from DAG authoring: images are built and pushed (often via CI/CD), then referenced in DAGs, enabling better testability, safer dependency boundaries, and easier reuse across tasks.

For scalability and cluster scheduling, the chapter transitions to Kubernetes, outlining core concepts (control plane, nodes, pods) and resources for storage and services. It shows how to adapt the same tasks to the KubernetesPodOperator, including mounting persistent volumes, setting namespaces, and controlling image pull policies, as well as diagnosing stuck or pending pods using kubectl commands and events. Key differences from the Docker-only approach include task execution on separate cluster nodes, storage provisioned through Kubernetes primitives, and richer resource management. The result is a consistent, container-first pattern that scales from a single host to a cluster while preserving clean dependency isolation and a uniform task interface.

Illustration of our movie recommender DAG. The DAG fetches movie recommendations, uses them to rank movies, and stores the result in a database. Each of these steps involves a different operator, thus adding complexity to the development and maintenance of the DAG.

Complex and conflicting dependencies between Airflow tasks or DAGs. Running many DAGs in a single environment can lead to conflicts when DAGs depend on different versions of the same (or related) packages. Python in particular does not support installing different versions of the same package in the same environment. This means that any conflicts in packages (right) would need to be resolved by rewriting the DAGs (or their dependencies) to use the same package versions.

Comparison between virtual machines (VMs) and containers. Note that containers are much more lightweight as they don’t require running a full guest OS for each application.

Example output from the wttr-example container for Amsterdam

Docker version of the recommender DAG from figure 11.1

Managing dependencies across different tasks using containers

Illustration of what happens when a task is executed using the DockerOperator. The image registry stores a collection of Docker images This can be a private registry (containing our own images), or a public registry like DockerHub (which is used by default when fetching images). Images are cached locally when fetched so that you only have to do it once (barring any updates to the image).

The Docker-based DAG in the Airflow UI

Common workflow for working with Docker images in Airflow

High-level overview of Kubernetes

Several successful runs of the recommender DAG based on the KubernetesPodOperator

Workflow for building DAGs using the KubernetesPodOperator

Summary

Airflow deployments can be difficult to manage if they involve many different operators, as this requires knowledge of the different APIs and complicates debugging and dependency management.
One way of tackling this issue is to use container technologies such as Docker to encapsulate your tasks inside container images and run these images from within Airflow.
This containerized approach has several advantages, including easier dependency management, a more uniform interface for running tasks, and improved testability of tasks.
Data can be shared between containers using volume mounts.
Using the DockerOperator, you can run tasks in container images directly using Docker, similar to the docker run CLI command.
You can use the KubernetesPodOperator to run containerized tasks in pods on a Kubernetes cluster.
Kubernetes allows you to scale your containerized tasks across a compute cluster, which provides (among other things) greater scalability and more flexibility in terms of computing resources.

FAQ

Why can using many different Airflow operators be challenging?

Each operator has its own interface, configuration, and dependency set. In larger deployments with many DAGs, this leads to a steep learning curve, more places for bugs, and a higher maintenance burden. It also increases the chance of dependency conflicts on worker/scheduler environments because all operator-specific packages must coexist there.

How do containers help simplify dependency management in Airflow DAGs?

Containers isolate task dependencies inside images. Each task can ship exactly what it needs (Python libs, system packages) without polluting the Airflow worker environment. This avoids version conflicts across tasks and reduces the security and maintenance surface on workers to just what’s required to run the container engine and the container-based operator.

What’s the key difference between containers and virtual machines for this use case?

VMs virtualize a full guest OS per application, which is heavy. Containers virtualize at the host OS kernel level, so multiple apps can run with isolation without their own OS. For Airflow tasks, this means faster startup, lower resource use, and simpler packaging and distribution of task runtimes.

How do I run a simple task in Docker and persist outputs?

- Build or pull an image that contains your code and dependencies. - Run the container with a volume mount to a host path (for example, map host “data” to container “/data”). - Have your task write to the mounted path so files persist after the container exits. This mirrors how you’ll mount volumes in Airflow’s DockerOperator.

What is the DockerOperator and when should I use it?

DockerOperator (from apache-airflow-providers-docker) runs a container on the worker host similar to “docker run”. Use it when a single machine (the Airflow worker host) is sufficient, you want uniform task interfaces, and you prefer per-task images that encapsulate their dependencies. You’ll provide the image, command/args, and any mounts (for example, host path → “/data”).

How do I pass data between containerized tasks in Airflow?

- Use a shared storage location both tasks can access. - With DockerOperator, mount the same host directory into each container and read/write files there. - With KubernetesPodOperator, attach a PersistentVolume (via a PersistentVolumeClaim) to both pods at a common mount point. - Alternatively, use network/cloud storage (S3, GCS) accessible by both tasks.

What’s the recommended workflow for building Docker-based DAGs?

- Develop task code as CLI scripts or entrypoints. - Create Dockerfiles per task image and build them locally or in CI. - Push images to a registry. - In your DAG, replace task implementations with DockerOperators referencing those images. - Schedule/run the DAG; workers pull images and execute containers with the provided args and mounts.

When should I use the KubernetesPodOperator instead of DockerOperator?

Use KubernetesPodOperator (from apache-airflow-providers-cncf-kubernetes) when you need cluster-level scheduling, autoscaling, or specialized resources (CPU/memory requests/limits, GPUs). Tasks run as pods on Kubernetes nodes (not on the Airflow worker host), and you can rely on Kubernetes primitives for storage, networking, secrets, and resource management.

How do I mount persistent storage to Kubernetes task pods?

Create a PersistentVolume and a PersistentVolumeClaim in your cluster. In the DAG, pass a V1Volume (pointing to the claim) and a V1VolumeMount (with mount_path such as “/data”) to the KubernetesPodOperator via its volumes and volume_mounts parameters. Tasks then read/write under that path, and data persists beyond the pod’s lifetime.

How do I troubleshoot Kubernetes tasks stuck in Pending/Running forever?

- Check the Airflow task logs for pod status hints. - Use kubectl to list pods (kubectl get pods -n ) and describe the affected pod (kubectl describe pod …). - Common issues include missing PersistentVolumeClaims, image pull problems, or insufficient cluster resources. - Fix the root cause (for example, apply missing storage YAML, grant image pull access, or adjust resource requests) and rerun the task.

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more

eBook

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

eBook

pdf, ePub, online

$47.99 $23.99

you save $24.00 (50%)

include audio $24.99 $12.49

pro $24.99 per month

access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
choose one free eBook per month to keep
exclusive 50% discount on all purchases
renews monthly, pause or cancel renewal anytime

lite $19.99 per month

access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more