Overview
Appendix A. Running code samples
FAQ
Where can I find the code for the book, and what does the repository include?
The code is available on GitHub at https://github.com/godatadriven/data-pipelines-with-airflow-2nd-ed. It contains the same code shown in the book plus ready-to-run Docker environments so you can execute the examples yourself.How is the repository structured across chapters?
The repository is organized per chapter, with directories numbered 01–17. Each chapter folder contains a self-contained set of code and configuration needed to run that chapter’s examples.What key files and folders are included in each chapter directory?
- dags: DAG files shown in the chapter - compose.yaml: The standard Airflow setup for running the DAGs - compose.override.yaml: Chapter-specific overrides and extra services - .env: Chapter-specific environment variables - README.md: Instructions and any chapter-specific run detailsDo code listings in the book map directly to files in the repo?
Where possible, code listings reference their corresponding files in the chapter directory. In some cases, multiple listings are combined into a single DAG file, especially for more complex examples.How do I start the Docker environment for a chapter?
From within the chapter directory, run:$ docker compose up --buildThis builds images (if needed) and starts the services required for the examples.
How can I run the environment in the background to avoid terminal noise?
Use detached mode:$ docker compose up --build -dThis starts the containers in the background.
Which services are typically started by the chapter environments?
By default, the environment includes: Airflow apiserver, scheduler, dag processor, triggerer, worker, a Postgres database for the Airflow metastore, and a Redis queue for the executor. Some chapters add services like Grafana, Prometheus, Flower, or Redis (configured via compose.override.yaml).How do I inspect running services and view logs?
- List running containers:$ docker psDocker Compose prefixes container names with the folder name (e.g., chapter02_...), making them easy to identify. - Follow logs for a specific container:
$ docker logs -f <container_name>
What system resources should I allocate to Docker for the examples?
For all examples to run smoothly, allocate about 4 CPUs, 16 GB memory, and 80 GB disk space. Earlier, simpler chapters can run with less.How do I stop and clean up the environment safely?
- Stop foreground compose with CTRL+C (if not in detached mode). - Tear everything down (containers, networks, and volumes) from the chapter directory:$ docker compose down -v- Check for any remaining stopped containers:
$ docker ps -a- Remove specific containers:
$ docker rm <container_id>- Remove all containers:
$ docker rm $(docker ps -aq)- Remove unused volumes (use with caution to avoid data loss):
$ docker volume prune
Data Pipelines with Apache Airflow, Second Edition ebook for free
