8 Communicating with external systems

This chapter explains how Airflow’s real power comes from orchestrating work across external systems through a rich ecosystem of provider packages and operators. By installing the right providers, you gain concise, high-level primitives that wrap cloud SDKs and APIs, letting you trigger storage, compute, and database actions without writing low-level integration code. The chapter emphasizes choosing system-specific operators when available, understanding that they evolve over time, and treating pipelines as reliable, schedulable workflows that can be tested in isolation even when they depend on external services.

The first use case builds an end‑to‑end machine learning flow on AWS: copying sample data to S3, transforming it for training, starting a SageMaker training job, and deploying an endpoint. It demonstrates local task testing with airflow tasks test using developer credentials, and highlights security best practices for handling secrets. Key lessons include templating parameters for idempotency (such as unique training job names), waiting for completion to ensure correct downstream ordering, and keeping Airflow focused on orchestration while the cloud service does the heavy lifting. The chapter also notes emerging Airflow features that simplify object store interactions, further reducing boilerplate around common data movement patterns.

The second use case focuses on moving data between systems in classic ETL scenarios. It surveys A‑to‑B operators and shows how to implement a custom Postgres‑to‑S3 transfer using hooks, discussing practical trade‑offs such as in‑memory buffers versus temporary files, output formats (e.g., writing CSV for broad compatibility), and idempotent overwrites. For heavier transforms, the chapter advocates offloading execution to external engines—running containers with the DockerOperator or invoking Spark via multiple access patterns—while Airflow coordinates, monitors logs, and enforces ordering. Throughout, the guidance centers on clear configuration, safe credential management, and designing tasks that can be rerun predictably without leaving residual state.

An Airflow operator translates given arguments to operations on the Cloud SDK.

Example handwritten digits in the MNIST data set

Rough outline of how a machine learning model is trained in one stage and classifies previously unseen samples in another

Logical steps to create a handwritten digit classifier

Logical steps implemented in Airflow DAG

After running the task locally with airflow tasks test, the data is copied to our own AWS S3 bucket.

Gzipped and pickled data was read and transformed into a usable format.

The SageMaker operators only succeed once the job is completed successfully in AWS.

In the SageMaker model menu, we can see the model was deployed and the endpoint is operational.

The handwritten digit classifier exists of more components than just the Airflow pipeline.

Example API input and output. A real product could display a nice UI for uploading images and displaying the predicted number.

Airflow managing jobs moving data between various systems

Table structure of example Inside Airbnb database

Exporting data from a Postgres database to stringified tuples

• Operators for external systems expose functionality by calling the client for a given system.
• Sometimes these operators are merely passing through arguments to the Python client.
• Other times they provide additional functionality, such as the SageMaker-Training-Operator, which continuously polls AWS and blocks until completion.
• If access to external services from the local machine is possible, we can test tasks using the CLI command airflow tasks test.
• Think wisely what task to create in your pipeline. One-off infrastructure setup tasks are better deployed as part of your CI/CD pipeline for example.
• The Airflow ecosystem has a lot of operators to transfer data between two systems (A-to-B operators). If not available you can write one yourself.
• Separate orchestration from execution, especially for large jobs.

FAQ