4 Asset-aware scheduling

Asset-aware scheduling introduces an event-driven alternative to purely time-based orchestration by modeling dependencies between DAGs as assets. Instead of aligning brittle schedules or tightly coupling workflows, teams split pipelines into producer DAGs that create or update assets (such as datasets or models identified by URIs) and consumer DAGs that are automatically triggered when those assets change. This approach reduces duplicated ingestion, minimizes wasted runs when data hasn’t changed, and scales cleanly as more consumers subscribe to the same upstream assets. It complements, rather than replaces, time-based schedules, enabling responsive yet orderly pipelines.

In practice, producers declare assets and attach them as task outputs so that successful updates emit asset events; consumers declare those assets in their schedule so Airflow triggers them on updates and passes triggering_asset_events with the event details. Because event-driven runs do not define a logical_date, consumers that need run context can derive it from the source DAG run or, more robustly, read explicit metadata embedded by producers in each event’s extra field (e.g., the processed date). Producers can also suppress noisy updates by raising AirflowSkipException when nothing actually changed. This event payload becomes part of a data contract between teams, so any changes to the included fields should be coordinated to avoid breaking consumers.

For more complex topologies, consumers can listen to multiple assets, with the default behavior triggering a run once every input has emitted at least one new event since the last run. Asset expressions support AND and OR to define richer conditions, and consumers can select the correct event per asset when building inputs. A key caveat is that Airflow does not align execution dates across assets, so consumers must validate and reconcile mismatches themselves. When both responsiveness and cadence are required, hybrid schedules combine asset triggers with time-based timetables, ensuring quick reactions to upstream changes while still enforcing regular runs.

Multiple teams relying on the same data can each do their own ingestion (A), but this is hardly efficient and can lead to inconsistencies. In this case, it’s more efficient to have a single team ingest the data and trigger the workloads of the other teams whenever new data is ingested (B). This raises a new question though: how do we align the schedules of the producer and consumer DAGs?

Splitting the events DAG into separate producer and consumer DAGs that are owned by different teams. The producer DAG creates and updates an asset representing the event dataset, which is consumed by the downstream DAGs owned by the other teams. Airflow ensures that these downstream DAGs are automatically triggered whenever the asset is updated.

The producer and consumer DAGs for our use case example. The producer DAG fetches event data and updates the file:///data/events asset. This triggers the consumer DAG which will calculate statistics for the new events.

Asset view in the Airflow UI. On the left, you can see the relationship between the asset and the producer DAG. On the right, you can see the event(s) produced by runs of the producer DAG.

View of the events dataset asset, now including the relationship with the consumer DAG.

View of the asset event, now including the data from the extra metadata field.

The second run of the task is now skipped as the data file already exists. This ensures that any DAGs downstream listening to the asset are not triggered as there isn’t any new data available to process.

Adding a second producer that (conceptually) fetches additional event data from a different source.

The DAG listing view, showing that the new DAG is waiting for events from two different assets. The first number will update when events occur and reset after each run of the consumer DAG.

The asset view showing the two assets and their relationship with the two producer and single consumer DAGs.

Every ‘star’ icon indicates an asset update, either for the first or second dataset. Only once both datasets have been updated is a DAGRun for our Consumer DAG triggered. The order in which the assets are updated does not matter.

• Airflow's asset feature allows for event-driven (or asset-aware) scheduling, where task execution is triggered by data availability or changes, as opposed to strict time intervals.
• In data pipelines, this is useful for splitting concerns between DAGs where producer DAGs focus on fetching/transforming data to produce one or more output datasets, and consumer DAGs are triggered by updates to these datasets (potentially producing their own datasets in turn).
• Information about upstream DAGs (e.g., a DAG’s scheduling interval) can be accessed via the triggering_asset_events variable provided to a consumer DAG.
• Producer DAGs can also include extra information in an asset event via the `extra` field. This information can be accessed by any downstream DAGs triggered by the event.
• A producer DAG can skip downstream updates by raising an AirflowSkipException. This is useful in situations where a DAG run didn’t modify an asset and should therefore not trigger any dependent consumer DAGs.
• Consumer DAGs can also depend on multiple assets, in which case they’ll by default be executed whenever all input assets have seen at least one event. You can use conditional logic for more complicated relationships between assets. Either way, this functionality does not take any scheduling semantics of upstream DAGs into account.
• DAGs can also combine time- and asset-based scheduling if they need to react in a timely fashion to dataset updates and also be run periodically.

FAQ