Appendix B. Prometheus metric mapping

This appendix explains how Airflow’s StatsD metrics are translated into Prometheus-friendly metrics via the Prometheus StatsD exporter. The goal is to standardize metric names, attach meaningful labels, and ensure efficient, Prometheus-aligned exposition. Because Airflow evolves, the mapping is explicit yet subject to change, providing a clear bridge between the hierarchical StatsD naming style and Prometheus’s labeled, snake_case conventions.

The mapping covers several categories of DAG-level observability. It includes total DAG parsing time, the count of discovered DAGs, and the number of import errors during DAG processing. It also maps loading durations per file (with concatenated DAG IDs when multiple DAGs reside in one file) and captures recent processing characteristics, such as last duration, last runtime, and seconds since the last run, labeled by filename. At the DAG run level, it exposes dependency checks, successful run durations, and schedule delay per dag_id, enabling analysis of scheduling latency and runtime behavior across pipelines.

Task and execution-layer visibility is addressed through task duration metrics labeled by dag_id and task_id, along with executor capacity indicators: open slots, queued tasks, and running tasks. Pattern-based matching (with wildcards and capture groups) is used to extract label values from StatsD metric paths, producing concise Prometheus metric names and consistent labels. Together, these mappings offer a coherent, label-rich metric surface for dashboards and alerting across DAG parsing, scheduling, task execution, and executor resource utilization.

FAQ