9 Adding observability with containerized monitoring

Modern containerized applications can scale and self-heal, but they still require human oversight through robust monitoring and alerting. This chapter explains how observability provides the needed insight into application behavior and performance, emphasizing a standard, container-friendly approach using Prometheus for metrics collection and Grafana for visualization. Because both tools run in containers and work across platforms, teams can achieve consistent, environment-agnostic visibility from development through production, even tapping into Docker Engine metrics to understand infrastructure health.

The chapter outlines a practical path to instrumenting applications: expose a metrics endpoint per container and have Prometheus scrape them on a schedule. It distinguishes three complementary layers of telemetry—infra-level (for Docker and the host), runtime-level (language/runtime specifics such as JVM memory or Go goroutines), and application-level (custom counters and gauges that reflect operations or business events). Prometheus’s pull model, service discovery, and metric labels make it well suited to dynamic, scaled workloads; simple PromQL queries (filtering, summing, grouping) let you analyze per-container data or aggregate it system-wide without complex configuration.

With metrics in Prometheus, Grafana turns queries into clear, actionable dashboards aligned to the “golden signals” of latency, traffic, errors, and saturation. The chapter demonstrates assembling panels that track usage, resource consumption, and error trends, focusing on patterns and anomalies over raw values. It also shows how to package Prometheus and Grafana with predefined configuration and dashboards into Docker images, making observability portable and repeatable. The result is a layered, production-ready monitoring model centered on Prometheus, with concise dashboards that surface the right signals to detect issues early and guide effective responses.

Running Prometheus in a container to monitor other containers and Docker itself

Configuring the Docker Engine to export metrics in Prometheus format

Sample metrics captured by Docker and exposed through the HTTP API

Prometheus has a simple web UI that you can use to find metrics and run queries.

Prometheus client libraries in your apps make the metrics endpoints available in the container.

Prometheus metrics about the Go runtime from the image gallery web container

A metrics endpoint that includes custom data as well as Node.js runtime data

Prometheus running in a container, configured to scrape metrics from app containers

Processing metrics can be used to verify that requests are being load-balanced.

Aggregating a metric to sum values from all containers and showing a graph of results

A simple Prometheus query. You don’t need to learn much more PromQL than this.

The Grafana dashboard for the application. Looks fancy, but it’s actually pretty simple to build.

Navigating dashboards in Grafana—recently used folders are shown here

A closer look at the application dashboard and how visualizations relate to the golden signals

Adding a new panel to the Grafana dashboard to show HTTP errors

The architecture of monitoring in a containerized application—Prometheus is at the center.

• Write a Docker Compose file that you can use to run the app, which also starts a Prometheus container and a Grafana container.
• The Prometheus container should already be configured to scrape metrics from the to-do list app.
• The Grafana container should be configured with a dashboard to show three key metrics from the app: number of tasks created, total number of HTTP requests processed, and number of HTTP requests currently being processed.

FAQ