16 Handling Stateful Applications with StatefulSets
This chapter explains how Kubernetes handles stateful workloads and why StatefulSets are the preferred abstraction compared to Deployments for applications that require durable state and stable identities. It contrasts stateless scaling with the complexities of stateful scaling, highlighting storage constraints (for example, common lack of ReadWriteMany in many environments) and the need for per-replica volumes and stable network identities. Using the Kiada suite’s Quiz service as a running example backed by MongoDB, it shows how a StatefulSet paired with a headless Service provides ordered, stable Pod identities, per-replica PersistentVolumeClaims from templates, and per-Pod DNS (including SRV records), while moving data import to a one-shot Pod to avoid duplicate ingestion.
The chapter dives into core StatefulSet behavior: ordinal Pod naming and identity persistence across restarts, controlled Pod creation (Parallel vs OrderedReady), and at-most-one semantics that prevent duplicate identities during node outages. It demonstrates lifecycle scenarios such as replacing missing Pods, handling node failures (including when volumes are local vs network-attached), and when manual intervention (force-deleting Pods or claims) is required. It covers scaling rules (removing highest ordinals first), default retention of PVCs with an option to change retention policies, and the common pattern of exposing Pods through two Services—regular (ready-only) for clients and headless (includes unready) for peer discovery—to satisfy both application readiness and cluster membership needs.
For change management, the chapter details StatefulSet updates: RollingUpdate (one Pod at a time, highest ordinal first, optional minReadySeconds) and OnDelete (operator-driven, in any order). It shows how partitioned rolling updates enable staged and canary rollouts by freezing updates below a chosen ordinal, and discusses pitfalls where OrderedReady can deadlock if readiness depends on quorum, along with remedies (Parallel policy or adjusted probes). It concludes with Kubernetes Operators as higher-level, application-aware controllers: installing and using the MongoDB Community Operator to manage a replica set via a custom resource, while the operator creates the StatefulSet and Services, initializes replication, reconciles scaling, and automates the application’s operational tasks beyond what a raw StatefulSet provides.
All Pods from a Deployment use the same PersistentVolumeClaim and PersistentVolume.
Providing each replica with its own volume and address.
Treating entities as pets vs. as cattle
A StatefulSet, its Pods, and PersistentVolumeClaims
A headless Service used in combination with a StatefulSet
StatefulSets don’t delete PersistentVolumeClaims when scaling down; then they reattach them when scaling back up.
Comparison between the OrderedReady and Parallel Pod management policy
How the Pods are updated over time with different update strategies
Partitioning a rolling update
Managing an application through custom resources and operators
Summary
- Stateful workloads are harder to manage than their stateless counterparts because managing state is difficult. However, with StatefulSets, managing stateful workloads becomes much easier because the StatefulSet controller automates most of the work.
- With StatefulSets you can manage a group of Pods as pets, whereas Deployments treat the Pods like cattle. The Pods in a StatefulSet use ordinal numbers instead of having random names.
- A StatefulSet ensures that each replica gets its own stable identity and its own PersistentVolumeClaim(s). These claims are always associated with the same Pods.
- In combination with a StatefulSet, a headless Service ensures that each Pod receives a DNS record that always resolves to the Pod’s IP address, even if the Pod is moved to another node and receives a new IP address.
- StatefulSet Pods are created in the order of ascending ordinal numbers, and deleted in reverse order.
- The Pod management policy configured in the StatefulSet determines whether Pods are created and deleted sequentially or simultaneously.
- The PersistentVolumeClaim retention policy determines whether claims are deleted or retained when you scale down or delete a StatefulSet.
- When you update the Pod template in a StatefulSet, the controller updates the underlying Pods. This happens on a rolling basis, from highest to lowest ordinal number. Alternatively, you can use a semi-automatic update strategy, where you delete a Pod and the controller then replaces it.
- Since StatefulSets don’t provide everything needed to fully manage a stateful workload, these types of workloads are typically managed via custom API object types and Kubernetes Operators. You create an instance of the custom object, and the Operator then creates the StatefulSet and supporting objects.
Kubernetes in Action, Second Edition ebook for free
