7 Reliability
Reliability is presented as the discipline that closes the gap between software being deployed and software actually working for users. The chapter frames reliability as a user-centered promise: a system should “just work,” even when components fail, dependencies time out, or load exceeds expectations. It explains that reliability is not a single property but the combined result of resilience, fault tolerance, robustness, durability, performance, scalability, and operational discipline.
The chapter then explores design choices that help systems fail in controlled ways instead of collapsing completely. Graceful degradation allows a service to reduce quality rather than become unavailable, using techniques such as load shedding or adaptive LIFO to preserve some successful responses under heavy load. It distinguishes resilience, fault tolerance, robustness, and reliability, then explains the trade-off between failing open, which favors availability, and failing closed, which favors control, safety, or security. It also emphasizes classifying dependencies as soft or hard, because this determines reliability expectations, fallback strategies, alerting, rollout coordination, and whether a supposedly optional dependency can silently become critical over time.
The second half focuses on diagnosing failures correctly under pressure. It warns against confusing sequence with causation through the post hoc ergo propter hoc fallacy, and against mistaking correlation for causation when lurking variables distort the data. The kernel-panic example shows how a visible rollout can look guilty while the real cause is a hidden shift in platform distribution. To avoid these traps, the chapter recommends disciplined hypothesis testing, data segmentation, proactive observability, and awareness of confirmation bias. Overall, strong reliability practice requires both building systems that degrade safely and debugging incidents with clear, evidence-based reasoning.
(A) I have to use console.log statements to debug my code (B) I switch to TDD (C) I write tests (D) I am no longer obliged to use console.log statements to debug my code.
A person pushing a subway car while it moves forward. The two events are correlated (happen together), but the pushing didn’t cause the movement.
Both the kernel version and the platform distribution impact kernel panic count.
Summary
- Reliability is what users actually care about. Performance, scalability, resilience, fault tolerance, and robustness are means to the same end: the system works consistently from the user's perspective. It just works is a workable definition.
- Graceful degradation beats catastrophic failure. When a system can't handle full load, deliberately reducing service quality — through load shedding, adaptive LIFO, or other creative approaches — keeps a portion of users served. Some response is almost always better than no response.
- Resilience, fault tolerance, robustness, and reliability are distinct properties. Resilience is about recovery; fault tolerance is about continuity through redundancy; robustness is about withstanding stress; reliability is the cumulative user-facing result. Confusing them produces muddled designs.
- Fail-open versus fail-closed is a deliberate trade-off. Fail-open prioritizes availability; fail-closed prioritizes control and security. Which is appropriate depends entirely on the cost of an unsafe operation versus the cost of an unavailable system.
- Hard dependencies must match or exceed the reliability of the services that depend on them. Soft dependencies don't, but they need to stay soft — without active maintenance and continuous fallback testing, soft dependencies silently turn hard.
- Post hoc ergo propter hoc confuses sequence with causation. A happened, then B happened doesn't mean A caused B. The fallacy is especially dangerous when combined with the assumption that avoiding A will prevent B in future.
- Lurking variables distort root-cause analysis. When two studied variables correlate, a hidden third variable may be driving both. Data segmentation — breaking a metric down by platform, region, cohort, or other potential factors — is the most reliable way to surface them.
- Confirmation bias is the silent failure mode of incident response. When we already have a hypothesis, we unconsciously interpret new evidence as supporting it. Discipline means actively seeking disconfirming evidence and being willing to abandon a hypothesis that doesn't fully explain what we're seeing.
FAQ
What does reliability mean from the user’s perspective?
Reliability means the system “just works.” For users, a feature is not truly done just because it was merged and deployed; it is done when it is available, working, and dependable. Reliability is the discipline that bridges the gap between deployed software and software that users can actually trust.
Why is reliability important even outside safety-critical domains?
In safety-critical domains like air traffic management, unreliable systems can put lives at risk. But reliability is also critical in everyday systems. For example, if a cloud photo service loses years of family photos, the damage may not be physical, but users lose trust immediately. A system can have many impressive features, but if it is unreliable, people will stop depending on it.
What is graceful degradation?
Graceful degradation is the ability of a system to deliberately reduce its quality of service instead of failing completely. For example, during heavy load, a service might reject or throttle some requests through load shedding so that the remaining requests can still be served. The core idea is that some service is better than no service.
How does load shedding help reliability?
Load shedding protects a system by rejecting or throttling some incoming traffic before critical resources are overwhelmed. In the chapter’s example, autoscaling created more service instances, but those instances required more database connections, eventually overloading the database. If the service had shed load earlier, it could have preserved capacity and continued serving a portion of users instead of timing out all requests.
What is adaptive LIFO and why can it improve successful request completion?
Adaptive LIFO is a queueing strategy where requests are processed normally in FIFO order under healthy conditions, but under heavy load the system switches to LIFO order. This means newer requests are handled first because users who just made a request are more likely to still be waiting. Although this is unfair to older requests, it can increase the number of requests that complete successfully instead of wasting resources on requests whose users may have already given up or retried.
What is the difference between resilient, fault-tolerant, robust, and reliable systems?
A resilient system recovers after disruption, like a rubber band returning to shape. A fault-tolerant system continues operating when components fail, often through redundancy. A robust system functions correctly under stressful conditions such as high load or unexpected inputs. A reliable system is the broader user-centric result: it performs as users expect for a specific period of time. Reliability is often built from resilience, fault tolerance, and robustness.
What is the difference between fail open and fail closed?
Fail open means that when failures occur, the system continues to allow operation, prioritizing availability over control. Fail closed means that when failures occur, the system stops or rejects operations, prioritizing control, safety, security, or data integrity over availability. The right choice depends on the risks involved. An e-commerce site may prefer failing open to avoid disruption, while a financial, healthcare, or safety-related system may need to fail closed to avoid harmful consequences.
What is the difference between a soft dependency and a hard dependency?
A soft dependency is non-critical: if it fails, the service can still operate acceptably, perhaps with degraded functionality. For example, a recommendation service may be soft for a video platform because users can still watch videos without recommendations. A hard dependency is critical: if it fails, the service cannot operate properly. For example, authentication is a hard dependency for a system that requires users to log in.
Can a soft dependency become a hard dependency over time?
Yes. A dependency can start as soft but silently become hard as system conditions change. For example, a cache may initially be optional because the service can fall back to the database. But as traffic grows, the database may no longer handle all requests without the cache. At that point, cache failure can bring down the service, making the cache a hard dependency in practice.
What are common reasoning traps during incident analysis?
The chapter highlights three major traps. Post hoc ergo propter hoc means confusing sequence with causality: assuming that because one event happened before another, it caused it. Lurking variables are hidden factors that distort analysis, such as platform distribution affecting kernel panic rates. Confirmation bias happens when engineers form an early hypothesis and interpret evidence in a way that supports it. Avoiding these traps requires clear evidence, alternative hypotheses, data segmentation, and strong observability.
The Coder Cafe ebook for free
