1 Thinking in distributed systems: Models, mindsets, and mechanics

This chapter argues that modern software is inherently distributed and frames the central question as how distributed an application needs to be. It motivates distribution as the only way to meet real-world fitness goals—correctness, scalability, and reliability—in the face of growing load and inevitable failures. A distributed system is presented as a set of concurrent components that communicate by exchanging messages, whose overall behavior and complexity emerge from the parts and their interactions. The author emphasizes moving from “knowing” to “understanding” through dependable mental models so we can reason with confidence about systems that are complex but unavoidable.

The core tool is the mental model: an internal representation of a system that should be both correct (no falsehoods) and complete (no relevant omissions). The chapter shows how multiple models can be equivalent or complementary, each illuminating different aspects, and recommends viewing distributed behavior as a state machine advancing one step at a time by a component or the network. It distinguishes global versus local viewpoints—an all-knowing observer versus components with only local state—and introduces “systems of systems” (holons/holarchies) to fluidly zoom between atomic components and higher-order subsystems. Correctness is framed via safety (nothing bad happens) and liveness (something good eventually happens); scalability and reliability are cast as responsiveness—meeting SLOs—formalized through SLIs, SLOs, error rates, and error budgets.

To make the mechanics tangible, the “Distributed Systems Inc.” analogy maps components to rooms, the network to pneumatic tubes, and the external interface to a mailbox, making it easy to reason about message loss, duplication, reordering, and crash semantics. Several AHA moments follow: interesting properties like scalability and reliability are emergent; different valid models exist for the same system; and the core challenge is to think globally while acting locally—designing global algorithms from local steps and limited knowledge. Finally, the chapter advocates “thinking above the code,” generalizing concepts like race conditions as incorrect subsets of possible interleavings (and connecting to serializability), setting up a disciplined mindset and vocabulary for the deeper, formal treatment that follows.

Mental model and system

Different models describing the same aspects of a system (the set of facts of each model totally overlaps)

The network as the buffer of inflight messages

The components as the buffer for inflight messages

Different models describing different aspects of a system (the set of facts of each model partially overlaps)

A distributed system as a set of concurrent, communicating components (local state of network not shown)

Behavior of a system as a sequence of states

Safety and liveness

Behavior space of a distributed transaction with two participants

A distributed system as a set of concurrent, communicating subsystems

Holons and holarchies

Two different holarchies, representing the same system

Global point of view

C1’s point of view

Distributed Systems Incorporated

Black box versus white box, a global point of view

Local point of view

Splitbrain

Reasoning about race conditions

Reasoning about serializability

• A mental model is the internal representation of the target system and is the basis of comprehension and communication.
• Striving for a deep understanding of distributed systems is better than merely knowing about their concepts.
• A distributed system is a set of concurrent components that communicate by sending and receiving messages over a network.
• The core challenge in designing distributed systems is creating a coherent system that functions as a whole despite each component having only local knowledge.
• Ultimately, we are interested in the guarantees a system provides. We reason about these guarantees in terms of correctness—that is, in terms of safety and liveness guarantees as well as scalability and reliability guarantees.
• Distributed systems can be visualized as a corporation, where rooms represent concurrent components, pneumatic tubes represent the network, and a mailbox represents the external interface.

FAQ