4 Message delivery and processing
This chapter builds a precise mental model for message delivery and processing in partially synchronous distributed systems. It distinguishes delivery (movement of messages through the network to a component) from processing (application logic and side effects), and clarifies common misconceptions—especially the difference between exactly-once processing and exactly-once processing semantics. Working within a request–response dialogue and a model of unreliable components and networks (without Byzantine behavior), it frames correctness around processing outcomes rather than raw delivery counts.
Because each component has only a local view, a sender moves from certainty (before sending) to uncertainty (after sending, before response) and back to certainty (upon response). Timeouts help detect failures but can be inaccurate under partial synchrony, and TCP’s reliability does not eliminate message loss at the application layer. Networks may drop, reorder, or duplicate messages, and receivers may crash before or after side effects, making guaranteed exactly-once delivery or processing impossible. Senders must choose between moving on (intending at-most-once) or retrying (intending at-least-once), and simple receiver-side tracking—whether before or after processing—cannot ensure exactly-once behavior under crashes.
Exactly-once processing semantics focus on equivalent outcomes: the state after one processing should be equivalent to the state after multiple processings. Achieving this end-to-end guarantee requires at-least-once delivery and idempotent effects, with equivalence defined by the application. Some operations are inherently idempotent (for example, write-once registers), but most are not (such as charging a credit card). Practical techniques include sender-generated idempotency keys and receiver-side transactional deduplication that records and processes atomically. The chapter’s credit-card case study highlights both mechanisms (including support from some payment processors) and the importance of user experience, noting that what engineers deem “equivalent” may still feel unacceptable to users.
Request-Response dialogue between two components
Step-by-step progression of the request response dialogue
C1’s point of view: there is only C1 and the rest of the system.
Booking an airline ticket
Request-Response dialogue charging a credit card
Summary
- In the absence of failure, exactly-once message delivery and processing is trivial.
- In the presence of failure, exactly-once message delivery and processing is impossible.
- We are able to guarantee exactly once processing semantics if we are able to guarantee at-least once delivery, at-least once delivery processing, and an idempotent processing step.
- Guaranteeing idempotence is one of the most complex aspects of distributed systems and a one-size-fits-all solution does not exist.
- Guaranteeing exactly once processing semantics and idempotence will require all your skills and all your wits.
- Various challenges include message reordering, message duplication, message loss, crashes, or simply delays.
FAQ
What’s the difference between “exactly once processing” and “exactly once processing semantics”?
Exactly once processing is the ideal where a message is processed a single time with no retries or duplicates. Exactly once processing semantics mean the system’s end state is equivalent to having processed the message once, even if the system actually retried or processed duplicates under the hood.How do message delivery and message processing differ?
Message delivery is about moving a message through the network and handing it to the component. Message processing is what the receiver does with that message: executing application logic, causing side effects, and producing a response.Why is guaranteeing exactly once delivery or processing impossible in this model?
In a partially synchronous system, networks can drop, reorder, or duplicate messages, and components can crash and recover. The sender cannot distinguish between loss, delay, or crash from its local view. These realities make a hard guarantee of exactly once delivery or processing impossible.What do at-most-once, at-least-once, and exactly-once mean for delivery vs. processing?
- Delivery: at-most-once = 0 or 1 delivery; at-least-once = 1 or more deliveries; exactly-once = 1 delivery.- Processing: at-most-once = 0 or 1 processing; at-least-once = 1 or more processings; exactly-once processing = 1 processing. In practice, we care most about processing because that’s where side effects happen.
Do timeouts reliably tell me a request failed?
Not always. In synchronous periods, timeouts are accurate (true positives). In asynchronous periods, timeouts can be false positives (the system was just slow). Before a response arrives, the sender cannot know whether work happened, will happen, or will never happen.Does TCP prevent message loss so I can rely on it for delivery guarantees?
No. TCP provides reliable, ordered delivery between kernels using acks and retransmissions, but the kernel can acknowledge data before the application receives or processes it. Crashes or drops between the kernel buffer and the app can still cause effective message loss at the application layer.After a timeout, should the sender move on or retry?
Those are the only two choices. Moving on aims for at-most-once (risking that the message was never processed), while retrying aims for at-least-once (risking duplicates). Exactly-once guarantees can’t be made; you must design for one of these intents.So how can a system achieve exactly once processing semantics?
End-to-end: combine at-least-once delivery/processing (sender retries until a response; receiver doesn’t skip) with idempotent processing so that repeated processing yields an equivalent final state to processing once.What is idempotence and how is it implemented in practice?
Idempotence means executing an operation multiple times has an equivalent effect to executing it once. Common techniques: include an idempotency key (e.g., UUID) on every retry and have the receiver deduplicate; record the key and perform side effects transactionally so either both record-and-effect happen once or neither do.What are common pitfalls when trying to prevent duplicates at the receiver?
- “Record before process” can drop work if a crash happens after recording but before processing (at-most-once).- “Process before record” can double-apply work if a crash happens after processing but before recording (at-least-once).
Use a single atomic transaction to both record the key and apply the side effect to avoid these.
Think Distributed Systems ebook for free
