5 Reliability

This chapter frames Kafka reliability around three intertwined pillars: durability, availability, and consistency. It explains how replication and the leader–follower model underpin durability and availability, while consistency requires additional controls and trade-offs. Readers are guided to think in failure scenarios—broker loss, recovery, and partial outages—and to balance the ability to read/write through disruptions against the need for strict correctness. The chapter previews the main mechanisms Kafka offers to shape that balance: producer acknowledgments, in-sync replicas (ISR) with configurable minimums, idempotent writes, transactions, and operational leader election behavior.

The producer acknowledgment modes define delivery guarantees and performance: acks=0 favors low latency but risks loss (at-most-once), acks=1 improves durability on the leader but can still lose data on fast failover, and acks=all coordinates with ISR to maximize safety. The topic’s min.insync.replicas setting is pivotal with acks=all; if not met, producers are rejected instead of silently weakening guarantees. The chapter then clarifies at-most-once, at-least-once, and exactly-once delivery, showing how idempotent producers (sequence numbers, producer IDs, NACKs for out-of-order records) remove duplicates and preserve per-partition ordering with minimal overhead. Practical guidance highlights keeping idempotence enabled, choosing sensible ISR and replication factors, and understanding how retries, in-flight limits, and delivery timeouts interact.

For end-to-end correctness across topics, Kafka transactions provide atomic writes to multiple partitions and allow coupling output records with consumed offsets for exactly-once processing. The producer initializes and bounds each transaction; a transaction coordinator writes commit/abort markers, and consumers must use isolation.level=read_committed to see only committed results. The chapter notes overhead characteristics (more partitions per transaction imply more coordination) and offers patterns for integrating databases (idempotent upserts or storing offsets with data). Finally, it deepens the leader–follower model: followers fetch from leaders and report progress to form the ISR; on leader failure, controllers elect a new leader from ISR, later restoring preferred leaders after recovery. Operational tools and cautions cover under-replicated diagnostics, preferred leader elections to rebalance, and the risks of unclean leader elections that trade availability for potential data loss.

Different ACK configurations. With acks=0 the producer sends messages and does not wait for confirmation if they have really arrived. With acks=1, the producer gets a response as soon as the leader has received the message. With acks=all (or acks=-1), the producer does not get feedback until all in-sync replicas have replicated the message.

With acks=0 we get at-most-once delivery guarantees.

Problems arise when the producer expects ACKs and these are lost on the way from the Leader to the producer, although the data was written successfully. The producer sends the message again in this case, and the message is in the log twice, possibly even in the wrong order (not shown).

If idempotency is enabled, the producer sends its producer ID with each message and also a sequence ID for each message. This allows the broker to detect if messages are missing, appear in the wrong sequence, or have been sent multiple times, and to respond accordingly.

If a message was received out of order, then the broker sends a NACK to the producer. This way, we can ensure the correct order of messages and prevent, for example, message B from arriving before message A.

Transactions in Kafka allow us to atomically write to two or more partitions. Either all messages are successfully written, or none are.

As soon as a message that belongs to an unfinished transaction is read by the consumer library, our code stops receiving new messages, even those that do not belong to that transaction. The ordering guarantee still holds. Internally, the messages are still buffered. When a transaction-commit is received, all messages are released from the buffer and sent to our code. If a transaction is aborted, our code does not receive messages belonging to the aborted transaction, but messages not belonging to the transaction are still received normally.

The partitions of our topic are distributed among our three brokers. Broker 2 is the leader for Partition 0, broker 3 is the leader for Partition 1 and Broker 1 is the leader for Partition 2. Since we have a replication factor of 3 and also three brokers, each partition is found on each broker.

Followers continuously fetch data from the leader to stay current and to be able to take over the leader's role if the leader fails.

The controller has detected the failure of Broker 3. Now that the leader for Partition 1 has failed, the controller picks another broker (e.g. broker 1) that is in-sync for this Partition to become the leader.

Although Broker 3 is accessible again, it does not immediately become the leader for Partition 1.

After the preferred leader selection (manual or automatic) is performed, Broker 3 again becomes the leader for Partition 1, and Broker 1 becomes the follower for that partition.

• Acknowledgements are pillars of Kafka's reliability.
• Acknowledgements are optional and configurable for each producer.
• Producers can opt for no acknowledgement (acks=0) for speed, sacrificing resilience.
• Producers can choose to receive ACKs after the leader receives the message (acks=1) or after replication to all ISR (acks=all), with the latter being now the default and most reliable.
• Kafka offers three message delivery guarantees: at most once, at least once, and exactly once.
• At most once is achieved with acks=0.
• At least once is achieved with acks=all and sufficient minimum ISRs.
• Exactly once requires enabling idempotence and setting acks=all.
• Transactions in Kafka enable atomic writes across multiple partitions, ensuring all messages in a transaction are written together or not at all, maintaining data consistency.
• Kafka uses idempotent producers for reliable message production and a variation of the 2-Phase-Commit protocol to manage transaction commit markers, ensuring messages are processed exactly once.
• Consumers must set the isolation.level to read_committed to avoid reading uncommitted messages, ensuring only fully completed transactions are processed to maintain data integrity.
• Kafka's transaction coordinator ensures reliability, even in the event of broker or producer crashes, though it introduces some performance overhead.
• For each partition, there is a broker acting as the leader, handling all requests, while followers replicate data by fetching from the leader.
• If a partition leader becomes unavailable, an in-sync replica (ISR) takes over the leader role.
• When the previous leader is back in-sync, it can become the leader again, as Kafka aims to reinstate the original leader, known as the preferred leader.
• In critical failures, Kafka can perform an unclean leader election, by allowing non-in-sync replicas to become leaders, which can lead to data loss.
• Kafka's leader-follower replication ensures high availability and fault tolerance in the event of broker failures.

FAQ