8 Producing and persisting messages

This chapter traces the full journey of producing and persisting messages in Kafka, showing how producers serialize records, choose partitions, batch data, and coordinate acknowledgments to balance latency, throughput, and durability. It explains why consistent partitioning across client libraries matters for key ordering, how idempotence and retries prevent duplicates and maintain sequence, and how timeouts and backoff settings shape delivery guarantees. Together, these producer-side controls form the first layer of reliability and performance in real-time pipelines.

On the broker side, the chapter outlines how network and I/O threads place produce requests on queues, append data to the end of partition log segments, and rely on the operating system’s page cache for durability, reducing synchronous flushes. It introduces purgatory as the waiting area for responses until required acknowledgments are met, and highlights the trade-offs of forcing fsync versus relying on replication. The storage model is unpacked through Kafka’s on-disk layout: metadata and checkpoint files, per-partition directories, append-only log segments with offset- and time-based indexes, leader-epoch tracking, segment rolling controls, lightweight snapshots for idempotent producers, and safe, delayed deletion of topics.

Finally, the chapter details replication mechanics: followers fetch from leaders, in-sync replicas are defined by bounded lag, and the High Watermark marks the commit point visible to consumers. It shows how replica lag slows consumption by holding back the High Watermark, why overly tight lag thresholds can thrash ISR membership, and how minimum in-sync replicas interact with acknowledgment settings to gate both production and consumption. Practical guidance includes keeping min.insync.replicas aligned with replication factor (for example, 2 with a factor of 3), distributing brokers for fault tolerance, and using idempotence and built-in retries while tuning batching and linger to meet performance targets.

What happens when a message is produced? After calling the produce() (1) method, the data must first be serialized using a serializer (2). The partitioner (3) then decides which partition to write to and stores the message in a buffer (4) for this partition. Once there are enough messages, the producer sends (5) entire batches to the corresponding brokers and, depending on the ACK setting, waits for an ACK (6). If we receive an ACK in time (7), the success-callback is called. If we do not receive an ACK (8), we try (9) again for a while. If an error occurs, the Kafka library throws an exception (10).

We can define numerous timeouts in the Producer. The most important are linger.ms, which we can use to configure batching, and delivery.timeout.ms, which we can use to configure when the Kafka library stops redelivering messages and throws an error.

When a broker receives a produce request (1), it is put on the request queue (2) and later handled by the IO-threads (3) and written to the page cache (4). The operating system flushes it to disk later. Then the message stays in the purgatory (5) until all in-sync followers acknowledge the message (6). When the wait (7) is over, the response is put on the response queue (8) and send back to the producer by one of the network threads (9).

Topics are divided into partitions. Partitions create segments as messages are produced. Segments consist of the actual log files, index files, and a snapshot file.

The High Watermark (HWM) is the offset that all ISRs have replicated and committed to the leader. The lowest follower is not in-sync and is not taken into account when calculating the HWM.

This shows the follower’s Last Committed Offset (LCO) before and after the fetch request (Current Position). The High Water Mark (HWM) indicates the latest offset committed by all in-sync replicas. Since the follower is behind, its LCO is lower than the HWM. The Leader’s Offset (LEO) is a bit ahead of the HWM, as new messages are arriving.

• Producers in Kafka typically use either the official Kafka Java library or librdkafka.
• Avoid using other libraries due to potential lack of features and optimizations.
• The producer workflow involves serialization, partitioning, and buffer management.
• Handling acknowledgments (ACKs) in the producer includes timeout settings and retry mechanisms.
• Kafka brokers delegate much of their work to clients, focusing on message reception and efficient message persistence.
• Upon receiving a produce request, the broker writes data to the OS's page cache, potentially waiting for followers to replicate before sending an ACK.
• Network threads manage message reception, queuing them for IO threads to write to the file system.
• Kafka relies on the OS to persist messages to disk, with options to influence disk flush timing.
• Broker components can be optimized and configured for specific use cases, with professional support advised for complex environments.
• Kafka's data structures, including metadata, checkpoints, and topics, organize and manage data within brokers.
• Partitions divide topics into segments, each with log and index files for efficient data retrieval.
• Log data and indices optimize message storage and retrieval within log segments.
• Segments, based on size or time, manage partition growth and optimize data storage efficiency.
• Replication involves followers fetching data from leaders, ensuring all brokers stay up to date.
• In-Sync-Replicas (ISR) are followers that have received all messages from the leader within a specified time frame.
• The Log End Offset (LEO) marks the last received message position, determining ISR status.
• The High Watermark (HWM) indicates the offset replicated and committed to by all ISRs, affecting message consumption and availability.
• Delays in replication can slow down or halt message consumption and production, with ISR lag impacting HWM and consumer availability.
• Adjusting parameters like replica.lag.time.max.ms and acks=all can impact replication efficiency and system performance, requiring careful configuration for balance.

FAQ