9 Consuming messages

This chapter explains Kafka’s end-to-end message consumption workflow. It introduces the fetch-based model in which consumers discover partition leaders via metadata and issue fetch requests that specify a partition and starting offset. It covers practical defaults and knobs—consumers typically read from the end unless auto.offset.reset is set to earliest—and notes the risk of always restarting from the beginning. It also describes consuming from the closest replica using rack awareness (client.rack and broker.rack), and how brokers serve fetches efficiently via network and I/O threads, the page cache, and latency/throughput controls such as fetch.min.bytes and fetch.max.wait.ms.

The chapter then dives into offsets—the per-partition progress marker that consumers advance—and options for managing them. Depending on needs, offsets may be ignored (ephemeral caches), kept locally, stored transactionally alongside sink writes for exactly-once behavior, or, most commonly, committed to Kafka’s compacted __consumer_offsets topic. Offsets are associated with consumer groups, enabling horizontal scaling and centralized progress tracking. Operationally, lag is the key health metric, while choosing and consistently using the correct group.id is critical to prevent offset theft and unintended message reprocessing.

Finally, it details consumer groups and the Kafka Rebalance Protocol. A broker acts as group coordinator; consumers FindCoordinator, JoinGroup, the first joiner becomes leader, computes assignments, and SyncGroup distributes them; heartbeats and session timeouts detect failures. Partition assignment strategies include Range (aligns same-index partitions across topics, useful for joins but can be uneven), RoundRobin (balances load), Sticky (minimizes churn), and Cooperative Sticky (incremental, two-phase rebalances that avoid stop-the-world). Because rebalances are costly and triggered by membership or partition changes, the chapter recommends Cooperative Sticky and, in automated environments, Static Memberships with stable group.instance.id and longer session.timeout to prevent unnecessary rebalances during restarts. It also notes forthcoming simplifications with the next-generation rebalance protocol and the need for all group members to use the same assignment strategy.

The consumers are consuming from the nearest replica (in the same rack) but they still have to produce to the leader of the partition.

The broker undergoes a similar process when receiving a fetch request as that of when receiving a produce request. However, instead of writing data to the disk, we need to read from it. Additionally, we do not need to wait for other brokers, but rather for more messages depending on the fetch.max.wait.ms and fetch.min.bytes settings.

A consumer group consists of any number of consumers that divide the desired partitions among themselves. Different consumer groups are isolated and unaware of each other. The Kafka brokers assist consumers in coordinating the task distribution.

The Kafka Rebalance Protocol allows a group to distribute certain resources among its members. For example, consumer partitions. In this process, a broker acts as the group coordinator, and one of the consumers serves as the group leader.

By default, partitions in a consumer group are assigned in such a way that the same partition across different topics is always consumed by the same consumer. While this may not be the most resource-efficient allocation, it allows us to easily perform joins across multiple topics.

The RoundRobin (and the more advanced Sticky, and Cooperative Sticky) assignor distributes the load evenly across the consumers of our consumer group.

The Cooperative Sticky Assignor avoids stopping the world rebalances by replacing a single rebalance by two consecutive ones. During the first rebalance, consumers are asked to let go of the partitions that should be moved to another consumer and in the second rebalance the now dangling partitions are assigned to the consumers.

• Kafka uses a fetch-based approach for consumers to retrieve messages.
• Kafka brokers handle most of the coordination work, minimizing overhead.
• Kafka brokers support rack IDs for fault tolerance and load distribution.
• Consumer Groups help manage offsets and distribute workload among consumers.
• Offsets are stored in Kafka as part of the consumer group's metadata in a special internal topic called __consumer_offsets.
• Offsets are crucial for consumers to keep track of which messages they have already consumed.
• The Kafka Rebalance Protocol coordinates task distribution among consumer group members.
• Rebalances are triggered by changes in group membership, topic partitions, or consumer failures.
• Static Memberships and Cooperative Sticky Assignor optimize rebalance behavior.
• Range Assignor and Round Robin Assignor are strategies for partition assignment.
• Range Assignor ensures that the same consumer handles the same partitions across topics.
• Static Memberships reduce rebalance frequency by extending session timeouts and using unique group instance IDs.
• Cooperative Sticky Assignor improves rebalance efficiency by iteratively approaching the desired state.
• Consumer groups facilitate parallel processing by distributing partitions among consumers.
• Round Robin Assignor evenly distributes partitions among consumers when joins across topics are not required.

FAQ