4 Creating consumer applications

This chapter shifts focus from producers to the other half of the pipeline: consumer applications. It explains how consumers receive and process messages at scale, how consumer groups unlock parallelism, and how batching and timeouts shape the trade-off between throughput and latency. The text surfaces the core correctness themes—offset management, ordering, duplicates, and lag—and frames operational realities such as rebalances and health checks. It also broadens the integration toolkit by discussing when HTTP access via a REST proxy can be useful and how compression choices affect performance and storage, all with the goal of designing consumers that are reliable, efficient, and easy to operate.

The chapter details the consumer polling loop—subscribe, fetch in batches, deserialize, process, and commit offsets—and shows how subscriptions can span multiple topics or use explicit positioning by offset or time. It explains how partitions are assigned one-per-consumer-instance within a group, the roles of the group coordinator and leader, and how joins, leaves, and failures trigger rebalances (eager vs. cooperative) governed by heartbeats, session timeouts, and processing deadlines. It compares assignment strategies (Range, RoundRobin, Sticky, CooperativeSticky), introduces static membership to avoid unnecessary rebalances on restarts, and outlines the next-generation, server-side assignment protocol in KRaft that consolidates heartbeats and assignments for more consistent, low-disruption rebalances. Additional nuances include reading compacted topics and how batching controls (such as size and wait thresholds) influence both resource usage and latency.

Practical guidance is grounded in the Customer 360 ODS example: agree on serializers/deserializers, define a consumer group and client identifiers, run separate consumers for profiles and transactions, prefer manual commits to avoid loss, and start from the earliest offsets when needed—while ensuring idempotency. The chapter warns about concurrent updates to the same aggregate from different consumers and proposes optimistic locking (or funneling updates through a single streaming service) to prevent overwrites. It emphasizes that scalability is bounded by partition count, advises monitoring and acting on consumer lag, and describes sharded worker pools to increase parallelism while preserving per-key order and safe offset commits. Common pitfalls include poorly tuned batch sizes and timeouts, fragile deserialization, mis-initialized offsets, and the lack of distributed transactions when coordinating with external systems. Finally, it recommends producer-side batch compression for better efficiency and positions REST access (e.g., via a proxy or bridge) as a convenient but higher-overhead option when native clients aren’t feasible or low latency isn’t critical.

Consumer message reception involves periodically requesting new data, receiving messages when available, deserializing and processing the messages, and acknowledging their processing.

Each service consumes data from the topic independently.

Workflow of the consumer application

Assignment of the topic partitions to the consumer instances within a consumer group

Various consumer topologies

Subscribing to multiple topics

Finding the group coordinator

Joining the group

Distributing the assignments

Assignment of the partitions under normal circumstances

Redistribution of the partitions after a failure

The RangeAssignor strategy

The RoundRobinAssignor strategy

Different assignments after rebalancing with the RoundRobinAssignor and StickyAssignor strategies

An example of the next-generation rebalance protocol. ConsumerInst 2 joins the group, and this triggers the rebalance.

Specifying consumer assignment by offsets

Current setup for Customer360Service consumers

Overwriting data by different consumers

Using optimistic locking

Calculating consumer lag

Different configurations of the auto.offset.reset property

Compressing the data on producer side

Communicating to Kafka through REST

• Consumers continuously request data, deserialize incoming messages, process them according to application-specific logic, and then commit the message offsets back to Kafka.
• To facilitate parallel processing, consumers operate within a consumer group, where partitions are equitably distributed among group members.
• Consumers can either subscribe to topics to dynamically receive partition assignments or explicitly assign specific partitions for targeted data processing.
• Each consumer group has a group coordinator, which is a broker, responsible for managing the members of the group and facilitating rebalances.
• Within each consumer group, a group leader is designated to manage partition assignments and coordinate with the group coordinator.
• Timeouts are set to efficiently detect and manage consumer inactivity or failures.
• Messages can be compressed by either the producer or the broker to enhance network throughput and reduce storage space. By default, compression is handled by the producer, which specifies the compression type in its configuration.
• When utilizing the Confluent Rest Proxy, clients can interact with Kafka via HTTP REST. The Rest Proxy serves as a mediator, converting requests and responses between the HTTP REST format and Kafka’s native protocol.

FAQ