Overview
3 Kafka clients and message production
This chapter pivots from broker internals to the way applications interact with Kafka, concentrating on producer behavior and configuration. It explains how client applications—often built with higher-level frameworks like Spring-Kafka or Micronaut-Kafka—connect to clusters, choose serializers, and send data efficiently and safely. The narrative balances practical guidance with architecture-minded trade-offs, aiming to help teams justify production settings that deliver reliability, throughput, and observability without obscuring essential Kafka concepts.
Core connectivity and configuration concepts include Kafka’s binary protocol with cross-version, bidirectional compatibility and broad language support. Brokers expose multiple listeners to separate inter-broker traffic from internal and external client access, with advertised addresses guiding clients after the initial bootstrap. Client libraries hide much of the machinery—metadata lookups, partition selection, batching, retries—while exposing behavior through name–value properties that are externalized for portability. Producers and consumers must agree on serialization (separate key/value serializers and matching deserializers), and clusters can enforce fairness via quotas, ideally keyed to authenticated principals rather than ad-hoc client IDs. Clear client identifiers remain important for monitoring and troubleshooting.
The producer workflow covers constructing a record, serializing key and value, choosing a partition, batching in memory, sending to the leader, replication, acknowledgments, and retries. Partitioning by key preserves per-key order; in the absence of keys, strategies like sticky or round-robin optimize distribution but sacrifice co-location and ordering. Throughput and latency are shaped by batch.size, linger.ms, buffer.memory, and timeout settings, while durability hinges on acks (0, 1, or all) in concert with replication and min.insync.replicas. With idempotence enabled, acks=all, controlled in-flight requests, and sensible retry/timeouts, producers can achieve strong ordering and duplicate avoidance. Common pitfalls include poor key choices that skew load or break ordering, under-provisioned buffers, incorrect timestamps, multi-producer interleaving without provenance headers, and the lack of atomic coordination with databases—each requiring deliberate design and documentation beyond the sensible Kafka defaults.
Summary
- The Kafka service can be exposed to clients on various ports and protocols, primarily to accommodate diverse security requirements.
- Client applications utilize a library to manage communication with Kafka, with its behavior customized through configuration parameters.
- Producers and consumers must align on the data format, employing specific components for serialization and deserialization, which are specified through configuration parameters.
- Producers prepare messages by serializing them, selecting appropriate partitions, batching them together, and sending them to Kafka brokers, awaiting acknowledgment.
- In the event of retriable errors, producers resend data seamlessly without manual intervention.
FAQ
How do Kafka clients and brokers handle version compatibility?
Kafka provides bidirectional compatibility between clients and brokers. During the initial handshake, the broker advertises a range of supported API versions and the client selects the highest version supported by both. If no common version exists, an exception is thrown. This negotiation is automatic in official client libraries, allowing independent upgrades of clients and clusters.How do I configure brokers and clients to connect (listeners, addresses, and security)?
- On the broker:
- listeners defines named endpoints (for example, EXTERNAL://host:9092, INTERNAL://host:9093, INTERBROKER://ip:9094).
- advertised.listeners lists the addresses returned to clients based on the port they connected to.
- listener.security.protocol.map maps each listener name to a security protocol (for example, EXTERNAL:SASL_SSL).
- inter.broker.listener.name selects which listener is used for broker-to-broker traffic.
- On the client:
- bootstrap.servers lists one or more broker addresses the client can initially contact (typically all brokers).
- security.protocol selects the matching protocol for the target listener.
Clients connect to one bootstrap address, fetch cluster metadata, and receive broker addresses corresponding to the listener they used. Returned addresses must be resolvable from the client environment.How do producers and consumers agree on message serialization?
Kafka brokers move opaque bytes; serialization is a client concern. Producers configure key.serializer and value.serializer, while consumers configure the corresponding key.deserializer and value.deserializer. The formats must match; otherwise, deserialization fails at runtime. Clients can use built-in serializers (strings, integers, longs, byte arrays) or provide custom implementations.What parameters does send() take, and what happens behind the scenes?
- Typical parameters: topic (required), key (optional), value (optional), plus optional headers, partition, timestamp, and a delivery callback.
- Internal flow after send():
1) Create a record from key, value, headers.
2) Serialize key and value independently.
3) Choose a partition.
4) Append the record to a per-partition batch in producer memory.
5) Dispatch the batch when it is full or linger time expires.
6) The leader writes it and replication to followers occurs.
7) On acknowledgment, delivery metadata (offset, partition) is available.
8) Retriable errors trigger resends until success or timeout; non-retriable errors are surfaced to the application.How does a producer discover leaders and react to leadership changes?
The producer periodically fetches and caches topic/partition metadata, including leader locations. The cache is refreshed based on metadata.max.age.ms (default 5 minutes) or when errors occur. If a send targets a broker that lost leadership, the client refreshes metadata and resends to the new leader, logging the event.How is partitioning decided, and which strategy should I choose?
- Default: key-based hashing routes the same key to the same partition, preserving per-key order.
- If key is null, the default UniformStickyPartitioner sends batches to a single partition and then rotates, providing even load without per-key ordering.
- Alternatives:
- Round robin (partitioner.class) ignores keys and balances evenly record-by-record.
- Explicit partition index per send.
- Custom partitioner logic.
Trade-offs: use stable keys to preserve ordering and support compaction; use round robin for even distribution when ordering is not required. Watch for key skew and consider custom strategies if certain keys dominate traffic.What acknowledgment (acks) options exist, and how do they affect durability and latency?
- acks=0: fastest, no acknowledgment requested; potential data loss and unknown offsets.
- acks=1: leader acknowledges after write, before replication; low latency but data can be lost if the leader fails before followers replicate.
- acks=all (or -1): leader waits for all in-sync replicas (ISR) to acknowledge for strong durability. Effectiveness depends on min.insync.replicas; if set too low (for example, 1), it can behave like acks=1. For critical data, set min.insync.replicas appropriately (commonly replication.factor - 1) to balance durability and availability.How do batching, buffering, retries, and timeouts influence throughput and reliability?
- Buffering and batches:
- buffer.memory controls total memory used for unsent records.
- batch.size (default ~16 KB) caps batch size per partition.
- linger.ms adds wait time to coalesce more records into a batch (0 sends as soon as possible).
- Retries and time limits:
- retries is effectively very large by default; use delivery.timeout.ms to bound the total time spent attempting delivery (includes retries and acks).
- request.timeout.ms sets the timeout for a single request/response round-trip (sends, acks, metadata).
- max.block.ms bounds how long producer calls block when memory/backpressure occurs.
In cases like lost acknowledgments, batches may be resent. To avoid duplicates and preserve order, enable idempotence and tune inflight limits (see below).How do I ensure ordering and avoid duplicates from the producer?
Kafka can provide strong guarantees with the right settings:
- enable.idempotence=true to deduplicate retried sends.
- acks=all to align with idempotence and durability.
- retries set to a large value (default) and delivery.timeout.ms configured appropriately.
- max.inflight.requests.per.connection tuned so that out-of-order retries don’t violate ordering (use a small value if strict order is required).
With these settings, records are delivered once, in order, per partition.How can I limit client resource usage with quotas, and how are clients identified?
Quotas protect shared clusters by throttling clients that exceed configured limits (for example, produce/consume bandwidth, broker CPU time). Throttling is applied by delaying responses on the broker. Quotas are set per broker and can be:
- Default (applies to all clients).
- Scoped to an authenticated user principal (preferred, reliable identity).
- Scoped to client.id (less reliable because it’s arbitrary and can be auto-generated).
Identification via principals is recommended for consistent governance and operations.What are common producer pitfalls, and how do I avoid them?
- Wrong partitioning: choose keys that preserve necessary ordering and avoid skew; use round robin when order is irrelevant.
- Insufficient memory: size buffer.memory and batch.size for your throughput; respect max batch size limits; monitor producer and broker metrics.
- Incorrect timestamps: keep clocks synchronized and agree on timestamp source (producer vs broker). With LogAppendTime, a batch shares a single append time.
- Multiple producers on the same partition: messages can interleave; include a producer identifier in headers for traceability.
- Cross-system transactions: Kafka doesn’t provide distributed transactions with external stores; plan for atomicity/consistency patterns (covered later), and avoid emitting events that don’t reflect committed state.
Kafka for Architects ebook for free
