7 Kafka interaction patterns

This chapter shows how to apply Kafka deliberately across architectural patterns rather than by default. It explains why Kafka naturally supports smart endpoints and dumb pipes, contrasting it with ESB-style middleware that centralizes logic and becomes a bottleneck. Cross-cutting concerns are better handled by a service mesh, though Kafka’s rich client libraries overlap with mesh sidecars and remain an evolving area. The text cautions against using Kafka for synchronous request-response due to correlation, availability, and recovery complexities, and instead recommends pub-sub patterns such as CQRS and event sourcing, pairing the latter with snapshotting to speed recovery. It also covers tiered storage to keep hot data local and offload cold data cost‑effectively.

The chapter extends the discussion from operational to analytical data through the data mesh paradigm. It frames Kafka topics and schemas as a good fit for domain ownership and “data as a product,” with Schema Registry enabling durable data contracts and controlled evolution. At the same time, it stresses the need for federated governance (privacy, standardization, access control) and a self‑serve platform for discovery, change workflows, and automation. While Kafka exposes well-governed event streams, the producing systems remain the systems of record, and cross‑domain use cases require coordination to align business semantics with technical schemas.

For moving data in and out, Kafka Connect provides pluggable source and sink connectors managed by workers and tasks, with converters for serialization and stateless single‑message transformations. The chapter surveys database integrations via JDBC connectors and change data capture tools, then highlights real‑world pitfalls: defining source offsets, limited parallelism, uneven task load, schema drift and “poison pills,” DLT handling, and capacity planning. Finally, it details delivery guarantees: idempotent producers and proper acknowledgments to avoid duplicates; transactions to achieve exactly‑once processing in read‑process‑write pipelines with isolation for consumers; and the transactional outbox to bridge databases and Kafka without two‑phase commit—accepting trade‑offs like potential duplicates, which must be addressed through idempotent consumers, monitoring, and reconciliation.

All the logic resides on the smart endpoints; the pipes serve only for transport.

Integration logic is handled by smart middleware (the enterprise service bus).

An example of a service mesh architecture. Business logic and integration logic are separated into different containers. All network traffic flows through the sidecar component, which acts as a proxy to the component’s pure business logic.

Configuring sidecar containers through the control plane

Both services participating in the request-response pattern have a producer and consumer container. The request and response are matched using the correlation ID, specified in the message header.

The availability of RequestResponseService is problematic, as another instance does not have the original request context.

Each message carries a correlation ID, a topic, and a partition so that ResponderService knows where to send the response.

Each instance of the service has its own consumer group assigned.

Each instance of the service has a dedicated topic for the responses.

Different object models are responsible for writing and reading the data.

Implementing the CQRS pattern with Kafka. Here OrderService is a command, which updates orders, and the other services are query services. Query services may have pretransformed denormalized data.

The state can be periodically uploaded to the database and then restored from there.

When the tiered storage feature is enabled, the cold data is moved periodically to cheaper, cloud-based storage.

Connector plugins must be installed in Kafka Connect to enable specific technologies. Connectors are configured via configuration files that contain instructions on how to pull or push data.

How Kafka Connect fits into the Kafka ecosystem

Connector tasks are distributed among workers, with each worker receiving its assigned portion.

ConnectRecord is the parent class for the SourceRecord and SinkRecord classes.

How converters are used in the source connector flow

How converters are used in the sink connector flow

Transformations are stateless and can be chained together.

Database tables from the source database are assigned to the tasks in the source connector.

Update_time is used as an offset for BalanceConnector. Each time, the connector loads data that was updated after the stored offset.

Sink connectors consume from Kafka with partitions balanced across tasks. Because tasks share a consumer group, each partition is assigned to only one task. Topic 2 has two partitions, so only two tasks consume it—task 3 gets no topic 2 assignment.

OrderService reads, processes, and writes each event atomically.

A transaction involves writing atomically to all four topics.

Creating a transaction using ClientAPI. Both send operations are part of a single transaction, ensuring that they either both succeed or both fail atomically.

Isolation levels determine whether consumers can read uncommitted messages. This figure shows two consumers with different isolation settings.

When the transactional outbox pattern is used, a separate process (step 3 in the figure) publishes events to Kafka.

• Kafka is best suited for publish-subscribe, event-driven architectures, rather than traditional request-response models.
• Kafka follows a “smart endpoints, dumb pipes” model, where message processing logic is handled at endpoints, leaving Kafka as a pure transport layer.
• The service mesh does not fully align with Kafka’s client libraries; however, experimental solutions, such as Kafka mesh filters, are emerging to bridge this gap.
• Kafka supports CQRS well, separating read and write operations across services and enabling event-driven updates to query models.
• Kafka supports event sourcing, but snapshotting is often necessary to optimize data restoration performance for consumers.
• Kafka can support data mesh architectures by enabling decentralized data ownership, data-as-a-product principles, federated governance, and self-serve data platforms.
• Kafka Connect simplifies data integration with source and sink connectors, though managing schema evolution and handling errors (e.g., poison pill messages) can be challenging.
• Kafka’s exactly-once semantics ensures that data is processed exactly once per transaction in a read, process, write pipeline, maintaining atomicity across multipartition writes.
• Kafka does not currently support two-phase commit (2PC) but may offer this in the future (e.g., KIP-939); the transactional outbox pattern is often used as a workaround.

FAQ