1 Getting to know Kafka as an architect

This chapter introduces Kafka as a cornerstone of modern, event-driven architecture, tracing its evolution from a high-throughput message broker to a comprehensive streaming platform for real-time pipelines and analytics. It sets expectations for architects, engineers, and leaders to think beyond getting Kafka to run—focusing instead on why Kafka behaves as it does and how to design systems that harness its strengths. The narrative emphasizes architectural tradeoffs and big-picture choices—event modeling, schema evolution, integration strategies, and the balance among performance, ordering, and fault tolerance—rather than code or client APIs.

From an architectural lens, the chapter contrasts synchronous request-response styles with event-driven design, highlighting Kafka’s ability to decouple producers and consumers, deliver low-latency fan-out, and improve resilience through asynchronous communication. It surfaces the realities of eventual consistency, idempotency, and ordering, and explains core Kafka principles: publish-subscribe, durable storage with replication, acknowledgments and retries, reprocessing via retention and replay, and the immutable commit log. The ecosystem view covers producers, brokers, and consumers; persistent, replicated logs; and KRaft controllers that coordinate cluster metadata and availability. Together, these capabilities make Kafka well-suited for high-volume, low-latency workloads like fraud detection, recommendations, and predictive maintenance.

The chapter then pivots to data flow design and operations. It shows how Schema Registry formalizes data contracts with versioning and compatibility, Kafka Connect moves data in and out of external systems without custom code, and streaming frameworks such as Kafka Streams or Flink power real-time transformations, joins, and routing with strong processing guarantees. Operational guidance spans deployment choices (on-premises, managed cloud, or hybrid), tuning, monitoring, security, and capacity planning. Finally, it outlines when to use Kafka for reliable event delivery versus long-term retention and event sourcing, clarifies how Kafka differs from databases, and encourages architects to weigh benefits, risks, and costs while building scalable, sustainable, and data-centric systems.

Request-response design pattern

The EDA style of communication: systems communicate by publishing events that describe changes, allowing others to react asynchronously.

The key components in the Kafka ecosystem are producers, brokers, and consumers.

Structure of a Kafka cluster: brokers handle client traffic; KRaft controllers manage metadata and coordination

Publish-subscribe example: CustomerService publishes a “customer updated” event to a channel; all subscribers receive it independently.

Acknowledgments: Once the cluster accepts a message, it sends an acknowledgement to the service. If no acknowledgment arrives within the timeout, the service treats the send as failed and retries.

Working with Schema Registry: Schemas are managed by a separate Schema Registry cluster; messages carry only a schema ID, which clients use to fetch (and cache) the writer schema.

The Kafka Connect architecture: connectors integrate Kafka with external systems, moving data in and out.

An example of a streaming application. RoutingService implements content-based routing, consuming messages from Addresses and, based on their contents (e.g., address type), publishing them to ShippingAddresses or BillingAddresses.

• There are two primary communication patterns between services: request-response and event-driven architecture.
• In the event-driven approach, services communicate by triggering events.
• The key components of the Kafka ecosystem include brokers, producers, consumers, Schema Registry, Kafka Connect, and streaming applications.
• Cluster metadata management is handled by KRaft controllers.
• Kafka is versatile and well-suited for various industries and use cases, including real-time data processing, log aggregation, and microservices communication.
• Kafka components can be deployed both on-premises and in the cloud.
• The platform supports two main use cases: message delivery and state storage.

FAQ