17 Comparison with other technologies

This chapter explains where Apache Kafka fits among common integration and data technologies and stresses that architectural choices depend on context and non-functional requirements. Using the “Data on the Inside vs. Data on the Outside” lens, it argues that service-internal data belongs in systems optimized for consistency and synchronous access, while service-external data should be exchanged in a service-agnostic, ideally immutable form. Kafka aligns with the latter: a durable, high-throughput, publish–subscribe log that enables loose coupling, asynchronous communication, and independent evolution of services in microservice environments.

Compared with classic messaging and ESB-style platforms, Kafka follows a “dumb pipe, smart endpoints” philosophy. ESBs centralize routing, transformation, and backpressure, often pushing messages and deleting them after consumption, which can create tight coupling, operational bottlenecks, and challenges for onboarding new consumers or replaying history. Kafka persists events, lets consumers pull at their own pace, supports replay, and is agnostic to platforms, formats, languages, and consumers—improving scalability and decoupling while shifting responsibility to services. With REST, the chapter highlights how synchronous calls can recreate monolithic entanglement and cascading failures; asynchronous, event-driven exchange via Kafka reduces temporal coupling and enables precomputation and responsiveness. Trade-offs remain: operating Kafka adds infrastructural overhead, teams may juggle internal and external APIs, and Kafka lacks a single, central ad hoc query interface—often addressed with stream SQL tools or downstream analytical systems.

Relational databases and Kafka are presented as complementary rather than competing. Databases excel at answering “What is?”—the current, consistent state for service-internal data—while Kafka captures “What was?”—the sequence of events from which any service can build the state it needs. Services consume Kafka topics (logs) to create local, fit-for-purpose materialized views using search, caches, relational stores, or stream processing frameworks such as Kafka Streams or Flink; Kafka Connect bridges data in and out of existing systems. Seen this way, Kafka becomes the core of a streaming platform—the organization’s “central nervous system”—augmented by connectors, stream processing, and automated governance (naming, ACLs, schemas, and Git-based workflows). Success, however, depends on aligning these practices with the organization and its broader architecture.

We differentiate between service-internal data, i.e., Data on the Inside, and data exchanged between services, i.e., Data on the Outside.

ESBs route messages across different locations, ensure data protection compliance, and support various message formats.

A slow consumer can slow down a producer in classic messaging systems.

Synchronous communication protocols like HTTP-REST often lead to strong dependencies between services. The failure of one service can cause domino effects, resulting in the failure of the entire system.

Relational databases excel with their consistency guarantees, but when a database is used by many services, this leads to very strong coupling, and even small changes must be coordinated with all teams.

Local materialized views with Kafka

Kafka as the central nervous system for data in the organization.

• IT architects must select the best technology for specific use cases, as solutions can vary between organizations.
• Data on the Inside is tightly coupled with services and requires high consistency, while Data on the Outside is service-agnostic and ideally immutable.
• Integrating new technologies like Kafka often involves adapting to existing IT landscapes rather than starting from scratch.
• Classic messaging systems aim to efficiently transfer messages but often require centralized management and can introduce operational complexities.
• Unlike traditional systems that delete data after consumption, Kafka retains messages, allowing consumers to access historical data independently.
• Kafka promotes decentralization, enabling teams to manage messaging autonomously, making it ideal for adapting to evolving product needs.
• HTTP-REST simplifies service communication but can create dependencies leading to systemic failures.
• Synchronous protocols like REST may cause cascading failures when services rely on each other for data.
• Asynchronous communication via Kafka enables real-time data flow but adds complexity in managing separate APIs for inside and outside data.
• Relational databases capture the current state of data but can create strong coupling risks when accessed by multiple services.
• Kafka focuses on the historical aspect of data, allowing services to independently reconstruct their states.
• We recommend using relational databases for internal data and Kafka for data exchange, enabling a flexible and decoupled architecture.
• Kafka can serve as the core of a streaming platform, enabling real-time data flow and rapid responses to organizational events.
• Stream processing tools like Kafka Streams and Apache Flink, along with Kafka Connect, facilitate near real-time data processing and enable seamless integration with third-party systems.
• Effective management and automation, using a Git-based approach, enhance Kafka’s maintainability and agility across teams.

FAQ