2 Kafka cluster data architecture

This chapter builds an architectural understanding of a Kafka cluster’s data layer: how topics and partitions organize and scale data flow, how messages are structured, how replication protects availability, and how data is stored and retained on disk. It contrasts streaming topics with state-storage topics by introducing non-compacted versus compacted retention modes. The aim is to equip architects to make decisions about throughput, ordering, durability, and cost before tackling client code.

Data is produced to named topics that are split into partitions, the unit of parallelism and redundancy. Producers append records to specific partitions; offsets mark their position, and consumer groups scale reads by assigning partitions to instances. Ordering is guaranteed only within a partition, so keys are used to co-locate related events while balancing load; changing partition counts later risks reordering. Replication is per topic, with leader–follower roles, in-sync replicas, and write guarantees governed by min.insync.replicas; a common resilient baseline is replication factor 3 with ISR 2, and consumers may read from followers. Practical guidance also covers partition-count selection, the impact on failover and metadata, and avoiding skew.

Messages consist of value (payload) plus key, headers, and timestamp; clients batch messages for efficiency and identify records by the tuple of topic, partition, and offset. Brokers persist logs as segmented files with indices, rely on the OS page cache, and enforce retention by time and/or size, with LogAppendTime often preferred for predictability. When only the latest value per key matters, compacted topics act as a durable changelog: keys are mandatory, deletions use tombstones, and background cleaners rewrite old segments without changing offsets, triggered by dirtiness ratios and compaction lags. The chapter closes with lightweight governance practices—capturing topics, partitions, replication, retention, and message contracts in AsyncAPI—so designs remain discoverable and evolvable.

An illustration of how data is processed and stored in Kafka. Data is organized into topics, which are further divided into partitions. The basic units of information are messages, which contain keys, values, and headers.

Kafka topics, where each distinct type of message is segregated into its designated topic. Here, we have two services functioning as producers, each sending data to its respective topic. Additionally, a single consumer service subscribes to and consumes data from both topics.

A topic is divided into partitions, each with its own index. When a producer sends a message, it specifies the partition to which the message should be directed.

Utilizing partitions for parallel consumption. In this setup, each thread within a service instance is allocated its own partition, with the total number of partitions determining the maximum degree of parallelization achievable.

All the partitions can be assigned to the same consumer instance.

Partitioning messages by customer name. This figure shows how messages related to the same customer are directed to a specific partition, ensuring cohesive data management.

Routing messages to the same service instance for processing

Partitions are distributed across various brokers in the cluster. The replication factor, which specifies the number of partition copies, is determined per topic. In the event of a broker failure, the data remains accessible.

Each partition has one replica designated as the leader, while the others serve as followers.

Selecting the key for the Balances topic. The account number is used as the key, ensuring that messages for the same account are routed to the same partition. This setup guarantees that updates for each account are received in the correct order.

An example of Kafka message

Each message within the partition is assigned an offset.

Kafka log files, index files, and other related files (less relevant for the current discussion)

A topic where only the final message is important for the consumer. The DashboardService is only interested in the latest balances, not their provenance or history.

During the compaction process, the data within a topic is rewritten. For each key, only the latest value is retained, and messages where the value is null are deleted.

• Topics act as destinations that group messages together, identified by their unique names.
• These topics are divided into partitions, facilitating parallel data processing. Message order is guaranteed within each partition. However, no ordering is guaranteed across different partitions within the same topic.
• Partitions are replicated across different brokers to ensure redundancy, with the number of replicas determined by the replication factor.
• Messages consist of keys, values, and optional headers, and they also include a timestamp and other metadata. Keys play a crucial role in partitioning, ensuring that messages with the same key are directed to the same partition.
• The offset signifies a message’s position within a partition, aiding in tracking which messages have been consumed by the consumer.
• Messages in Kafka are retained based on specific criteria, such as a defined time period, the size of the topic, or a combination of both factors.
• Messages are consistently persisted on the disk of the broker in batches. To optimize disk usage, expedite read and write operations, and enforce retention policies, the data is partitioned into segments.
• Specially configured topics, known as compacted topics, store only the latest value for each key in Kafka.
• The organization of Kafka topics can be documented effectively using AsyncAPI.

FAQ