10 Cleaning up messages

Managing the lifecycle of messages is critical in Kafka to avoid unbounded storage growth, performance degradation, and legal noncompliance. This chapter explains Kafka’s two cleanup strategies—log retention and log compaction—which can be applied individually or together. Retention deletes data by age or size at segment boundaries, providing simple, predictable controls for data that only needs to live for a fixed period. Compaction, by contrast, keeps the latest record per key and discards older versions, making it ideal for datasets like customer profiles where only the current state matters.

Retention operates at the segment level: when a partition exceeds retention.bytes or when the newest record in a segment is older than retention.ms (seven days by default), Kafka removes the entire segment. Because deletion is segment-based, effective retention depends on rolling segments regularly (segment.ms) and on the periodic background check interval (log.retention.check.interval.ms). The chapter shows how to tune these settings to satisfy precise retention windows and cautions against accidental loss of important, older data. It also covers cluster/topic defaults (cleanup.policy, log.cleanup.policy) and offset retention for consumer groups via offsets.retention.minutes, typically kept at seven days or longer since offsets consume little space.

Compaction requires keyed messages and is driven by the log cleaner, which periodically evaluates partitions (log.cleaner.backoff.ms), prioritizes compaction based on the dirty ratio (min.cleanable.dirty.ratio), and respects minimum and maximum compaction lags to balance resource cost and consumer needs. Compaction never processes the active segment, so creating new segments (for example via max.compaction.lag.ms or segment.ms) helps older data become eligible. Compaction preserves offsets and order while removing superseded entries; selective, permanent deletions are achieved with tombstones (null values) that are themselves retained for a period (delete.retention.ms) so consumers can observe deletions. The chapter emphasizes the trade-offs—compaction’s overhead versus its ability to maintain authoritative state—and notes that retention and compaction can be combined for robust, space‑efficient data governance.

The log cleaner always operates on a segment-by-segment basis. With the delete policy, an entire segment is deleted once the newest message in the segment is older than a predefined value. If we use the compact policy, the log cleaner only deletes messages with a key for which there is a newer message with the same key. It's also possible to employ both methods simultaneously.

Log Retention checks whether a segment is larger than 1GB or whether the newest message is older than 7 days. If so, the whole segment is deleted.

Log Compaction reduces the amount of disk space used by deleting old messages for those there is a newer message with the same key.

By default, compaction is triggered when the number of dirty segments is larger than the number of clean segments. Often it is also useful to set the max compaction lag, too.

• Kafka offers two main approaches for message cleanup: log retention and log compaction.
• Log retention deletes messages based on age, while log compaction removes outdated data based on keys.
• Log compaction ensures that only the latest message for each key is retained, suitable for scenarios like updating customer data.
• Log retention is simpler to implement, clearing messages based on age and supporting various use cases such as data privacy compliance and changelog data management.
• Both methods have trade-offs; log retention is easier but less precise, while log compaction requires more effort but ensures data accuracy.
• Log retention can be configured based on partition size or time, offering flexibility in managing storage space.
• Log compaction parameters determine the threshold for segment cleaning and ensure that only outdated data is removed.
• Log compaction maintains message order and offsets, preserving data consistency.
• If a consumer tries to fetch messages which do not exist, the brokers simply return the next messages.
• Kafka's log cleaner periodically checks for outdated data and optimizes log storage by removing unnecessary segments.
• Regular segment rotation ensures efficient log management, preventing log bloat and optimizing storage usage.
• Kafka's flexibility in message cleanup allows for tailored data management strategies based on specific use cases.
• Tombstone messages facilitate selective deletion in log compaction, ensuring efficient data cleanup.
• Tombstones are eventually removed to prevent log inflation, with configurable retention periods.

FAQ