12 Stream processing

Stream processing extends Kafka from a fast, durable data bus and batch source into a real-time analytics and decision platform. Instead of waiting for nightly jobs, event-at-a-time pipelines react immediately to changes such as price updates, inventory shifts, or customer signals. The chapter positions stream processing alongside producers, consumers, and Kafka Connect, and contrasts its immediacy and responsiveness with batch’s definitive but delayed results. The trade-off is eventual consistency: intermediate answers arrive now and refine over time, enabling timely interventions in business workflows.

The chapter surveys the ecosystem—Kafka Streams as a Kafka-native, fault-tolerant library; Apache Flink with broader features and SQL; plus Faust (Python) and Goka (Go)—and advises against hand-rolling stream logic with plain consumers/producers. It introduces functional topologies built from processors like filter, map/mapValues, merge and split, and moves to stateful operations that maintain per-key state. A key conceptual bridge is streams and tables: KStreams/DataStreams for change events and KTables/Tables for current state, typically backed by RocksDB with changelog topics for durability. Aggregations (count, sum, average) need an initial state and an update function; joins appear as stream–table enrichment, table–table joins on primary keys, and windowed stream–stream joins, all relying on co-partitioned keys. Time is treated explicitly via event time, create time, log append time, and stream time, with timestamp extractors, grace periods for late data, and windowing patterns—tumbling, hopping/sliding, and session—to express temporal analytics.

Practical patterns illustrate these ideas: detecting significant price shifts, enriching sales with product master data, fan-out notifications by aggregating a user’s active devices and joining with new alerts, and fraud detection by counting authentication attempts per user within a minute. For operations and scale, Kafka Streams parallelizes by partitions into tasks distributed across instances and threads, with state recovery from changelog topics and optional standby replicas for faster failover. Correctness and performance hinge on keys and partitions: prefer mapValues to avoid unnecessary repartitioning, ensure co-partitioning for joins and aggregations, and expect sub-topologies and internal -repartition topics when keys change. Flink SQL offers a declarative alternative (often run headless) but requires performance review and a heavier operational footprint; Kafka Streams remains simpler to operate when you are already centered on Kafka.

In Batch-processing, a large amount of data is processed at-once at specific times and the results are definite. When processing one event at a time, data is processed almost immediately, but it is difficult to provide definite results, as the system is eventually consistent.

The filter and map processors.

The split and merge processors.

The join and count processors.

On the left we have a simple topology that filters data from a topic and writes it back to another topic. On the right, we merge two topics and then write the result to a topic and in parallel count the number of messages per key. The result is then also written to a topic.

We can convert streams into tables, and the transformation works in the opposite direction as well.

Processors like map, filter, and flatMap work on both KStreams and KTables. We use aggregations to create KTables from KStreams and toStream to produce a KStream from a KTable.

To count the number of messages per key, a state store with the current count per key is needed. The changes to the state store are stored in a compacted changelog-topic.

The topology for a Join

A streaming topology consisting of aggregation and join.

Stream processing frameworks support different types of time windows: Tumbling, Hopping, and Session Windows.

Kafka Streams topology to detect fraud attempts in real-time.

Kafka Streams applications are divided into tasks that are distributed across different threads on different instances. If an instance fails, another instance takes over the tasks.

If we first execute a selectKey() and then an aggregation like count(), the topology needs to be split up into multiple sub-topologies.

• Kafka enables near real-time data exchange, eliminating delays from traditional batch processing.
• Kafka Streams, Apache Flink, Faust, and Goka are common stream-processing libraries, each with different strengths.
• Stream processing processes events individually in real time but requires managing eventual consistency.
• Stream processors like those in Kafka Streams and Apache Flink transform, filter, and analyze real-time data streams using functional or declarative programming approaches.
• Processor types include filter() for selecting data, map() for transforming data, merge() for combining streams, and split() for branching data based on conditions.
• Stateful processing enables operations like counting (count()) and aggregating data, which requires storing intermediate states.
• Processor topologies connect multiple processors to build complex workflows, such as merging streams, filtering significant changes, and aggregating data for insights.
• Streaming SQL allows real-time data processing similar to traditional SQL but continuously outputs results instead of a final response.
• Flink SQL enables SQL-based stream processing by defining tables linked to Kafka topics rather than storing data within the Flink cluster.
• Queries like filtering (WHERE), aggregating (GROUP BY), and counting (COUNT(*)) work similarly to database SQL but operate on continuous data streams.
• Production usage typically relies on headless mode, where predefined SQL queries run continuously, with results written back to Kafka or other databases.
• Streams capture continuous data, while tables store the latest state for easier querying. Kafka Streams and Flink support both, using key-value stores and changelog topics to prevent data loss.
• Aggregations are used to determine the current state, like summing transactions for an account. Results are stored in tables for further analysis.
• Joins enrich data by combining streams with tables or other streams. Stream-table joins trigger on new stream data, while table-table joins trigger on changes from both sides. Stream-stream joins are possible with time windows.
• Time in Kafka involves Event Time (event occurrence), Create Time (producer creation), Log Append Time (broker write time), and Stream Time (processing time).
• Time-based operations in Kafka rely on time windows to aggregate or correlate data.
• Common window types include Tumbling (non-overlapping), Hopping (overlapping), and Session Windows (based on inactivity).
• Kafka Streams parallelization relies on splitting tasks based on partitions. Tasks are the smallest processing units and are distributed across Kafka Streams instances.
• State stores are created per task, and if an instance fails, the state store is rebuilt from the changelog topic. Configuring num.standby.replicas helps avoid rebuilding by using backup state stores.
• Repartitioning occurs when operations like selectKey() are performed. Kafka Streams creates an internal repartition topic for joins and aggregations to ensure correct processing based on keys, often splitting processing into sub-topologies.

FAQ