12 What’s next for Kafka
Kafka emerged from LinkedIn’s need to unify massive, low-latency data flows with durable replay, replacing brittle batch pipelines with a central, distributed commit log. Since then, it has matured into a full platform for real-time event processing and integration. This chapter looks ahead to how Kafka will continue evolving: lifting developer focus from plumbing to business logic, extending into new execution environments, rethinking storage for cloud-native elasticity, and deepening its role as the backbone for ML and AI-driven systems.
The chapter envisions Kafka as an orchestration backbone powering declarative and low-code platforms that auto-generate and provision topics, schemas, and processing jobs, replacing today’s handcrafted integrations. It also explores Kafka’s fit with emerging runtimes: early WebAssembly experiments for broker-side transforms and browser interaction via proxies; serverless Kafka that abstracts capacity planning and bills by usage, best for stateless, bursty workloads rather than long-lived, stateful consumption; and edge deployments that run small, resilient clusters close to devices for local filtering and enrichment, with replication back to central clusters when connectivity permits.
On the architecture front, “diskless Kafka” proposes separating brokers from storage by persisting records in cloud object stores while a batch coordinator maintains ordering and offsets—unlocking elastic, largely stateless brokers and cost savings, with initial limitations (such as no compaction or transactions) and a tradeoff of higher read latency. For AI/ML, Kafka enables real-time inference through model servers, supports incremental learning that updates models continuously from event streams, and powers streaming feature engineering and serving—often via compacted topics as a live feature store. Together, these directions position Kafka as both the operational fabric and shared memory for event-native, autonomous AI systems.
Creating a streaming workflow. The user defines a workflow using a visual or declarative tool, and then deploys it to a streaming platform. As part of the deployment process, all necessary artifacts, such as Kafka topics, schema definitions, and Apache Flink jobs, are automatically generated and provisioned.
WASM programs are compiled into a lightweight runtime that executes alongside JavaScript in the browser. WASM interacts with HTML indirectly by invoking JavaScript functions to access the DOM or browser APIs.
Using Lambda functions to publish events. When an HTTP request is received, it triggers a Lambda function that processes the request and publishes the resulting data to a Kafka topic.
The data is replicated from the small cluster at the edge to the full cluster in the data center.
Producer path in diskless Kafka
The ML model is deployed in advance and is used by the Kafka Streams application for predictions.
Retraining a machine learning model incrementally
Summary
- Kafka’s role has expanded far beyond messaging infrastructure, evolving into a central platform for event-driven architecture and modern system design.
- Kafka can serve as a foundation for orchestration, enabling developers to model workflows and business logic with greater speed and abstraction, often supported by low-code and declarative tooling.
- Kafka is adapting to emerging runtimes, such as WebAssembly, serverless environments, and edge computing platforms, showing its flexibility in supporting next-generation deployment models.
- The proposal for diskless Kafka marks a significant architectural shift, enabling Kafka to decouple compute from storage, reduce cloud costs, and scale more efficiently in dynamic environments.
- Kafka is ready to play a critical role in machine learning systems, acting as the backbone for real-time inference pipelines, incremental learning processes, and continuous feature delivery.
- Kafka enables the construction of distributed AI agents, serving as a coordination layer that allows agents to observe, decide, and act autonomously through streams of events.
- Together, these trends point to a future where Kafka is not just a log, but the nervous system of intelligent, reactive systems that can adapt, respond, and evolve in real time.
FAQ
Why is there a push to move from handcrafted Kafka integrations to orchestration platforms?
Most Kafka apps today are built with low-level APIs and lots of custom wiring for consumption, retries, error handling, and schema management. This flexibility slows teams down and increases operational risk. Orchestration platforms raise the abstraction: they apply standard patterns, automate plumbing, and let developers focus on business logic instead of infrastructure details.How do low-code or declarative streaming platforms work with Kafka?
- You define logic visually or via configuration (for example, YAML), using predefined building blocks and optional code extensions.- Deployment tools package services, provision containers, and auto-create artifacts such as Kafka topics and schemas.
Platforms like Nussknacker and Quix use Kafka as the backbone, while Apache StreamPipes can plug into multiple messaging systems.
Can WebAssembly (WASM) apps connect directly to Kafka?
Browsers can’t speak Kafka’s TCP protocol; they support HTTP and WebSockets. Practical options include using a Kafka REST Proxy or a WebSocket-based proxy to bridge the gap. On the server side, broker-adjacent WASM modules (for example, wasm-based transforms) show promise for safe, in-broker filtering or enrichment. These approaches are experimental but point to new front-end and broker-side patterns.What is “serverless Kafka,” and how is it different from a managed cluster?
Serverless Kafka removes capacity planning and broker management entirely: you define topics and the platform handles scaling, failover, and throughput. Billing is usage-based (data volume) rather than on pre-provisioned brokers. It’s still real Kafka with the same protocol and semantics—just delivered in a serverless operational model.When is serverless a poor fit for Kafka consumers and stream processing?
- Cold starts add latency for sporadic workloads.- Kafka’s model expects long-lived consumers with stable partition ownership and offset management; short-lived functions cause overhead and rebalances.
- Stateful logic needs external stores, adding complexity and latency.
Use long-running services or frameworks like Kafka Streams or Apache Flink for low-latency, stateful, or exactly-once workloads.
Kafka for Architects ebook for free
