7 Cluster management

This chapter explains how Kafka manages clusters in a distributed environment, why consensus is central to reliable operations, and how roles and metadata coordination enable consistent state across brokers. It contrasts past and present approaches to coordination, showing how the system elects and replaces controllers, keeps cluster configuration consistent, and ensures partitions and consumer groups are handled correctly. It also ties cluster management to client behavior, outlining how producers and consumers discover leaders and adapt to changes.

The modern approach, KRaft, is production-ready and replaces Zookeeper by embedding a Raft-based controller quorum directly in Kafka. Brokers run in distinct roles—controllers for coordination and brokers for data—and an odd-sized controller set elects an active controller with standby controllers ready to take over. Cluster metadata is persisted in an internal, replicated topic (__cluster_metadata) so all brokers share a consistent view. This architecture reduces operational burden, improves scalability and reliability (especially for large partition counts), and shortens failover times compared with Zookeeper-backed clusters.

The legacy model used Zookeeper to elect the controller and store metadata via ZNodes with Paxos-based consensus, but it added operational complexity, introduced latency, and encouraged local broker caches that could drift from Zookeeper’s state, leading to stale decisions and periodic polling delays. The chapter outlines a no-downtime migration path: stand up a dedicated KRaft controller quorum with the same cluster ID, enable metadata migration, let controllers mirror Zookeeper state, then switch brokers to KRaft mode, run safely in this mixed state, and finally disable migration and complete the cutover. It closes by detailing client connectivity: applications bootstrap to any broker (preferably multiple or via DNS/load balancing), request metadata to learn partition leaders, connect directly to the appropriate brokers, and refresh metadata on leader changes triggered by controller-managed failovers.

With Kafka 3.3, Zookeeper was replaced by a coordination cluster based on the Raft consensus protocol.

Before KRaft, a Kafka cluster consisted of multiple brokers and additionally a Zookeeper ensemble consisting usually of three or five Zookeeper nodes. The controller was simply one of the brokers. If the controller failed, another broker took over these tasks.

Before our clients can produce or consume messages, they need to know which broker is responsible for what. This is achieved through a metadata request.

• Kafka 3.3.0 introduced KRaft, a new coordination method based on the Raft protocol, replacing Zookeeper from Kafka 3.5.0 onwards.
• The Raft protocol resolves the complexities of achieving consensus in distributed systems, previously handled by Zookeeper in Kafka.
• KRaft integrates coordination directly within Kafka brokers, eliminating the need for a separate Zookeeper system, thereby improving scalability and reducing operational complexity.
• Controllers in KRaft manage partition assignments, leader elections, and failover mechanisms, ensuring cluster stability and facilitating efficient scaling and management of Kafka clusters.
• KRaft introduces the __cluster_metadata topic, ensuring consistent metadata storage across all brokers, which was previously managed by Zookeeper, thereby streamlining cluster operations and enhancing reliability.
• Despite Zookeeper being phased out in Kafka 4.0, a significant number of Kafka clusters currently rely on Zookeeper for essential functions.
• It's advisable to migrate these clusters to KRaft promptly to leverage improved performance and simplified management offered by the Raft-based coordination.
• Zookeeper serves critical roles in Kafka, such as electing the controller and storing metadata like partition assignments and leader information, ensuring consistent state across all nodes.
• While Zookeeper's reliability in maintaining consistent state is crucial, its inherent performance limitations and operational complexities have prompted Kafka's shift towards the more integrated KRaft solution.
• Migrating metadata from Zookeeper to KRaft in Kafka involves careful planning and execution without downtime. Start by upgrading your Kafka cluster to the latest version that supports Zookeeper.
• Provision three to five additional brokers as KRaft controllers with identical cluster IDs as the Zookeeper-cluster. Enable migration mode and configure Zookeeper connection details.
• Configure existing brokers for metadata migration by enabling migration mode and specifying new controller-node connections. Verify successful metadata migration.
• Transition normal brokers to KRaft mode by adjusting configurations, testing stability, and finalizing the migration. Disable migration mode on controllers and remove Zookeeper connections for full transition to KRaft mode.
• Connecting to a Kafka cluster involves querying metadata to understand broker roles and partition leadership distribution across the cluster.
• Clients initiate connectivity through a bootstrap server, typically one of several Kafka brokers listed in the --bootstrap-server parameter, which provides initial metadata.
• To ensure resilience, production setups should specify multiple brokers in --bootstrap-server parameters or utilize DNS for load balancing.
• If a broker failure occurs, clients can send a new metadata request to a bootstrap server to discover new partition leaders, enabling uninterrupted message production or consumption by establishing connections with the newly elected leaders.

FAQ