13 Governance

Effective governance is presented as essential for Kafka-based data platforms. As organizations depend on real-time streams, they need structured practices that preserve data integrity, ensure producer–consumer compatibility, secure access to data, and prevent any single client from exhausting shared resources. The chapter motivates this with concrete failure modes—schema drift breaking consumers, weak security leading to data exposure, and uncontrolled clients overloading brokers—framing governance as both a technical necessity and a business reliability imperative.

The chapter positions schema governance as the backbone of interoperability. It emphasizes documenting schemas centrally, planning for evolution, and choosing explicit compatibility strategies (none, backward, forward, full, including transitive variants) because Kafka retains historical data and supports many independent producers and consumers. Schema registries are introduced as the source of truth that version schemas, check compatibility, and integrate with clients, while also noting operational trade-offs (complexity, opaque IDs, migration challenges). Avro’s self-describing model and registry-backed IDs are used to illustrate efficient, traceable evolution. For enforcement, patterns include limiting write access, broker-side validation in commercial distributions, or validation via Kafka proxies, and teams are urged to route schema changes through CI/CD rather than allowing ad hoc version creation.

Security guidance spans transport encryption with TLS, client authentication via mutual TLS or SASL mechanisms (Kerberos, OpenID Connect, SCRAM), and least-privilege authorization with ACLs, along with pragmatic advice on encrypting data at rest, when true end-to-end encryption must be implemented in clients, and how to handle key management. The chapter also explains securing ZooKeeper where applicable and outlines a zero-downtime path to migrate from an unsecured to a secured cluster using additional listeners and staged client cutovers. Finally, it covers quotas as safeguards against overload—per-user/client byte-rate and request-percentage limits enforced per broker, using conservative defaults with targeted exceptions, understanding throttling behavior, and monitoring throttle-time metrics—so that clusters are protected from misconfigurations without penalizing healthy workloads.

We start with a simple JSON object for a product. Based on this, we can add, delete, or modify fields. From the beginning, we should consider which schema changes should be supported and what the process for this looks like.

We consider schema compatibility from the consumer's perspective and which schema versions they can understand. If we cannot or do not want to guarantee any compatibility, a consumer can only understand their own schema version. A backward-compatible consumer can read historical data but cannot handle newer data. A forward-compatible consumer can read new data, but not historical data. The optimum is a consumer that can read both historical and new data, which is fully compatible in both directions.

Securing a Kafka cluster involves transport encryption for communication, user authentication and authorization, and, lastly, encryption of data at rest.

An encryption proxy can reduce the risk of using an untrusted Kafka provider.

Required steps to secure an unsecured Kafka cluster.

• Proper schema management in Kafka helps ensure data consistency and compatibility.
• Using schema registries allows for the centralized management of data schemas, enabling easier updates.
• Schemas facilitate versioning, allowing producers and consumers to evolve independently without breaking changes.
• Compatibility checks can be enforced to prevent incompatible schema updates that might lead to data issues.
• Schemas can be serialized in different formats, such as Avro or JSON, enhancing interoperability across systems.
• Transport Layer Security (TLS) utilizing encryption, authentication, and authorization is mandatory for safeguarding data in Kafka.
• Authentication in Kafka can be implemented using various mechanisms, such as Simple Authentication and Security Layer (SASL), to verify the identity of clients and ensure secure connections.
• Authorization is managed through Access Control Lists (ACLs) to regulate which users or applications can access specific Kafka resources, ensuring only authorized actions are permitted.
• Data at rest must be encrypted using underlying storage systems, as Kafka lacks native support for this.
• End-to-end encryption is recommended to secure sensitive data, implemented at the client level.
• The key management process for encryption requires careful handling, potentially utilizing tools like Vault.
• Zookeeper must be secured as it stores critical Kafka metadata and transmits data unencrypted by default. Kafka 2.5 and later supports TLS connections for secure communication with Zookeeper, and mutual TLS is advised for broker-to-Zookeeper communication.
• It is possible to migrate from an unsecured Kafka cluster to a secured one without impacting availability, by sequentially configuring TLS listeners and restarting brokers.
• Quotas are essential for protecting Kafka clusters from excessive load caused by misconfigured or malicious clients.
• Quotas can limit the number of produce or consume requests and restrict CPU time on brokers.
• Quotas can be set at the client ID level or user level. User-level quotas provide better reliability.
• Kafka brokers monitor throttle-time metrics to detect and manage quota violations effectively.
• Care must be taken when setting quotas, as overly aggressive limits can disrupt service and lead to cascading failures.

FAQ