Overview
6 Defining data contracts
This chapter explains how to turn business facts into durable, evolvable data contracts for Kafka. It frames events as immutable records whose structure is enforced by schemas, emphasizes that Kafka brokers do not interpret payloads, and positions a schema registry as the shared source of truth for versioned contracts. The author connects analysis-time event modeling to implementation-ready schemas, highlighting ownership, communication, and governance so teams can move reliably from whiteboard to production without losing semantics.
Design guidance centers on drawing the right boundaries and granularity for events and choosing among fact/state and delta styles, as well as composite, atomic, and aggregate patterns—each with trade-offs in size, resilience, ordering, and reuse. A notification-then-pull approach is discussed for its simplicity and coupling risks. Contracts map to Kafka through three elements: event structure (schema), channel (topic), and triggering logic (business semantics). The chapter surveys common schema formats (Avro, Protobuf, JSON Schema) and their trade-offs, then covers governance: who owns contracts, how to publish and discover them, and how to keep analytical specifications aligned with code. It also explains how subjects in the registry decouple event types from topic layout via TopicNameStrategy, RecordNameStrategy, and TopicRecordNameStrategy, enabling either single- or multi-type topics while preserving evolution rules.
Type evolution is managed with explicit compatibility modes (backward, forward, full, and transitive variants), schema registration practices (favoring controlled, prevalidated registration), and the operational realities of client-side validation, poison messages, dead-letter handling, and offset management. For breaking changes, the chapter outlines migration options: temporary compatibility relaxation, transformation services, parallel topics/pipelines, and full state rewrites for compacted topics, while noting pitfalls like environment-specific schema IDs. It reviews alternatives to a centralized registry and client-packaged schemas, along with their costs, and mentions enterprise extensions for richer constraints, metadata, policies, and migrations. A running example (Customer 360) shows pragmatic choices—fact events, Avro, and lightweight early flexibility—evolving into stricter governance as adoption grows.
Managing schemas in Kafka. Schemas are deployed via a REST API and stored in a Kafka topic. The Schema Registry provides access to these schemas through its API, allowing clients to retrieve and cache the necessary schemas locally.
An example of a composite entity. The UserProfile aggregates a collection of SocialAccount and Address entities. Together, they represent the complete state of a UserProfile.
Each entity is mapped to a distinct event. For example, if an address changes, only an AddressUpdated event will be sent, without any information about the UserProfile or SocialAccount.
The UserPhoneNumberUpdated event contains only the one changed attribute.
An intermediate service aggregates three events into one.
The UserProfileChanged event does not contain details about which attributes were modified. Instead, it serves as a simple notification indicating that a change has occurred. Upon receiving this notification, the client calls a synchronous service to retrieve the current state.
Due to the asynchronous nature of events, a service might request a profile that no longer exists.
When using backward compatibility, consumers are upgraded first. The new version of the consumer can handle the old messages.
With forward compatibility, producers are upgraded first and the old consumer must handle the new events.
The transformation to the old format is handled by a component in the middle. This assumes the creation of the new channel.
The new version of producer, consumer, and channel are deployed in the same environment as the old ones. After validation of the new version, the old one can be retired.
Interactions with the Schema Registry. The schema is registered with the Schema Registry server, which stores it in Kafka. Producers use this schema to serialize messages, and consumers retrieve the schema from the Schema Registry to deserialize messages.
How a message, containing a schema, is serialized. Consumers decode a schema ID and use it to retrieve the schema from the Schema Registry, which they then use to deserialize the data.
An example of backward compatibility. The new consumer must understand the previous version of the schema.
An example of forward compatibility. The old consumer must understand the new version of the schema.
Distributing schemas, serializers, and migration and validation rules as a part of the client package
Migrating messages using a special transformation service
Summary
- Designing events is challenging, requiring careful consideration of the right attributes and optimal granularity to balance flexibility, performance, and data relevance for consumers.
- Establishing clear rules for defining, updating, and communicating event changes is crucial to avoid chaos as systems scale. A best practice is maintaining a centralized, version-controlled schema repository to facilitate collaboration between developers and business analysts and ensure consistent event structures.
- State events capture the full state of an entity, while delta events convey only the changes. Each approach offers tradeoffs: state events provide simplicity and completeness, while delta events reduce message size and bandwidth usage but increase consumer complexity.
- Data contracts in Kafka are represented by schemas, which act as agreements between producers and consumers to define the structure of shared data, ensuring consistency and preventing misinterpretation across systems.
- The Schema Registry manages event schemas and ensures compatibility between schema versions, supporting both forward and backward compatibility to allow producers and consumers to evolve independently without causing system failures. It stores schemas in Kafka topics for versioning.
- Kafka brokers do not enforce schema validation—all message validation happens on the client side, making it critical for producers and consumers to correctly implement schema checks.
- As requirements change, schemas must evolve gracefully. Following best practices such as backward or forward compatibility ensures that new schema versions can coexist with old ones without disrupting the system.
- When breaking changes are unavoidable, migration strategies should be implemented, including phased deployments and transformation components to convert older event formats into newer ones, ensuring a smooth transition without service downtime.
FAQ
What is an “event” and what makes up a Kafka data contract?
An event is an immutable fact that something happened at a point in time (with a timestamp and context). A Kafka data contract comprises: 1) the schema that defines the event’s structure, 2) the channel (topic) it is sent on, and 3) the business semantics of when/why it is produced (usually documented outside Kafka). Producers (the domain owners) typically own these contracts; the Schema Registry standardizes how schemas are shared and evolved but does not convey business meaning.How does Schema Registry work with Kafka?
Schema Registry is a separate, highly available service that stores schemas (Avro, Protobuf, JSON Schema) in a compacted Kafka topic named _schemas. Schemas are versioned immutably and managed via REST. Producers serialize records with a format-specific serializer that embeds a schema ID in each message; consumers read the ID, fetch the schema, and deserialize. Validation happens on clients; brokers treat payloads as opaque bytes. Clients cache schemas to minimize lookups.Fact (state) vs delta events: when should I use each?
- Fact/state events carry the full entity state each time. Pros: simple onboarding, easy recovery (read latest), decouples producer from consumer logic. Cons: larger messages, broader data exposure, access control concerns.- Delta events carry only what changed. Pros: smaller, selective exposure. Cons: harder state reconstruction and ordering, limited reuse. Use fact events for broad, multi-consumer use and recovery simplicity; use deltas within tightly scoped integrations or when minimizing payloads. Group atomic multi-field changes (like address fields) to avoid invalid intermediate states.
What are composite, atomic, and aggregate events?
- Composite events bundle related changes (e.g., a whole profile) into one event—good when correlation/order across sub-entities is needed.- Atomic events represent a single small change (e.g., SocialAccountCreated)—offer fine granularity but require careful ordering and correlation on consumers.
- Aggregate events are produced by combining multiple upstream events (e.g., OrderCompleted after OrderCreated, InventoryChecked, ShipmentScheduled), often via a stream-processing/aggregation service so many consumers can reuse the final signal.
What is the “pull on notification” pattern and what are its trade-offs?
A lightweight event notifies that “something changed,” and the consumer synchronously pulls the current state from the producer. Pros: tiny events, producer stays unaware of consumer needs. Cons: tighter coupling, eventual-consistency gaps (state may be gone/older), and operational complexity (versioning, retries, fallbacks). Use with care; consider versioning and explicit handling of missing or stale data.How do schema compatibility modes work and what changes are allowed?
- Backward (consumers upgraded first): allowed to delete fields and add optional fields with defaults; new consumers can read old data.- Forward (producers upgraded first): allowed to add fields (old consumers ignore) and delete optional fields (old consumers use defaults).
- Full: both backward and forward simultaneously.
- Transitive variants check compatibility against all prior versions (useful for long-lived data).
- None: no checks (use temporarily for planned breaking changes). Avoid renames and type changes unless you plan a migration.
What is a “subject” in Schema Registry and how do subject strategies differ?
A subject is the unit of schema versioning/evolution. Strategies:- TopicNameStrategy (default): subject is tied to a topic (e.g., CLIENTS-value). Ensures one compatible schema lineage per topic.
- RecordNameStrategy: subject is the fully qualified record name (e.g., com.example.ClientEvent), enabling multiple event types in one topic (polyglot topics). Consumers must handle multiple schemas.
- TopicRecordNameStrategy: hybrid—subjects are per topic and record name, allowing polyglot topics while scoping evolution to a topic.
Can I put multiple event types in the same Kafka topic?
Yes, with RecordNameStrategy or TopicRecordNameStrategy. Use a partition key (e.g., entity ID) to preserve per-entity ordering. Consumers must detect the schema/event type per message and deserialize accordingly. Trade-offs: simpler ordering/correlation across types vs. more complex consumer logic and schema governance.How should I register and reference schemas in practice?
Prefer manual registration via the Registry’s REST API as part of your release process. Disable auto-registration and pin serializers to the latest registered version (Confluent settings: auto.register.schemas=false, use.latest.version=true). Store schemas in Git, reference them from specs (e.g., AsyncAPI), and notify consumers of new versions. Restrict write access and back up the _schemas topic. Note: these properties are Confluent-specific.What common pitfalls exist with Kafka data contracts and how do I mitigate them?
- No server-side validation: enforce serializers, add contract tests, monitor and route deserialization failures to a dead-letter topic, and have playbooks for resetting offsets past a poison pill.- Incompatible changes: plan migrations—pause old writers, register new schema (temporarily set compatibility to NONE if needed), use transformer services or versioned topics/pipelines, then cut over. For compacted topics, migrate all messages to a new topic preserving keys/timestamps.
- Environment-specific schema IDs: when moving data across clusters, use migration components to remap schema IDs. Choose formats and strategies that fit your team’s tooling and governance.
Kafka for Architects ebook for free
