10 Organizing a Kafka project

This chapter argues that a successful Kafka adoption is as much about process and governance as it is about code. It begins with business discovery—clarifying outcomes, mapping real-world workflows, and cataloging meaningful state-change events with clear ownership and semantics. From there, teams formalize functional requirements for topics and contracts (naming, keys and partitioning, replication, retention and compaction, schemas and compatibility, headers, consumer groups and assignment) and capture nonfunctional needs (message size and compression, latency targets via percentiles, throughput patterns and growth, security and compliance controls, loss tolerance, and dead-letter handling). Treating these as first-class requirements aligns cost, reliability, and performance expectations before any workload goes live.

To keep clusters consistent and auditable across environments, the chapter promotes a declarative, GitOps-led model over ad hoc CLI/UI changes and auto-creation defaults. Kafka artifacts—topics, schemas, ACLs, connectors—are stored in version control, applied by CI/CD, and guarded by validation and drift detection; collaboration flows through pull requests owned by producer teams. Automation can leverage Kafka Admin, Schema Registry, and Connect APIs; in platformed estates, Kubernetes operators orchestrate CRDs via tools like Argo CD or Flux, while non-Kubernetes setups rely on IaC alternatives. Environment strategy favors separate clusters per stage, or, when sharing is unavoidable, strict isolation via prefixes, distinct registries, ACLs, quotas, and controlled, masked migration of production data.

Quality assurance spans unit, integration, and performance testing tailored to event-driven components. Unit tests isolate service logic with mocks and validate producer/consumer behavior at API boundaries; Kafka Streams topologies are exercised with a dedicated test driver. Integration tests verify real communication paths using lightweight brokers (embedded) or disposable, Dockerized clusters (Testcontainers) to cover serialization, offsets, security, and rebalances. Finally, performance tests use producer/consumer benchmarks and load-testing tools to measure throughput and latency percentiles against the stated nonfunctional requirements, feeding data back into tuning, sizing, and quota decisions.

Applying Kafka cluster modifications via GitOps: After the updated Kafka cluster configuration is pushed, the CI/CD pipeline retrieves the current cluster state, calculates the required modifications, and applies them.

Collaborating on changes via pull requests. Developers from the consumer team use pull requests to propose changes, which are then reviewed and either approved or rejected by the data owners (producer team).

Applying changes with a Kubernetes operator. Argo CD synchronizes the desired state from Git with the current state of custom resources in the cluster, and it applies the differences automatically.

Unit testing of the ProfileService component. The test invokes the service with test input and verifies its behavior without involving a Kafka cluster. Evaluation is done by inspecting the return value or verifying that the producer was called as expected.

Unit testing the Customer360Service component. The test directly invokes the consumer’s handler method with a deserialized Kafka message, simulating message delivery without involving a Kafka cluster. The test then evaluates the service’s behavior by inspecting its side effects, such as interactions with downstream components or state changes.

Unit testing Kafka components using mock implementations. The services are tested in isolation by replacing the actual Kafka producer and consumer with mocks. This allows the test logic to focus on verifying functional behavior without involving a Kafka cluster or real network communication.

Integration testing of the ProfileService producer. The test invokes the service, triggering the actual producer code that sends a message to Kafka. A test consumer subscribes to the topic and receives the message from the Kafka cluster. The test then evaluates the content and metadata of the received message to verify correct delivery and structure.

Integration testing of the consumer component in Customer360Service. The test sends a message to the Kafka cluster using a test producer. When the message is consumed, ArgumentCaptor is used to capture the parameters passed to the service logic. The test then evaluates the captured data to verify that the consumer received and handled the message correctly.

Integration testing using an embedded Kafka broker. ProfileService sends a message to an in-memory Kafka instance provided by the EmbeddedKafkaBroker component. The test code verifies the message by consuming it from the embedded broker and evaluating its content and metadata.

Integration testing of the consumer logic using an embedded Kafka broker. A test message is sent to the in-memory broker, which is consumed by Customer360Service. The test verifies that the consumer processes the message correctly, without requiring an external Kafka cluster.

Integration testing of ProfileService using a Kafka cluster launched via Testcontainers. The test code starts a real Kafka broker inside a Docker container, invokes the service, and validates that the message was correctly sent to the Kafka topic.

Integration testing of the Customer360Service using a Dockerized Kafka cluster. A test message is sent to the cluster, and the consumer processes it as it would in production. The test verifies the consumer’s behavior by evaluating the resulting side effects or service invocations.

• Business-first event modeling is essential. Start by identifying meaningful state changes and who needs to react. Don’t just jump ahead to topic design.
• Event catalogues bridge business and tech. Capture each event’s purpose, producer, consumer, and data long before thinking about serializers or wire formats.
• Functional requirements help validate Kafka as the right fit. They also guide topic design, including ordering, partitioning, retention, and data sensitivity.
• Nonfunctional requirements shape cluster sizing and tuning. Collect data on latency, throughput, and message size to configure producers, consumers, and brokers effectively.
• Declarative GitOps workflows promote consistency. Using tools like Kafka-GitOps or Terraform helps maintain reliable, version-controlled topic and schema definitions.
• Unit and integration testing are critical for Kafka clients. Frameworks like Mockito exist to simulate producers, consumers, and end-to-end flows in automated tests.
• Performance testing reveals real-world behavior. Use tools like JMeter and Trogdor to experiment with different configurations and validate how they impact throughput and latency under load.

FAQ