Appendix A. Setting up a Kafka test environment

This appendix outlines how to create a realistic, multi-node Kafka test environment on a single machine for learning and experimentation. Instead of a single-broker demo, it uses three brokers to observe behavior under partial failures and to mirror key aspects of a real cluster. Each broker also acts as a controller (dual mode), with brokers listening on ports 9092, 9093, and 9094, and controller listeners on 9192, 9193, and 9194. The environment is explicitly intended for testing and not for production use.

The setup works across major operating systems because Kafka runs on the JVM; it was validated on macOS Sequoia (Apple Silicon) and Ubuntu 24.04 LTS, with WSL recommended for Windows users. The process involves downloading the latest Kafka release, extracting it to a local directory (for example, ~/kafka), reviewing its standard subdirectories (bin, config, libs, etc.), and adding the bin folder to the PATH for convenient use of command-line tools. Three broker configuration files are created, each defining a unique broker.id, data directory, dual listeners for PLAINTEXT and CONTROLLER, process roles, a shared controller quorum (listing all three nodes), the designated controller listener name, and a simple security protocol map for PLAINTEXT traffic.

Before starting the cluster, dedicated data directories are created and formatted with a single, shared cluster ID to prevent mixing data across clusters. Brokers are then launched—initially in the foreground to catch configuration issues—where early warnings about unreachable peers disappear once all nodes are up. After verification, brokers can be restarted in daemon mode to run in the background. Finally, a broker API versions command can be used against the three bootstrap addresses to confirm each node is online and to validate the cluster is operating correctly.

Figure A.1 Our test environment consists of three Kafka brokers in dual-mode.

~/kafka/bin/kafka-server-stop.sh

#!/bin/bash
BROKER_ID="$1"
 
if [ -z "$BROKER_ID" ]; then
  echo "usage ./kafka-broker-stop.sh [BROKER-ID]"
  exit 1
fi
 
PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep "kafka${BROKER_ID}.properties" | grep -v grep | awk '{print $1}')
 
if [ -z "$PIDS" ]; then
  echo "No kafka server to stop"
  exit 1
else
  kill -s TERM $PIDS
fi

$ chmod +x ~/kafka/bin/kafka-broker-stop.sh
$ ~/kafka/bin/kafka-broker-stop.sh 1

$ kafka-broker-api-versions.sh \
    --bootstrap-server localhost:9092,localhost:9093,localhost:9094
Connection to node -1 (localhost/127.0.0
.1:9092) could not be established. Broker may not be available.    #A
localhost:9093 (id: 2 rack: null) -> (
# A lot of text
)
localhost:9094 (id: 3 rack: null) -> (
# A lot of text
)

FAQ