How to run a multi-node Kafka cluster in Local?

1. Download Kafka Confluent Community Edition
   • https://www.confluent.io/download/

2. Unzip the downloaded file

3. To run 3 clusters go to confluent-6.0.0/etc/kafka directory

4. Copy server.properties 3 times and rename it as

5. Next step is to change the configuration inside these property files. Properties to be changed are 3
   1. Broker id: Every cluster should have a unique broker id.Assign a unique id for this configuration, example
      • broker.id=1 # <<For server1.properties>>
      • broker.id=2 # <<For server2.properties>>
      • broker.id=3 # <<For server3.properties>>
   2. Listener port: This is the port at which producer will send data to Kafka broker and consumer will listen to this port for incoming messages. Therefore, in a single machine this should be unique for each clusterUncomment below configuration and assign a port number, example
      • listeners=PLAINTEXT://:9092 # <<For server1.properties>>
      • listeners=PLAINTEXT://:9093 # <<For server2.properties>>
      • listeners=PLAINTEXT://:9094 # <<For server3.properties>>
   3. Log file Directory: This is the location to store the data for each of the cluster
      • Assign a unique directory for each broker , by modifying below config
        • log.dirs=/tmp/kafka-logs-1 # <<For server1.properties>>
        • log.dirs=/tmp/kafka-logs-2 # <<For server2.properties>>
        • log.dirs=/tmp/kafka-logs-3 # <<For server3.properties>>

6. Now Run zookeeper by giving below command. Change the path if you are doing this on windows
   • zookeeper-server-start ../etc/kafka/zookeeper.properties

7. Open 3 terminal for each broker and start the broker by giving following command.
   • For Server 1 : kafka-server-start ../etc/kafka/server1.properties
   • For Server 2 : kafka-server-start ../etc/kafka/server2.properties
   • For Server 3 : kafka-server-start ../etc/kafka/server3.properties

With this you will have a 3 node Kafka cluster running in your local.

Thanks !