Kafka Internals - Handson.txt - Single Node-Multiple Broker Cluster Setup In'Kafka Premiera Ola we described the steps to configure Single node-Single

Kafka Internals - Handson.txt - Single Node-Multiple Broker...

This preview shows page 1 - 3 out of 20 pages.

Single Node-Multiple Broker Cluster - SetupIn 'Kafka - Premiera Ola', we described the steps to configure Single node-Single broker cluster.Following are the steps to configure a Single node-Multiple broker cluster.Start the Zookeeper server.Create three separate brokers and change their configuration parameters.Run each broker in three separate terminals.Create a topic Multibrokerapplication with replication factor 3.Create a Producer and send messages to the topic.Start consumer and consume messages.Starting Zookeeper ServerFollowing is the command to start the Zookeeper server.bin/zookeeper-server-start.sh config/zookeeper.propertiesBrokers and Parameters Configurationi. Make copies of config/server.properties file into two new config files config/server-one.properties and config/server-two.properties, for broker 1 and broker 2 respectively.ii. Edit the following parameters of config/server-one.properties.The id of the broker must be set to a unique integer.broker.id=1The port to which the socket server listens on.port=9093A comma-separated list of directories in which the log files are stored.log.dirs=/tmp/kafka-logs-1Repeat the same configurations for config/server-two.propertieswith appropriate parameters set.Running Each Broker in Three TerminalsAfter changing parameters for the two copies of server.properties file, open three separate terminals, one for each broker and start Kafka server one by one.Broker1bin/kafka-server-start.sh config/server.propertiesBroker2bin/kafka-server-start.sh config/server-one.propertiesBroker3bin/kafka-server-start.sh config/server-two.propertiesNow, our machine is running three Kafka brokers.Creating a Topic with Replication FactorNow, let us create a topic Multibrokerapplication with replication factor 3 as we have three separate brokers.Let us use the command below to run in the terminal.bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 -partitions 1 --topic MultibrokerapplicationChecking Topic and Partition Summary
Background image
Using the following command, you can check the summary of partitions, topic name, replication factor and the in sync replicas.bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic MultibrokerapplicationThe output of the above snippet is given below:Topic:Multibrokerapplication PartitionCount:1 ReplicationFactor:3 Configs:Topic:Multibrokerapplication Partition:0 Leader:0 Replicas:0,2,1 Isr:0,2,1From the above output, we can understand Multibrokerapplication has one partition with three replicas.The first broker (broker.id 0) is the leader of the partition and all other brokers replicate this partition.Creating a Producer and Send MessagesCreate a Producer to send messages to the topic Multibrokerapplication. You willget a prompt. Type messages to be sent to the topic.
Background image
Image of page 3

You've reached the end of your free preview.

Want to read all 20 pages?

  • Summer '17
  • producer, Application programming interface, Kafka Broker

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture