Kafka Storage Architecture

In this article we will check how Kafka stores and organizes the data.

  • Kafka records are organized and stored in a Topic.
  • Producer applications write data to a Topic and consumer applications read data from a Topic.
  • Topics are similar to the folders in a file system and the messages that are sent by producers are stored in the files in that folder. For example if you have a use case to send a weather data to Kafka, you will create a topic called “weather” and the data will be organized under it. Or if the use case is to send coordinates of a moving vehicle , you will create a topic with the name “location-coordinates” and the coordinates will be organized under it.
  • A topic is stored as log which is partitioned, replicated and segmented
  • In a topic , data is stored as a log and it is partitioned as per the configuration. Each topic can have 1 or more partitions. 
  • Partition is the unit of parallelism in Kafka. If you create a topic with 5 partition , then a producer can write to all 5 partitions for the topic in parallel.
  • Partitions are physical directory with the topic name.
  • Replication factor tells us how many copies for each partition should be replicated as the message is written. If you have a replication factor of 3, then up to 2 servers can fail before you will lose access to your data.
  • Partition * Replication Factor = Number of replicas
  • Replication factor is defined at topic level and Replicas are distributed evenly among Kafka brokers in a cluster.
  • Let us create a topic with below configuration and see how this is stored on each of the broker.
    • Topic Name : Inventory
    • Partition: 5
    • Replication-Factor : 3
kafka-topics --create --zookeeper localhost:2181 --topic inventory --partitions 5 --replication-factor 3
  • If you will check the data folder for the brokers, you will see something like this
  • There are 5 folders created with the topic name. Each of the 5 folders are 5 partitions.
  • We mentioned 5 partitions but there are total of 15 folders and that is because we created the topic with 3 replication factor. So while the data is saved in one partition , it gets replicated to other 2.
  • Therefore 5 partitions are replicated to 10 more partitions.
  • In this setting amongst each partition there is a leader partition and rest are followers. Kafka chooses one broker’s partition’s replicas as leader using ZooKeeper.
  • The broker that has the partition leader handles all reads and writes of records for the partition. Kafka replicates writes to the leader partition to followers . Leader partitions are responsible for ensuring that the follower partitions keep their records in sync.
  • We can find which is the leader partition and which is follower partition by describing the topic.
kafka-topics --describe --zookeeper localhost:2181 --topic inventory
  • From the above output we can see that for Partition 0, Broker 3 has the leader partition and followers are in other brokers. Similarly for Partition 1, Broker 1 has the leader partition.
  • Let us now send some data to this topic and see how this data is organized under Topic & Partitions.
  • Now before sending the data to the partition, i will make a change to the log segment setting of each of the broker. By default inside the partition folder data is stored in log and the default size of the file is 1 GB. When the data reaches 1 GB size, Kafka creates a new segment of the log file. For this example i changed 1 GB to 1 MB
log.segment.bytes=1000000
  • If your brokers are running, close the broker and restart the brokers
  • Let us now send some data to the topic which was created. i am sending a csv file with insurance data
kafka-console-producer --topic inventory --broker-list localhost:9092 < /Users/rajeshp/data/insurance-sample.csv
  • If you will open the partition folder , inside it you will find *.log file . When the size of the data crosses 1 MB, then a new segment file is created.
  • In each of the log file within a partition, Kafka maintains the order of data. The data offset starts with 0 and it increases. The log file name also starts with the starting offset number it holds for that partition. This is how data is segmented within a partition.
  • In order to find data , Kafka requires 3 things- Topic Name, Partition and Offset number.
  • Kafka indexes this data and that is stored in .index file . This is also segmented
  • There are use cases to fetch data based on timestamp and to support such cases Kafka also creates a time index and this is also segmented .

This was all about how Kafka stores the data . Share your comments and feedback.

Thanks !

2 thoughts on “Kafka Storage Architecture

Leave a Reply to Justin Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s