Apache Kafka No Longer Requires ZooKeeper | Kraft Mode | Detailed Explanation | JavaTechie

hi everyone welcome to Java techy recently I have asked one question in my community that is can we run Apache Kappa without zookeeper and I could see that people are getting confused cool this is interesting right let me clear this for you yes this could be possible after the Kafka 2.8 version onwards we can run Apache Kappa without using zookeeper so in this tutorial I will demonstrate how to run Apache Kappa without Zookeeper okay so without any further delay let's get started thank you [Music] so before we start let me give you some hints about the role of Zookeeper and advantages of using Kafka without Zookeeper okay usually Kafka uses zookeeper to store and manage all the metadata information about the Kafka cluster Kafka also used the Zookeeper as a centralized controller that manages all the Kafka Brokers or servers however in the new Kafka version instead of storing all the server config information in Zookeeper you can store them as a topic partition inside the Kafka server itself okay so that is what the advantages of using Kafka without zookeeper so you no need to keep all the Kafka clustered metadata inside the Zookeeper rather you can store them in a cup cup topic partition itself in the Kafka server okay but to start with the Kafka without zookeeper you should run Kafka with Kraft mode or Kraft metadata mode which is also called Kraft okay don't worry I'll show you step by step how you can start Kafka server in the Kraft mode without using the Zookeeper okay also I will show you the internal how this Kafka without zookeeper stores the metadata okay so to begin with this first we need to go to the Kafka folder where you installed it so we installed it in our download softwares folder you can see the latest version of Kafka we are using 3.4.0 okay just go inside this particular version then go inside the config if I'll Zoom this for you can you see something called Kraft I was talking about this particular mode just open this particular folder you will find broker dot properties controller dot properties and server.properties we are not able to see something called zookeeper.properties okay it means if you will start in this particular Kraft mode you no need to start the Zookeeper directly you can start the Kafka server okay so to prove that we can start Kafka without zookeeper so I'll go to the official page of Kafka I'll just open the Apache Kappa now you can see here there is a option Kafka let me Zoom this for you with K wrapped okay so you need to follow these three steps first you need to generate the cluster uid I mean it will generate one random string for you so you can use this particular string to format the log directory that is the next step okay so we will follow these two steps then the direct command to execute the Kafka server so first let me run this just go to the terminal or go to the folder go to the I mean let me open the terminal let's run this okay now I need to format the log directory by taking this cluster ID which we generate just now so let me run the second command okay now this is clearly visible let me run it so what this will do it will take the cluster ID and it will create a folder called Kraft combined logs inside this particular log directory it will append your cluster ID and will store the metadata okay I'll show you that in Practical how it stores the metadata fine now if we'll go to the server.properties file that is where we have done the changes right I mean we have modified the log directory this is your right so if you go here and if you go inside the bin not bin config kerat and if you open this let me see if I can open it let me Zoom this there is something log basic can you see this this is the log directory which we have and we have generated the random uid to specify the new log directory inside this particular folder okay so I'll show you this but before that first let's start the Kafka server and then we'll create a topic and will push some message to that then you can see the proper log directory okay so what I will do I will go to the browser and I will start the Kafka server so we have not started any zookeeper related stock let's verify without zookeeper whether you can able to execute this Kafka server start.sh or not let me copy this go to the terminal and I will just hit this particular endpoint I mean this command can you see here we're able to start our Kafka server without using zookeeper so here is the response right it started on Port 9092 and these are the log statement when you start our Kafka server right now simply we can create the topic and we'll play with the producer and consumer so what I can do I will open the command which we execute in our last video so I'll create the topic first so to create a topic I can execute this particular command so I'll open a new terminal just execute this which will create a topic let me Zoom this for you which will create a topic with three partition and one replication factor and topic name is new topic okay just enter it it created the topic now let's start our producer and consumer to play with this particular topic with some messages okay so I have the comment handy with me even I share this particular Link in the video description so that you guys can familiar with the command or you can keep this as a reference okay now I need to start the producer to produce some messages just run this I will also need to open the consumer right so I will create a new terminal and I will just paste the command to start the consumer so we have started the consumer now what I'll do I'll put some messages from the producer and we'll see in the consumer so better I'll make this in a proper format so that we can compare the producer and consumer this already we discussed in our last CLI video so but I'm just trying to demonstrate without zookeeper also you can start the Kafka and you cannot power producer and consumer they can happily play Okay cool so I will just send some messages hi My Consumer can receive it right let me Zoom this for you welcome some random string with number can you see here whatever the messages we are sending here we can able to see in our consumer I mean the entire Kafka ecosystem is perfectly working without using zookeeper that is what the moral to start the Kafka server without zookeeper in Kraft mode now let's go one step ahead and we'll try to understand if now all the metadata is not storing in the Zookeeper since we didn't start the Zookeeper then how Kafka server manages the metadata of the broker or the or of the server so we'll just verify it right away now if you go to the directory we have the log directory called Kraft hyphen combined logs right now you can find it inside temp folder so what I'll do I will just go to the temp folder so I can go here go to the folder can you see here the folder structure so if You observe carefully it created a folder called cluster metadata right and if you go inside this particular folder you can find the log file so we'll decode this particular log file to see how the clustered metadata looks like how it created the topic how it started the server how I mean each and every steps in the clustered metadata we can extract from this particular log in a moment but before that let's verify other folder there is the consumer offset I mean again the structure looks same but it will keep the information about the consumer offset once you publish the message to the consumer this offset will be maintained in this different directory okay and if you will scroll down there is something called meta dot properties let me open this this is what the random uuid we generated and that is assigned to our cluster ID hope you remember the steps we have executed the first step we have executed to generate the cluster ID using the random uuid and that is what the cluster ID assigned here can you see here and version is one something is there and then these are the three different partition we created for our topic new topic zero then 1 and 2 okay then something called bootstrap checkpoint let's open this I don't understand which language I mean it's in different format so similarly you can play with each and every folder to understand what exactly it contains or how it manages the data but before that we are focusing more on the Kraft cluster metadata so what I'll do I will try to decode this particular log file to verify various state in the Kafka okay so for that what I can do simply I will copy this particular log file and I'll go to my folder not here I mean the cop car installation folder and I will paste it inside the bin okay I mean you can execute from that temp directory as well but I am trying to do it from the pane itself okay so I copied it now there is something you will find called cop cut Dom log dot sh so let's source that particular cell script file it should start from d right see here there is something called Kafka hyphen dump hyphen log dot sh you can execute this cell script file by giving this particular log file to decode the log and to visualize the state of Kafka so I will simply execute that so let's go to the terminal for now let me close this consumer and producer I mean we understand that we can start the Kafka without zookeeper so it does not make any sense to again produce and consume so let me close one of them and what I will do I will just run the command let me Zoom this for you this is the command we ran okay I want to run Kafka hip and Dom pipe and log.sh and I want to see the cluster metadata in the form of decode and then I am giving the log file path okay and I want to print the data log just run it now you can see here it gives us lot of statement right now if you will visualize them let's search for few of them okay I mean once you have the log file with you you can decode and you can understand all the log statement if you want but I just want to walk you through the Quick Step to understand so let me sort something called topic record this should be hyphen right okay spelling mistake topic hyphen record so this should be underscore I guess topic underscore record can you see here is one value size 29 sequence one and it topic record you can find something called Partition record and this is something different I mean this is what the statement you can find about your partition and topic similarly each component of Kafka you can find from this particular log you want to go one step ahead and if you want to check the status of your bootstrap server or if you want to describe your replication factor that also you can do from the command okay so just go through this log and try to understand the each statement if you have enough knowledge on the this particular Kraft mode okay otherwise to just give a confirmation this is where it maintains the state the topic created partition assigned all the statement you can find from this law okay now I want to describe the bootstrap server and I want to describe the replication factor or any other component how you can do that just go to the folder where you install your Kafka you can find something here Kafka hyphen metadata hyphen Quorum okay so let's check that see here this is the cell script okay using the cell script you can check the cluster status and replication status okay so what I'll do I'll just run this particular command by giving the bootstrap server so I will go here I'll clear everything I want to run this by giving the bootstrap server where is my Kafka server is up and running and I want to describe the status of it just enter it can you see here this is what the cluster ID the uid we created leader ID 1 leader Epoch one I don't understand all the component what it suggests but to just verify the Kafka cluster you can find this command okay and if you want to see the replication just modify this command you want to describe the replication okay we have created one so just enter this you can find this node at equal to 1 login offset equal to this last phase timestamp this is about your replication Factor okay so this is a you can verify the Kafka component and it clearly says that in Kraft mode it created its own metadata and it stored the log and everything rather than depending on Zookeeper okay so this is what the way you can start your Kafka server without using zookeeper now the question I have is this recommended to use in production I can say yes if you are using any version more than 3.4 so I will just give you the statistic or the screenshot which I captured from the Kafka portal let me show you that the each version and which is ready for production and which is not can I zoom this okay this is not clearly captured but can you see here the Kraft Early Access mode is on the 2.8 version so this particular version is not recommended to use in the production and 3.3 onwards it is ready for production and 3.4 migration script Early Access 3.5 which is not yet released I believe migration script production ready use of Zookeeper deprecated still it is not deprecated for us because we are using the 3.4 and if you go in the Apache Kappa let me see what is the latest version is available click on download the latest version so far we have is 3.4.0 fine so 3.5 is completely deprecating the Zookeeper and 4.0 onwards you won't find zookeeper it will not support at all okay Kafka will completely remove the dependency of Zookeeper from there ecosystem from 4.0 onwards okay so these are the various Kafka version and you can state of the carrot without zookeeper fine now let's do a quick brush up to just understand what are the advantages we will found if we will not run Kafka without zookeeper or if I'll run my Kafka without Zookeeper okay so in that situation the steps are very simple just to observe just now we are eliminating system complexity we now need to rely on starting the Zookeeper then Kafka server and managing the metadata in the Zookeeper so we are avoiding it or we are just eliminating system complexity right now next is the data redundancy while running Kafka without zookeeper and the main advantage is I personally feel we are simplifying the Kafka architecture without any third party service or dependency we no need to rely on any third party service to manage the Kafka ecosystem metadata since it will be managed by Kafka itself so we are simplifying the architecture okay so these are the three key advantages we can achieve if you are running our Kafka without zookeeper in Kraft mode okay so just do let me know in a comment section if you guys have any doubt I found this is really interesting and I am more excited to word for the other religious so that we can completely decoupled from zookeeper itself that's all about this particular video guys thanks for watching this video meet you soon with A New Concept

Channel: Java Techie

Views: 30,133

Keywords:

Id: k_JbBYjZ6G4

Length: 20min 15sec (1215 seconds)

Published: Sun Apr 16 2023