Apache Kafka® & Confluent Kafka CLI hands on tutorial | Producer & Consumer Flow | JavaTechie

hi everyone welcome to Java techy in this tutorial we'll understand Kafka producer and consumer workflow in detail using command line interface okay all right so before I start explaining the things I am assuming that you have gone through my Kafka components and architecture video if not then I would strongly suggest you to check that particular video then come back to this one so if you go to my YouTube channel you can find a video called Apache Kappa components and architecture so you must have to watch this particular video before you jump to the current one okay I'll share the link in video description so that you guys can refer it basically to understand this complete pops up workflow we need to play with all the Kafka components like topic partition replication Factor producer and consumer so I believe this tutorial will definitely give you good knowledge on Kafka internal okay now let's get our hands dirty on Kafka command line interface so without any further delay let's get started foreign [Music] distribution right now what next first we need to create or start a Kafka ecosystem for that we need to start a Kafka server or broker okay but to manage this Kafka server we need a manager which is Zookeeper so the steps we need to follow start the Zookeeper then next we need to start the Kafka server or broker ecosystem ready with you then we can start producer and consumer since we are playing with command line interface we'll use the command to create producer and consumer for now but in my upcoming session I will create two different application for producer and consumer okay cool next producer wants to send a message to the consumer but they can't directly communicate with each other so we need another component called topic so that producer can push the message to the topic and consumer can consume it okay once you have created the topic next you can Define n number of partitions to distribute the load coming from producer for example let's say my producer wants to send 10 messages then those message will distribute to all three partitions assigning with offset number so while creating a topic you need to specify the partition count you need to take the decision based on the load coming to your application how many partition will be perfect fit for your topic okay that decision you need to take while creating the topic once a message reached to the partition then immediately whoever consumer listen to it they can happily consume it so this is the typical producer and consumer flow we will follow the same steps to Deep dive further okay so using the command line interface we'll start the Zookeeper Kafka server will create the topic then I will show you how you can Define the partition then also will understand how the bulk message is being distributed to the multiple partition and how it is being consumed by the consumer okay so to play with the command line interface you need to go to the folder where you install the Kafka so it is there in my download folder I have created a folder called softwares inside this I have kept these two Kafka okay so I will show you how you can do using Kafka also I will demonstrate how we can do using the confluent Kafka Okay cool so First Step what we understand first we need to start the Zookeeper right then you need to start the Kafka server so what you can do go to the folder then go to the Kafka open a new terminal fine then I will also create couple of terminal just go here new terminal fine let's go with these three let me minimize this okay fine so first I need to start the Zookeeper so how can I start the Zookeeper you need to fire the command so there is a cell script file you will find inside the Kafka bin directory so if you go inside Kafka if you go inside bin you can find your something called zookeeper server hyphen start dot sh okay so what we can do we will run this particular command so this is inside bin folder zookeeper start server start dot sh okay now to start the Zookeeper it needs to read The Zookeeper dot properties file so again if you go inside the config you will find a property called zookeeper dot properties can you see here this properties it needs to start the Zookeeper server so you just need to Simply give the path of it it is inside config let me Zoom this for you it is inside config and the file name is zookeeper dot properties file now just enter it it will take few second to start the Zookeeper and the default Port it will run two one eight one okay so I can show you that let me filter it two one eight one can you see here the default Port is two one eight one now as per the presentation we understand after start the Zookeeper we need to start the Kafka server so go to another terminal let me use the second one here you need to start the Kafka server how can we start the Kafka server again if you go and check in your bin folder you can find something called Kafka server start dot sh this is the cell script file if you are using Windows you can go inside the windows and you can search for where is the Kafka server yeah this one you can run this Kafka hyphen server start dot batch file okay since I am in the windows sorry I'm in the Mac I am using the sh file so just go to the terminal and just run it is inside bin okay hyphen server start dot sh and to start the Kafka server it needs to read the properties file called server dot properties so if you go inside the properties file or config folder you will find something called server dot properties okay so you need to give this properties file to start the Kafka server so again give the path config slash server.properties just enter it yeah it take few second to start your Kafka server now to check which Port this Kafka server is running you can filter something nine zero nine two can you see here this is the default Port of Kafka server so just keep a note the default Port of Zookeeper is two one eight one and Kafka server or broker is 9092 okay so that's it now our Kafka ecosystem is ready now what is the next step if you understand correctly then the next step producer and consumer need to communicate with each other using the topic so you need to create a topic and we need to specify the partition and replication Factor okay so to run a topic again I will take the help from this command line interface okay now if you go inside the bin folder you will find something hyphen topic dot sh okay so what I can do I'll just run here I need to go inside bin then Kafka hyphen topic dot SS also I need to tell here while creating the topic I need to give my bootstrap server host and Port when I say bootstrap server it's about your Kafka server okay now I'll Define bootstrap St rap Dot server my bootstrap server is running on host localhost and Port is 9092 this is where my capca server is up and running right so let me make this white so that we can type in a single line okay I have ran the Kafka hyphen topic.sh and I am giving where is my Kafka server is up and running the host is localhost and Port is 9092 now what action you want to do I want to create a topic okay and what is the topic name you want to give let's say I want to give JT topic or let's give a valid name Java techy underscore or hyphen topic okay now how many partition you want to Define so if you wont Define anything it will take the default partition as a one but you have option to Define n number of partition so as of now I just want to Define three partition okay partition count equal to 3. now how many copy you want to keep that is called replication Factor right since we are running on a single broker so let's keep the replication Factor as 1. so you can Define replication Factor equal to 1. so there is nothing complex to remember this command or this particular command whatever we have type here I am just running my topic by giving the information where is my Kafka service up and running by defining the host and port and what action you want to perform I want to create a topic and the name of the topic is Java Tech and topic and the number of partition I want to assign as 3 to this particular topic and I want to keep a single clone of this particular broker so that is the reason I have defined replication Factor as one now click on enter so the mistake we have done we no need to add the dot it need to be hyphen okay so let me Zoom this for you now let me drag this yeah now let me enter it can you see here it created the topic Java techy hyphen topic you can Define n number of topic if you want so let me create another topic topic one okay now we have created two topic so to list down all the topic available inside your Kafka server what you can do you can simply run bin I mean same command Kafka topic.sh in this particular server just give me the list of topic okay so I can happily copy this command then you can run it here and just ask him to give me the list of topic now if you will enter we'll find the two topic Java techy topic Java techy topic one now if you want to describe this topic to know what is the number of partition replication Factor who is the leader so if you want to describe this topic that is also straightforward you can run a command so what you can do run the same command and ask him to describe the topic name so I just want to describe Java techy hyphen topic okay this is what the topic name we have right I just want to describe this run it what is the error okay I need to Define that I want to describe a topic right so describe then you can tell him what you want to describe I want to describe topic there is something wrong I might be doing some spelling mistake let's see bootstrap server 9092 okay I don't want this list because I want to describe so this is the mistake I mean you need to understand this particular command okay you just no need to remember it just understand what you want to perform okay now just enter it can you see here okay let me take the background as different fine now if You observe here the topic name is this and the number of partition we have defined three zero one and two okay and replica is zero so you can see here partition count equal to 3 replication Factor equal to 1 there is no additional config given so this is empty okay you can describe the topic if you want and also we have defined the number of partition so we are done up to the creation of topic and defining the partition now we want to play with the producer and consumer so since we are playing with the command line interface I will run the consumer and producer application using the command line interface or using the command okay so we'll run the producer we'll push some message to the topic and we'll see whether it is being consumed by the consumer or not next step will push bulk number of messages from the producer to the topic and we'll see whether the bulk messages is being getting distributed to the different partition or not how we can monitor those things we have installed the offset Explorer right so what I can do I will open the offset Explorer so what you can do here very simple step you just need to create a connection and to create a connection you just need to give a name of your cluster so to not confuse you what I'll do I will create a new connection let's say JT I'll give the cluster name Jetty hyphen con okay look close to 181 everything it will take by default okay so I will give Java techy hyphen new that's it click on test is now if I open this I have the connection to the cluster so that I can monitor everything going on this specific cluster okay now if I will expand this how many topic you can see here Java ticket topic and Java ticket topic one so this is the two topic we created and we can able to see inside this particular Zookeeper okay now if you will go and click on the data and if we'll hit this particular thing there is no data is getting filtered if you will check the topic one there is no data right we want to push the data and we want to verify whether it is going to the topic or not if it is going to the topic then we will verify whether it is being consumed by a consumer or not okay so what we can do quickly we just need to start a producer and then we just need to start a consumer fine so what we can do to start a producer you will find something called Kafka console producer.sh given by the Kafka binary go inside the bin and if you will search here can you see here there is something called Kafka console producer.sh to just kicked up by producer application and Kafka console consumer.sh to run this command to be a virgin consumer okay so we'll run these two command okay so what you can do go to the terminal this guy now I'll clear everything what I want to do here I want to let me what okay I'll minimize these two okay we started the Zookeeper and we started the Kafka server let's minimize it now to make it clear I am just running the producer application here so you just need to run go inside bin Kafka console right console hyphen producer dot sh and then you need to give all the list of Kafka server available okay so you can Define broker hyphen list currently we have a single broker which is running on localhost 9092 but in real time you can find n number of Brokers right so that is the reason you need to Define them inside this list now to which topic you want to push your messages from producer you need to Define that topic name so our topic name is Java techy hyphen topic okay that's it just enter it now it is asking you to push some messages so we can push the messages and we can verify in the Kafka offset Explorer but meanwhile let me start the consumer application so that parallel we can push the messages from here it will go to the topic then consumer will consume from it so I want to demonstrate the complete Pub sub flow so what I'll do I will open a new terminal to start a consumer so go here and I just want to open a new terminal to start a consumer what you can do go inside bin then simply let me Zoom this for you go inside bin then run Kafka console consumer.sh okay now where is your bootstrap to whom you want to listen you can Define bootstrap hyphen server which is running on Port localhost 9092 and 2 which topic you want to listen I want to listen from the topic foreign to listen from the beginning if yes just Define I want to listen from beginning okay don't do any spelling mistake while typing the command otherwise it will give you the error fine now let's run this consumer there is no messages being showing here so what I'll do to make it simple let me split into the two terminal so that we can see whether what we are sending we are able to receive it from the consumer or not so let me split these two tab cool now the left side is my producer and the right side is my consumer and they both can connect it using this particular topic Java Tech hyphen topic okay producer is publishing to this topic and my consumer also listening to this particular topic if you do the spelling mistake in the syntax or in the comment it don't work so let's see whether we are correct or not let me send some messages hi can you see here we're able to see the messages in right side let me Zoom this for you let me send something hello we can see here let me send something Java techy we can see here let's send welcome let me some random number okay so this is my producer application from producer I am sending the messages okay and this is the right part is my consumer let me minimize right part is my consumer he is able to consume it now whether this message is being passed through the topic or not how you can verify go to the offset Explorer now just refresh your topic okay we have these two topic Java ticket topic and topic one but we are dealing with this Java techy topic and there is a consumer offset okay I mean it will just assign the offset and it will just set up to what your consumer read okay now what I want to show you I want to show you the data in this topic the vertical topic just run it can you see here how many messages we have sent five messages so the offset count being getting increased by one zero one two three four and all the messages what we have been sent to this particular topic now from this topic consumer is able to listen the messages and we can able to see here right now if you deep dive one step ahead this particular topic we have created the partitions how many partitions we have created three zero one and two now if you check the messages in the topic let me show you the messages inside the topic I'll go to the data all the messages went to partition 0 but still we have two more partition partition one and partition two but why all the messages is going to partition 0 that is not in our hand that will be taken care by zookeeper and coordinator okay now we are playing with the very least messages so all the messages is going to a single partition now let's say I want to push thousands of row in single sort from the producer then maybe the Thousand row could be distributed to different partition okay this is what we understand from the theoretically now let's prove it what I will do I will send a bulk file CSV data through the Kafka and we'll see whether the data is being passed to the different partition or not so for that what I can do I will cancel the producer okay Let It Be My Consumer is keep listening I am not sending anything from the producer that is fine you won't find anything but once I will start publishing the message will keep my My Consumer will keep listening to it so what I want to do I want to publish a bulk messages so what I have done in my download section I have downloaded a user.csv file if I'll open this it's loading if you'll observe here the total number of row present inside the CSV file is thousand okay this is a triple nine and this is thousand including the header it is thousand one okay so I want to push this particular messages from the Kafka producer I want to push this user.csv file how we can do that that is very simple so you can simply Define the path of your file it is there in my download folder so I just need a path of my download folder so what I can do I'll just take the path of it PWD close it I want to send the entire users dot CSV file from my producer just enter it can you see here if I zoom My Consumer we can see all the Thousand messages in my consumer section it means the CSV file the bulk CSV file is being loaded to my topic can you see here all the messages including the header we can able to see the total number of row total number of row is thousand one so let's verify it in our offset Explorer now we are seeing only the five messages I didn't refresh it Let me refresh it can you see here there are so many messages see these are the CSV data ID first name last name email and this is the IP address that is what we have there in our uh CSV file okay now all messages went to single partition or different partition let's verify it partition 0 the data okay now let's go to the partition one it didn't receive any data the count is zero partition two also didn't receive anything okay so all the messages go into the partition zero but that is not always the same behavior what we can verify I will send it again or I have another file call customers dot CSV okay it is not there let me send the same users dot CSV fine all the messages again being sent here will how we can verify let me go to the topic fine now if you will see in the partition 0 the total number of record received by partition 0 is 1546. so first we published five messages then we publish thousand messages then again we publish next thousand messages so partition 0 all total received 1546 messages now if you open the partition one it receives 461 messages okay partition two didn't receive anything so whoever partition will get the messages is not in our hand this will be handled by The Zookeeper or coordinator okay now partition 0 and partition one have the messages so don't worry we'll demonstrate this behavior from our application once we will create the producer app and consumer app this is what we are just trying to understand the behavior of partition and distributing the messages to multiple partition through the command line interface okay so I believe we are very clear about the producer consumer workflow and we have discussed everything what we have covered in this particular slide we have started the Zookeeper we created the Kafka server we create the topic Define the partition and also we understand how the load is getting distributed to the partition how the offset is being assigned how the message is being published from producer to the topic from the topic to the consumer the entire workflow we have covered just now okay the next whatever we have discussed everything on open source Apache Kappa now the next I will walk you through a small steps how you can play with the confluent Kafka okay so which is the Community Edition we have installed or we have downloaded yesterday right so if you go to the desktop if you will go to the downloads software we have this confluent right conflict now to start this conflict you need to stop the open source Apache Kappa because the Zookeeper server and Kafka server both are using the same port the configuration is same right so you need to stop them first what I'll do I'll stop everything Ctrl C I stop my producer I stop my consumer which I ran in open source Apache Kafka then I need to stop this Kafka server then I need to stop my zookeeper where it is this guy yeah just stop it fine so I'll clear everything anyway I need to close this particular terminal because we we want to play with the conference Kafka now so what you can do go inside this folder just open a new terminal art folder everything is same each and every command is same only you need to give the different path of your config okay so what I mean I will show you will follow the same first we'll start the Zookeeper then we'll start the Kafka server then we'll create the topic by defining the partition then we'll see the messages so for that what I can do I will just go to the folder and I will open a terminal new terminal let me open couple of terminal okay fine I have created four terminal now first let me start the bootstrap server sorry zookeeper right so to start the Zookeeper I will run the same command see here this is what we have ran in let me Zoom this for you the same command we have ran for open source Apache Kappa only thing you need to Define instead of this config you need to give the actual path of your zookeeper.properties file so if you check here the zookeeper.properties file is inside Etc and then the property file right so you need to give that path instead of config just change it to the it is inside Etc hyphen Kafka and zookeeper.properties just run it okay so I am already there insert bin no right okay so what I'll do I'll clear it let me copy the command okay I have already documented the command so let me run this particular command I mean see here the command is same I'll Zoom this for you only we are changing the path of your config properties okay product properties so let's see what is the Mystic we are doing great I've ran the same command there is no change okay now what is the next step we need to start the Kafka server so to start the Kafka server the command is same here see bin hyphen sorry Bean slash Kafka even server start only we are changing the path of server dot properties just copy this open a new terminal just run it cool it started the Kafka server now we need to create a topic so what I will do I will copy the same command I mean the same command you can use which we use in open source Apache Kappa to create the topic only the Zookeeper and Kafka server changes to this particular config path rest all are same because on top of bootstrap or on top of your broker you are creating the topic so what I'll do let's copy the same command anyway it is same right just copy this go to the terminal any terminal just run it we are creating the topic with name new topic one okay all good we are able to create the topic now if I list down all the topic just see the command see already I explained in open source purpose of each word right so I'm even I'm using the same again for the conference so let's run it I just want to describe the topic I mean I just want to list out all the topics available in this 9092 can you see here new topic one we're able to see this Java ticket topic and Java ticket topic one as well because they both are using the same 9092 and host I mean earlier where it started my open source Apache Kafka and this conference they are using the same configuration okay so that is the reason you can able to see but don't focus on that can you see the topic which we just created new topic one cool now if you want you can describe your new topic again the command is same let me copy this and go to the terminal run it okay I did a mistake Ctrl C I just want to describe it can you see here there is a one replication factor and we have defined the three partition we can able to see that okay all cool now let's start a producer message I mean producer application and consumer application then just publish a message to visualize whether we are able to push the messages in confront Kafka or not okay so I'll just copy this command to start a producer go to the terminal I mean I can use the same right I can use this control sorry command k now meanwhile I need to start a consumer I don't want to read it from the beginning I want to read it because we have not sent anything right now I will go to the new terminal this one this will make consumer now this is my producer cool just run it both are pointing to the same topic he will publish to the topic one I mean this is the producer will publish to the topic one consumer will read from the topic one I'll send something will come to Java techy or I'll open my Postman I'll take any payload I'll send in the form of Json okay I mean Json string you can send any type of data okay so let's open it I'll take something let me take this okay I am sending some Json string fine some random messages can you see here now if we verify we have sent three messages to this particular topic right if you go and check the topic is not listed here I need to refresh it refresh the topic can you see here we can able to see our new topic and we have three partition zero one two zero one and two okay go to this new topic one run it can you see here all the messages I mean whatever the messages we send in the Json string each line is published as a individual messages so this is not the way we can send the Json to just show you that we can able to communicate producer and consumer I am randomly copy paste the Json string and I am sending it okay so now what I'll do total number of messages is 9 the offset count is reached up to eight okay and all messages went to the partition 0. 1 and 2 is empty right so what I'll do I will publish I mean we can give a try to publish a bulk messages to see whether in confluent Kafka the load is being distributed to multiple partition or not okay so I'll just copy these I need a new okay I can use the same producer topic one and then I need the location of my download right because that is where my file is present so CD come out this is right close it you just dot CSP just enter it can you see here it sent all the rows now we'll verify whether thousand row went to the topic one in confluent Let me refresh this topic now if we'll run it you can see so many messages now did you observe one thing some messages went to the partition one some messages went to the partition two some messages went to the partition zero now if you see in the partition 0 total nine messages received to the partition 0 1 received 925 now rest 75 will be there in the partition 2. all good 76 okay the messages will or the events will be distributed to the number of partitions okay again we can't guarantee who will get the more messages or who will get the less messages that will be again managed by the coordinator okay and this behavior is also same in the open source Apache only few of the utility will find in the confluent Kappa okay now in my all the upcoming videos I will use the open source Apache Kappa but if you are familiar now about the confluent Kafka of Community Edition I mean I already show you right how we can play with each component in open source Apache cupcan conference now it is up to you when we are doing any program using the open source Apache Kappa if you want you can give a try with this confront cap casual okay now in my next tutorial I will show you whether do we need really zookeepers to start a Kafka or without zookeeper also we can play with the Kafka okay do let me know in a comment section if you guys have any doubts

Channel: Java Techie

Views: 45,105

Keywords: Apache Kafka, kafka for beginners, javatechie kafka, javatechie, kafka, kafka CLI

Id: jhPTpurb5sA

Length: 42min 15sec (2535 seconds)

Published: Sun Apr 09 2023