Install Apache Kafka on Windows PC | Kafka Installation Step-By-Step Guide #kafka #apachekafka

hello and welcome back to the channel so this lecture is all about installing kafka and running it on your windows pc so without further ado let's get into it so as you're already aware that apache kafka is nothing but a distributed straining platform which is used for processing streams of data all in real time so that will be able to build real-time data pipelines as well as data processing and integration at very large scale so without wasting any time let's start installing kafka and we just need to set up some configuration and we are good to go to ingest some data by using the producer and consumer from the command line so installing kafka can be done in several ways so the first one is we can download the open source apache kafka which we will get from the official website as well as we have the confluent service which will give us the additional capability to the kafka which is provided by confluent so that could be a little tricky because it is more suitable for linux as well as mac systems but for windows you need to install windows subsystem for linux which is nothing but running linux environment in your windows and it will take lots of configuration so to get you started open source kafka will be more than enough for you but we are also going to cover the confluent installation in the upcoming lectures and there are so many other ways that you can directly host your kafka server in cloud and it is provided by many major cloud service providers such as aws azure as well as gcp so this lecture we are only going to focus on the open source apache kafka which we can get from the official website so without wasting any time just go to your favorite browser and search for download kafka so once you go to your favorite browser just go like download kafka and this is the official website which is kafka.org so just click on the first one and here you can see the latest version at the time of the shooting the latest version is 3.2.1 so just click on the dot tgz file to download the tar file so it will just around 100 mb so it will not take much time okay so it seems the kafka has been downloaded so just go to your downloads folder and then you just have to right click and just extract that file so i'll just give extract here yeah so this is extracted file so what we will do is we'll just copy that file and put it in the c drive it will be easier to set up the paths and also you can set up the environment variable very easily with this so i'll just keep it in the c and i also just rename this file only to kafka so i'll just give kafka and if you just go into the file you have all the directories present so in the config you'll find all the configuration of the server as well as the zookeeper and in the bin directory you'll have all the executable files but you have to see that the type of the files is shell script so we cannot run those on windows machine so you have to go to the windows so that you will find the windows batch file so the extension for this is dot bat so these files we can execute directly from the command line but before that we just need to set some log directories for our zookeeper as well as kafka so let's do that first so just come back into kafka and you have to go to the config and here you can see the server and zoo cable properties so the extension for this file is server.properties and the zookeeper.properties so just go to the right click and if you have the notepad or notepad plus plus anything will be fine so here you have to go to the locks and as you can see this is the directory where logs will be residing so here you have to provide the path where we have extracted our kafka folder so he which is nothing but c drive colon slash kafka if everything looks good just save this file and close it and the same you have to do for the zookeeper so for zookeeper just right click and edit with notepad and here also you can see the path for the data directory so i'll just give it as c colon slash kafka slash and instead of zookeeper we'll just give data this will be the data directory for our zookeeper so zookeeper is also required to able to run our kafka server as it will act as a resource manager for our kafka server so just save this file and close it that's it so we have set up the server.properties as well as zookeeper.properties so now we are able to kick off our kafka server so are you ready okay then so just go to the kafka and from here you have to open the command prompt so to do that just click on this and type cmd so this will start the command prompt from the specific path so we are in the c drive in kafka directory and from here we can kick off our kafka server okay so to kick off the kafka server we have the file in the windows directory which is zookeeper dash servers dash start dot bat so that means batch file so to do that just give like dot slash bin slash windows so this is the path and in windows we have file named as zookeeper dash server dash start dot bat and also we need to provide our configuration files which we have just edited so that file is present in the config so dot slash config slash zookeeper dot properties so if everything looks good just hit enter so this will just kick off our zookeeper so as you can see but zoo keeper has been started okay so you don't have to close this window you have to open it so just minimize this window and open the next command prompt to kick off the kafka server okay so we have the zookeeper running so just minimize it and again open the command from from the kafka and here you have to start the kafka server so again the path would be same so dot slash bin slash windows now here the file name is kafka dash server dash start dot bat which is bat file and again the configuration file which is again present in the config directory so again give like dot slash config slash server dot properties so if everything looks good hit enter so this will just kick off our kafka server and as you can see our kafka server is finally running yeah so just minimize it again and now we can able to kick off our producer function from the command line and we will push some data and consume it from the another command line through our consumer function so now you have to go to the bin directory then go to the windows and here you can see the producer and consumer files consumer batch files which we can execute through the command prompt so again from here just open the command prompt which is go cmd enter and now we need to first create the topic on our kafka server so you may ask what really is a topic so topic is nothing but a table in a relational database but in topic the data can be of a flexible schema but it stores the data in the json format so what we'll do is we'll first create a test topic and will push some data into it so to create the topic all you have to do is type like kafka dash topic dot bat so again this is the batch file we will pass the argument as create so give dash dash create then give dash dash bootstrap dash server so here our kafka service uses the local host 909 to port so if you have to communicate with our kafka server you have to use the port as 9092 so just remember that so just give like localhost colon 9092 so to create the topic all you have to type is kafka dash topics dot bat so again this is a batch file then you have to pass the argument so first one is create so as you are creating the topic you dash dash create then give the bootstrap server so give like bootstrap server so for bootstrap server we have to provide the local host and the port name by which we can communicate with our kafka cluster so you have to remember that the kafka is by default using the port 9092 so you have to provide that localhost colon 9092 and the last one is the topic name so give the topic and the topic name would be test so if everything looks good just hit enter and as you can see topic test has been created so the next step would be we need to kick off our producer function to be able to produce some data and on the other hand we will kick off our consumer function from the another command line so that we will able to see the data which we have pushed into our test topic so to kick off our producer function all you have to do is just type like kafka dash console dash producer dot bat now we have to provide the broker list so give like broker dash list and you have to provide the local host here you have to provide the local host colon 9092 again which is the port number for our kafka server and the topic name so our topic name is test so give like dash dash topic test everything looks good just hit enter and that's it we can push any sorts of data in our kafka topic from here so but before that let's kick off the consumer function as well at the same time from the another command line okay so again just kick off another command line so give cmd and here just give similarly for the consumer so just give like kafka dash console dash consumer dot bat then give like dash dash the topic so the topic name is test and then give the bootstrap server bootstrap server again the bootstrap server would be localhost 9092 and will give the argument as from beginning so it will kept pulling from the beginning so if everything looks good hit enter and as you can see now our consumer is listening for any data being pushed into our kafka topic so it's time to push some data into our kafka topic so i have some raw data which we can push from our producer function so as you can see our producer is started now so if you just hit back to notepad i have some json data present so i'll just copy the first row and try to paste it over here and if i enter it and as you can see in the consumer end we got that record so it was in the real time so here in the producer we have pushed the data and in consumer we are able to read that data so if you do it simultaneously i don't think it will be possible but i hope you got an idea so as you can see this is our producer and this is our consumer so if i try to push data here and hit enter as you can see we got the data with less than one second so it is happening all in the real time so again we'll try to push another line so here is our producer we'll paste it hit enter and as you can see we got the data immediately so this is the power of kafka we can able to ingest real times data and also we can process it by using kafka so as you can see it is very powerful and this tutorial was only to get started with kafka so here we have downloaded open source kafka from the official website then we have extracted into a specific directory in c drive and then do some configuration and we were good to go we have just kick off our zoo keeper and kafka server and we were able to produce the data and consume it from the consumer function so i hope you got a clear idea how we can set up and start our kafka server so if you have any difficulties just let me know in the comments and i'll get back to you as soon as possible so this is how you can install kafka on your windows machine and get started and learn kafka so here once you've done everything you have to just stop the zookeeper and kafka by just executing these commands so these are pretty straightforward so i'll just give each and every command in the description below so that it will be easier for your reference so if you like this video please subscribe to our channel and also ring the notification bell to get the latest updates and don't forget to follow us on our social media which i have provided in the description below thanks for watching

Channel: AmpCode

Views: 76,604

Keywords: how to install kafka on windows, apache kafka, kafka, install apache kafka on windows pc, apache kafka tutorial for beginners, how to install apache kafka, install kafka, how to install kafka, how to install kafka on windows 10, install apache kafka, kafka installation, kafka producer, kafka consumer, how to setup kafka cluster, how to create kafka topic, data engineering, what is data engineering, data science, ampcode, big data, big data tutorial, kafka streaming

Id: BwYFuhVhshI

Length: 13min 13sec (793 seconds)

Published: Fri Sep 09 2022