How to install Kafka using Docker & Docker Compose in any OS | Windows | MacOS | Linux | JavaTechie

hi everyone welcome to Java techy in this tutorial I'll walk you through the steps to install Kafka using Docker okay all right you can install kapka on any operating system like Windows Mac or Linux but the installation process is somewhat different for every operating system so instead you will use Docker I think it's an even better option since you don't have to install the tools manually instead you just need to write one simple Docker compose file which will take care of everything and the best part is it will work irrespective of any operating system so it does not matter what operating system you are using everything will still work or in other word you can say this will act as a platform independent okay and I would say this will be a quick way to get started with Kafka so without any further delay let's move into the installation process so let's get started foreign [Music] therefore we are going to create a Docker ipin compose dot to ml file and we are going to Define two different service one is Zookeeper service and other one is the Kafka service and will ask Docker to create these two instance for us okay so let's have a look how we can Define the docker hypn compost.aml file and how we can Define the services inside that particular yml file okay let's go to the intelligent idea or you can create in the any editor I'll go to the IntelliJ IDEA I have just created one simple folder Kafka hyphen installation I'll create a file called new click on file and I will give the file name as docker hyphen compose dot yml okay you can keep the extension either yml or Y A ml it is up to you okay I will just define y m l just click on enter now in this Docker compose file you can Define what all services you need as part of our scenario we need two Services zookeeper and Kafka okay and to start any Kafka server first you need to start the Zookeeper that is what we will learn before right we have tried multiple example using the command line interface so you need to first start the Zookeeper then we need to start the cup correct so I need to define those two services so first I will Define the version which is 3.1 then I will Define the services you can Define n number of services here okay so that the docker will create the instance for these Services whatever you will Define in this compose file in our case we need the services called zookeeper okay now how can I get these Services simply you need to specify the image name of Zookeeper but I don't know what is the image name simply go to the Chrome browser and you can type docker zookeeper you can find there will be so many Docker images from the docker Hub but I am using the stable one this one okay if we will Zoom this this is the image name copy this name just paste it Define the image I want this image as a zookeeper so that Docker will start this zookeeper instances by pulling the image from this okay now once you have defined the images you can Define the name of the container so I'll give the same name as a zookeeper okay now once you have defined the image name and container name next you can Define the ports where this zookeeper will run host port and your machine port so I will Define two one eight one this is the default one so I'll specify the same fine we have defined one Services similarly we need to Define another services that is Kafka so let me copy this and I will simply paste it second service name is Kafka and images name you can search the image just simply change the name of your service what you want to use this particular tag having this zookeeper and Kafka so if you want I can show you that in this repository if you will filter see here we have the Kafka okay I'm using the same now just change the container name as a Kappa and the default Port of the Kafka 9092 9092 once you have defined the service zookeeper and Kafka then you just need to define the environment by telling to the Kafka where is your zookeeper up and running okay so simply just Define the environments or environment attribute just Define here advertised hostname and the host is localhost okay now you just need to Define Kappa zookeeper connect where is your zookeeper upon running you just need to Define that okay so let me quickly Define it now just Define zookeeper it is on Port 2181 okay all looks good next once we have created this Docker compose file we can execute it using the docker compose command Okay Docker compose up but before that I just want to show you the feature in ID if you want to run The Individual Services go to that particular service name and here you can see the option called Docker compose of Zookeeper okay similarly if you want to start only Kafka just go to this service I mean this icon here you will find the option to Docker compose of Kafka if you want to run all the services what you have defined in your file just run on the I mean just click on this particular icon okay it will execute all the services defined inside this compose file okay but I just want to run it explicitly in the command prompt so that you will be aware about the command so what I'll do I'll go to this particular directory first CD Kafka installation okay now if I'll check the list I have this Docker compose ml file so what I want to do here I'll simply run the command docker compose then give the file name which is Docker compose yml and of that particular compose file in the background so that is the reason I have defined Dash D okay so the main command you just need to run Docker compose give the file name and define up okay just run it can you see here it's pulling the Kafka pulling the zookeeper it will take few second to download those two images okay if You observe here container zookeeper started and container Kafka started also Network up consolation default created it created the default Network and it started these two container so we didn't run this from our local binary distribution rather we asked to the docker to create these two instance for us okay now to verify whether these two container is running or not what I can do first let me check the images can you see here it created the image called Kafka and you'll also find the image called zookeeper okay now we'll verify whether these two instance is running or not so simply you can run Docker PS can you see here let me Zoom this so container ID is this and the name of the container we have defined zookeeper and Kafka okay these two instances up and running now what we'll do let's move ahead one step into the docker container or Kafka container and we'll execute the command to see whether the Kafka is up or not okay so for that what I can do you need to fire the command let me clear it docker execute integrated terminal and name of your container ID in our case the container ID is Kafka right that is what we have defined our container name next just Define slash bin slash sh before that just add a dash here okay now if you'll enter it now if you'll do the ls you can find these many folder okay just go inside CD opt now if you will do the ls you can find your Kafka binary distribution installed by the docker itself okay now go inside this particular folder I'll copy this now we are inside the if we'll check the directory where insert slash opt okay now if we'll do the ls you will find license notice being everything now if you go inside the bin you will find all the script file cell script file or batch file for Windows can you see here all the cell script file to start the cup card to create the topic to produce the messages to consume the messages everything you will find now what we'll do we'll quickly end of this we'll just create a topic and we'll publish a message and we'll just consume it or we'll just publish a message and we'll verify that in upset Kafka Explorer okay so to create a topic you know the command so let me enter it simply just run this Kafka topic.sh we want to create the topic and give the host and the service name of Zookeeper replication factor and we need partition one topic name is Quick Start okay let me Zoom this quick start now just enter it it created the topic called quick start so we'll verify whether this topic is created or not so what I can do I will open the Explorer then just click on this I mean you just need to give the see here if you will see the connection property I am giving where is my zookeeper up and running now if you open the topic we can see the topic got created name quick start so there will be no data since you didn't publish anything to this topic so to publish to the topic what I can do I'll go to the terminal then I will simply run the producer command Kafka console producer.sh to which topic you want to produce the messages and what is the host and Port where your bootstrap server or Kafka server is up and running right this is the basic command we already tried this many times in our Kafka Series so I will just enter it then I will send few message hi hello Java Tiki then install Kappa using Docker and docker compose okay or anything some number some random messages now how can we verify whether the messages were type produce is there in the Kafka server or not or it is there in the topic or not simply go to the upset Explorer now just go to the data I want to see all the latest data just execute this can you see here all the messages what we have just produces is there in the topic now what I want to verify that is fine producer is able to publish the messages to the topic now I want to verify whether consumer is able to consume these messages or not okay so that is again Quick Step what I'll do I'll just open a new terminal then I will just um it's fine I can run from this command also so I'll just execute the command this then I'll go to the directory CD opt ions I want to go to this particular directory CD go inside the bin LS that's it right now we just need to start a consumer app so that it will read whatever the messages we have published to this particular topic so simply go to the terminal what I'll do I'll open the background here okay now let me clear everything I'll just run this command read the messages from this topic from the beginning read it and where is my Kafka server okay just enter it can you see here it take few seconds and we are able to consume all the messages okay so in real time you can follow this approach where you can install the Kafka zookeeper MySQL ready so whatever the service is you don't want to install on your machine and you want to play with them or you are hosted to the server where you don't want to install them physically but rather you are you want to get it from the docker so you can go with this particular approach all the services you can install through this particular Docker compose approach okay and this is really needed if your application hosted to the cloud infrastructure and if you are not using the managed service given by the cloud provider then you must need to play with this container platform using this Docker compose so do let me know in a comment section if you guys have any doubts that's all about this particular video guys thanks for watching this video meet you soon with A New Concept

Channel: Java Techie

Views: 41,746

Keywords: install kafka using docker, kafka using docker, javatechie, kafka, kafka installation, kafka docker, docker kafka

Id: EiMi11slVnY

Length: 16min 10sec (970 seconds)

Published: Sat May 06 2023