Apache Kafka installation on ubuntu

what's up guys in this video i'm going to show you how we will install apache kafka on aws so let's get started [Music] now let's launch and configure our ec2 instance linkedin and uh this is my post on this kapka installation server installation and all the steps which i am gonna use in my practical is mentioned right here so now let's go to our first step launching and configuring easy to instance one so here what i have done is what we have to do is initialize our first one ec2 instance of ubuntu so let's go into our ec2 so this is my ews console and i'll get into ec2 right here so i'll get into ec2 so this is my ec2 and i will hit this instances tab find so instances i will from right hand corner there is launch instances so i will hit it i will search for ubuntu this one and hit enter so after this this is the 2004 version which i have to use so i will select it okay now this has been selected now since as per the post i want to medium because this is the one so which is good enough because it has i3 and memory is that is means ram is 4gb so i3 and 4gb is quite enough for our one server you can also use a t2 large one because it's like a lot memory right so this one t2 large which has is i3 and 8gb of ram so let's configure so let them default vpc and all because we don't need to configure it now this is the storage 8gb is enough for for this task one task only but in your development in your production or development environment in this enterprise version you need some more gb approximately take up to 16 to 18 gb because since you are clustering and preparing a cluster there so you need um some more space for topics and the configuration files and the logs which are going to generated by those topic so that's why while messaging you need them so okay so i will use add tab tags and this name name here i will use this uh kafka server okay and now configure the security group so security group let's leave for now and in the future uh we will surely configure it but for now i will leave for this step so we will launch and let's launch our instance so here i will use this uh nippy one so okay so what i will do is i will generate one keep here because it may create some problem so let's generate a kafka server keyboard right so i will download it okay so in the downloads let's save it fine kafka server pam is getting saved and i will launch it so now it is launched and this one is right here okay so let's register an eli elastic ip for this instance for now so this is starting i think so let's go to dashboard this one it's quite running right now this is our instance this one kafka server so since we have okay so it's starting now it's running since in its initializing when it is done it will show us uh tick mark and two health checks so let's see let's just start with so by then uh let's register for elastic ip so right from the left hand pin scroll down and inside the networking and security section you can see this elastic ip right so this is the elastic ip and in this plastic ip hit this allocate elastic ip and for elastic ip usb 1 right and just allocate one so this will provide you one ipv4 this that is this your ip address for and this one is allocated so i will what i will do is associate it with my instance and choose instance right here doctor server like address this one and allocate this ipad just to be reassociated and associate so this one is associated 107 22 64 145 now this one is getting shown in our instance that's the public icon let's refresh it yeah so this one is being shown as a public ip in our instance see this fine so what i have to do is get inside my post and since all these steps are configured the last step is the ssh so for ssh let's use first the browser-based tool let's try to connect browser with browser first so yeah so browser one is working fine so let's use this one so this one but for your the sake i i will do one bit wise also otherwise this is right this is my device so for bit wise if you are using bitwise that is sh client you the alternative to this is moba x stream and putty so putty is more popular but this one is also good one because i am using it so and now import that key that key which we have downloaded so search for all files and now this is that file in the download so open it and now import it import okay so we have imported so i will close it select global to ubuntu is fine we have to change just our ipf yes so make sure you have the same settings which i am using later now hit the login so now accept and save so this one is quite running and authentication is completed now open the new terminal okay so this is how you will ssh using our bitwise or ssh claim now let's install java in ec2 instance now the second step we will move so this is installing java in easy to instance so now we have to configure java so these are the command which i have mentioned so in my linkedin post mentioned in the description section of this video so copy this first command in the snippet that is sudo apt-get update so it will update our ubuntu okay so it has starting updating it okay now let's minimize it and hit the second command to install our jdk jdk8 okay so it's extracting by the time it is extracting let's copy this version command to check the version of java it's unpacking punjabi yeah java version oops there is a shoe this one so there is an issue in my command so java iphone version it's the right command so i will edit it and configure it properly so java iphone version is the right command so sorry for it so this is how now let's download apache kafka we have configured our java version now in the third step let's get into my article so here we have to download it so in the first snapbit there is wget command so it will like a download so it's it will take this from this url from this url what it will do is take this and copy it and paste it right there so it is extracting here okay so i will use the star command and paste it what it will do is so this is the whole thing and since this tar we have extracted it using this command so what i will do is change this name of the this whole file to kafka short name so i will use this symbolic link so this one will paste it right there and this this one is changed to this symbolic link i have done so this one now let's configure apache kafka now let's make a directory inside our root so use this first command to make the directory so open a new terminal this one is so minus so this is quite here so we are in our root so i will fire this command now our this zookeeper is created directory is created so now get inside our kafka config which is inside our kafka so i now we are inside our kafka so what we there are many properties file so there is update properties so i have to we undo it via is the editor like notepad in linux so this one the press and just see the content right it's point same so what the changes since our we have created the folders inside so i will use this root and skipfold and wg right now what i have to do is delete this file that is server.properties so let's see this server.properties i have to delete so i use rm minus rf command to create it now also our dot property has been deleted so to create the new file use vi edit or copy this command and then paste it right this will create file server.properties for us now it into insert mode so this is one now what we have to do is just copy all the content from this file this one is the content create right here okay so we have pasted all the contents which are there right let's go up off okay so the only change we have to make it is this one so according to this we have to replace it with our elastic ip address this your dns dns so get into your aws now since this one is our elastic ip i will copy it from here and now get back to my this one and i will create it replace this too all right dns okay since so all the things are done now what i will do is escape hold on w cube to write into it my file now this step is also done so we have completed our configuring now let's start apache kafka now we have to start our kafka server so let's get into our terminal make sure you are on root so to check pwd i will use so i'm in the root directory now let's get back to our engine post so copy the first command for starting the zookeeper so what i'm doing is hitting this inside pin so keep a server as such this is the file inside and telling the domain that i have to pass this file also which i have created zookoop.properties i will use this now this is hit now let's check echo that are you okay so this is the command to check whether our instance is okay the creeper is okay or not okay so rd is not it's not a white list okay so it does not working right now let's see how this locks work [Music] commandless serving okay so this is working now let's start the brokers so this one is copied and paste it here so the dot properties we have passed and now let's see it okay this command doesn't work so this log one error through keeper is not okay so down let's first open our quotes security group i will open our ports do one hit one with this one and one so zero nine two okay now save tools send and run and let's see start our bootstrap server let's see if it works forever so got a new controller comment and started our server has been started so okay now to test it so the main problem was since we haven't configured our security group so that was the problem so 22 port 2181.9092 port should be open after this so i will mention it now let's test apache kafka now for testing let's copy the first command so for this command there is kafka so we have to copy this and let's prepare it first in my notepad copy it here so instead of your this address i what i have to do is from my easy pc to console i will get into dashboard in running instances scroll down okay this is kaka server and inside the details there is our ip address prototype address so what i will do is copy it and now in this from here i will paste right there okay first command is ready now copy it and inside your terminal let's clear the window controller and now paste it okay so we it has created our topic with the name demo topic right so since we have specified the topic hyphen hyphen topic demo topic now let's see how we will use our producer and now what we have to do is start two things like two windows one for producer and one for consumer right so let's close it right from here okay and what i will do is i will open two terminals so i will go to bitwise and click on new terminal one terminal is right here and another is right there okay so there are two terminals let me minimize it okay so there are two terminals so one terminal i will consider as producer and in the another terminal i will consider as consume so producer consumers are already so the command i will prepare open for this producer and another for consumer so this one i will cut down paste it right there and replace this ip address with my public one which i have copied previously paste it right here and in the consumer also prepare the command okay so this one is right here i will replace this one with this copy right here and install so these these are the commands so i will show you how this producers and all the things about it and now in the one window let's configure the producer so let's see this window let's continue so okay we have to get into boot my bad so now this is our producer and in the another we have to configure consumer so this is our before that let's get into root otherwise it will give us error now this is our consumer can be so i this message is my first demo message so now in the real time i am producing from one with terminal and those terminals are and the messages which i am typing are being consumed in real time by kafka consumer produced by producer and consumed by consumer so this is the thing so thanks for it okay so this one is also done now so let's minimize it and now let's see this one okay so since we have installed and configured it successfully and which we have seen using this one so let me show you some of the details inside the kafka so let's get inside the kafka so this one is about this and inside the seed in the pin inside the pin there are many shell scripts which are mentioned so what i am doing is i am using this one for the topic configuration i have used right here topics right this one for topping for listing also it is used and for consumer i am using this one which is mentioned right here console consumer is right here and producer is right here right these two commands so this one is producer this one is control consumer so this way we have configured it in one video i will show you how we will use and configure kafka service in our linux that is ubuntu instant so stay tuned for now and for more content you can visit our website and our youtube channel make sure you subscribe to our channel and hit the bell icon to never miss the update from us further like share and subscribe for more support visit our website rdtag.in this is our website type simply architect dotted [Music] thanks

Channel: rdtech_in

Views: 366

Keywords: apache kafka, kafka, apache kafka tutorial, apache kafka tutorial for beginners, apache kafka explained, kafka apache, kafka streams, kafka tutorial, introduction to apache kafka, apache, apache kafka basics, kafka connect, kafka producer, kafka consumer, what is kafka, apache kafka intro, apache kafka vs etl, apache kafka vs esb, curso apache kafka, apache kafka nedir, apache kafka course, apache kafka nodejs, what is apache kafka, devops apache kafka, apache kafka example

Id: J3TvpbDaXDw

Length: 27min 44sec (1664 seconds)

Published: Fri Apr 16 2021