Kafka + Microservices + Event Driven Architecture with NodeJS

hello and welcome back to another exciting session i'm your host karya you're watching me on youtube now and today i'm gonna take you through apache kafka so we are gonna create the micro services and we are gonna communicate between the containers using apache kafka and we are gonna send the message from here and there so without wasting any time let's deep dive into it so i'm gonna start with the normal vs code everything that we write from scratch every docker compose file everything okay so for that you need two things first of all search for docker and quickly install docker from here after that click here on get started and you will be greeted by this page where you can download this from here okay so i already have it so i'm not gonna download this thing after that you need docker compose file so we can run multiple micro services okay so search for docker compose and hit enter click here and install uh docker compose from here for your system you can choose anything by the way after that this is it okay uh you need to install few extensions so i'm gonna use docker so you can search docker from here okay so i already have it it it's actually very good extension okay after that you need a mongodb so what we are going to do is we are going to create two things okay we are going to create two containers in one container we will use a mongodb in another container we will use a postgres and with using kafka what we'll do is we'll call the for request um we'll use express of course so we'll call the request from one container okay so we'll call any api let's say in a first container and it will hit into the another container okay so let's say i'm gonna create a user so user will be created in both the containers in post race container and in container okay so let's see how it goes so without wasting any time let's deep dive into it so you have to install this thing as well uh mongodb extension so you don't need to use atlas you need to install this thing as well which is for a database connection okay so you can search here for mongodb okay and here you need to install after that search for database client so you can connect all the sql databases from here um okay database client we also need this thunder client and the client instead of postman i'll be using this thunder client and optionally you can use this uh kafka okay so from here you can um you can check all the things so what's going on all the topics inside the kafka so this is it so without wasting any time we can do one thing uh create new folder name it order it maybe i'm gonna name it kafka okay so kafka quickly or drag your folder here and uh let's start with the your terminal and we we're gonna in it two folders quickly so mkdir now let's say f1 and app2 will be better okay uh app2 okay here we'll start with the uh docker compose file docker compose dot yaml okay here uh all the code that will write here first of all we'll start with the making kafka thing okay so here you need to define a version version is 3.8 or currently i'm using uh services after that services you need to define zookeeper um zookeeper okay then you need to define container name our container name will be again zookeeper you can grab the same here as well after that you need to define a image first of all so i forgot to define image so image which is uh started with w this one okay uh after that you need to define our ports so search for ports okay and default port is two one eight one okay so i'll be using two one eight one i don't need this thing i'll use the minimal uh approach okay after that you need to define uh this is it i guess we can proceed with the kafka so kafka and after that inside kafka we need to define container name of course container name will be again kafka and after that we need to define image uh image will be again same uh after that default port is 9092 okay then we can define the volumes to process the data okay so so every time if you make any changes it will process the data somewhere so we can uh define uh this will be my directory okay so i'll say data i'll create a data directory in my current folder and i'll name it kafka like this okay and this is my now where thing actually so let me write slashware okay slash run this is something that i want okay so this is it okay this this thing you will get from official documentation don't worry about this thing okay and apart from that last thing we need environment okay so in that environment you need um a few things so first of all let's say kafka underscore host okay uh this will be a cough cost and you need zookeeper contact so where you can write kafka oh sorry kafka underscore or zookeeper okay so this is the container name okay so this is kafka this is the name of it should be like it should match to this one and it should match to this one so this is how we are connected with the both the things after that we need okay because i told you guys we are gonna use and now is very easy okay of course defining container okay here the ports 27017 this is it uh we don't need to define volume so you can define the volumes as well our data it will create a data folder of course in my current directory inside that and this data uh that will get from inside container okay environment variable you can define of course but i'll stick to the normal minimal approach for now and now let's say i have a post list okay so for that post is now i'll be using container name posters image post sport default port volume data and this uh something that you will get from official documentation because inside that postcase container you will you can process the data from here to here okay so you can define the password we need to define the password so we can define all the things here or you can define the password only so after that we can now proceed with the further things okay i don't have f1 and f2 so let's quickly create a uh or terminal open the terminal from here okay and uh quickly start with f1 here okay so what we'll do is uh npm init hyphen y okay and pm install express quickly over there uh and i'm assuming that f1 will create on postgres okay so npm install sqlize if you are not familiar sqlize is orm that we use uh after that we need a driver so pg and pga store uh you can get from uh you can refer the official docs if you are not familiar with this you can go here search for sqlize and hit enter the very first link you will get and you can see you can refer the documents from here as well okay uh which is available from uh if you click here on getting started you can see for my sequel sqlite all the other things you can install okay this is not limited to the post list okay after that what you have to do is uh you need to install uh we quickly check the package.json because we need the kafka node as well here to connect with the kafka so kafka quickly have a node and hit enter uh this is it okay now we'll uh quickly go to the app to okay cd into app2 here what i'll do is again npm in it hyphen y ah npm install express but uh this time i'll be using uh mongoose and kafka node of course so kafka i have a node oh this is it so yeah this is done okay now what i'll do is i need to create a docker file quickly over here uh so docker file okay so docker file will be very easy you need to select the node version over here so my current node version let's say or like this but i need to select the alpine so alpine is available for uh this version okay so now you need to define a work directory where what you can do is you can define the name like f1 app to water maybe let's create f1 so after that let's stick to app first so you can do the copy like this or first of all you can uh get the packet.json and index.json uh js sorry how to slash app okay slash app slash like this after that you can run the command npm install okay inside that container and after that you can run one more command like npm install now let's say hyphen g nodemon to run this thing in uh kind of div mode okay so you don't need to restart your container every time and you can copy this thing to here as well so both the things will be most of things is similar over there and here and after that we can link these two things over here okay so how you do that will quickly define f1 over here okay and here what we have to do is we nee you need to provide a container name of course after that we need to define a port so here instead of this image what we'll do is we'll provide a build okay so build what the build will do is we need to define path over here so this is a part of my directory okay after that this is the port 8080 i'm assuming i'll pass the port 80 tt over here our data f1 and uh it should be app so we are creating f over there okay so that's why it should be like that so it should not be data it's my current f1 directory i've created so this is my volumes okay after that we need to define environment variables so let's remove everything let's create with normal port okay support is say dtt of course uh that make sure to match this thing over here okay so url but the first one is depends on postgres uh url okay so postgres url so this is the postgres colon slash slides this is my username this is my password this is my uh name of my service if you guys remember this the name of service this username password and this database name which is default database name so this is it now we need a kafka okay so in in kafka we can say kafka bootstrap url okay like this uh servers okay so this is the name of my service uh which is it should be this one okay let me close this thing so it will be visible to you properly guys now so yeah now you can see uh now you can define the kafka topic as well if you want guys so to save the time i'll use topic one okay this is it these four things that we need to define okay and this depends on something so here you can define the depends on as well this depends on the first one is the posters of course uh suppose and another container is kafka so we need to define this thing here as well okay so this is it i guess no need to define anything anything else and uh if you go back here and if you define app2 here okay so quickly you can see uh i'll get the suggestion which is almost um it will be equal to this thing up to this it will be eight one okay you can't run both the things on the same port otherwise you need to install the nginx whatever reverse proxy it depends okay what you use uh this is topic to know i i'll use topic one uh same i'll pass something from there and i'll get something here okay so i don't need posters url here i need url here so what i'll do is i'll hit underscore you issue url okay uh so you can see here you can define app too okay uh database you can define anything if you have defined username password you can define username column password here make sure to not use that symbol in password okay otherwise we do generally this kind of mistakes okay this is it uh we have created this now we can proceed with the f1 where we are gonna create index.js quickly and we are gonna write the code for that so a normal express f we need to create const express equal to require express require express const app it goes to express quickly and app.get this will be posted pass something from here and uh we'll do something there so request response of course but before that if we are getting something over here we need to define our app.use here to allow body okay so json we need to allow here and app.listen will listen to the port that we are providing from there if you guys remember process dot env dot port okay this port uh we are providing in docker compose file if you can see here in in this port that we are passing over here this one so that we are getting here okay so this is it this is the boiler plate that uh we uh we can get start with okay and after that we need to define the kafka node over here so const kafka is equal to require quickly and here you can define the kafka node okay of course this is it and after we can proceed with the actual code okay so what we have to do is we have to wait for the kafka to get started okay so how how you will know that how much time it will take so you can write the shell script or whatever it may be but what i'll do is i'll i'll write the set time of method okay so i'll i'll say and um for now i'll now say it will take approx that much of time 30 seconds 60 seconds like that i'm assuming so now let's create a client first okay so this is my client quickly okay and after that we need to do one thing we need to create now producer okay so this is my producer okay so once you create a producer after that you need to you can use producer.com on method producer dot on and where what you can do is you can say ready on ready state you can do something over here okay we will pass the message and we can do anything okay it's not working actually i hope it should work okay cool it's working so producer not ready we can define over here after that we can proceed and let me turn off my hoop pilot actually disable globally disable for javascript disabled globally we can do the app.post here actually app.post here so i don't want to do this thing over here okay so let me um cut this thing from here and let me paste this thing over here okay like this so then it should work whatever the producer you want to create so we'll say producer don't send quickly okay so producer dot send and this is um something that we are sending over here or you can define the topic name okay if something goes wrong you can do this okay and uh topic name we know guys so a topic name that we are providing from the uh process dot e b okay so you can remove this thing okay process dot env or dot kafka let me use kafka i'm not sure i need to check over there once kafka topic like this we also have to do one more thing uh here we have to define our process dot env dot uh [Music] of this thing that we have defined over here oh kafka um bootstrap [Music] so so yeah like that so this is it uh we got the client we got this thing we need to string a file message because uh it won't work okay it's only allows strings to parts only okay after that you can do whatever you want to do over here so let's do one more thing okay as i told you guys this is uh based on the sqlize okay so here what we can do is we can use equalize quickly sql is equal to requires equalize okay uh then um we can do cons db is equal to new or new sqlize okay uh this is something that we are getting from our process.env again so process dot env of env dot let's grab this thing from here so if you remember we have used this url so i'll be using over there that is it so we got the db after that what do we gonna do is we gonna create a new user okay so this news that we need to define first okay so dv dot defined and inside that you can pass your user after that you can pass your attributes like name email password name email after that you can have your password after that you can define your user okay so this thing that we are gonna do inside this thing once we now [Music] gito once we uh consider that everything is running fine so let's say running fine okay or kafka running fine more specifically our dbs let's say dbs are running so then only we can proceed with this thing okay uh so this is my function overall function okay so it will only start once we set the timeout so set timeout is the time required to start the service for the kafka let's say so it will around take 10 seconds okay so this ms so this is how we have created the user now here what we have to do is once we send something if we get error then it's fine but we don't get error else so in else block what we are gonna do is uh we are gonna use user dot create and request body so this is what we want to create okay and uh in here once we create something we can pass rds dot sign because dot body is created or something like that so this is it is what we have to do and uh let's go quickly over there as well in mongodb we have to do the same things okay so index.js and here most of the code will remain same like this instead of sqlite we are gonna use here mongoose okay so let's use the mongoose here and this one is also mongoose and let's grab the cool name of this thing will remain the same let's grab the complete file actually to save the time and let's remove the unwanted code okay the unwanted code means so this one so we'll create everything from scratch here once again so we'll say uh mongoose.connect here or double anyct connect okay here we have to define the url okay this is we are getting from there okay once we define this thing after that we can define uh the user as well so const user let's say so which will be new now start schema okay so mongoose dot model actually start model where you define you need to define the user where you need to define um the further things like name of course so our name i'm not sure why i'm not getting this name okay email password okay cool so now we have defined this thing now we'll go there here [Music] now we need to create the kafka again okay so here what we can write is [Music] we don't need to write a client actually we need to write yeah client is needed but uh this time we need to create a consumer okay consumer so this is my consumer where we can define the topic name okay oh auto comment which is false which is uh depends like you need to create a batch actually okay so let's say i'm sending a message a one two three four and you're gonna get all the previous message then you can write out a comment false otherwise it will automatically commit the message and you will get the current message only so that's how you can now define the max limit as well over here if you want like this okay if you write the fetch max or something like that let me check if i get the suggestion yeah max limit or something like that i remember anyways we'll see in the docs so this is how i have created my consumer okay so but while creating a consumer you also have to do one thing now after this or you can say before this here you have defined the topic uh you can define the uh partition over here if you want so you can have your yeah multiple topics as well and uh after this array okay once we define this array over here you can define your partition as well not here actually here only partition yeah here like this so far now it looks nice to me okay our changes are not needed now we can quickly uh go to the consumer dot on message okay so on message whatever the message will get that after that we will create the user okay so it depends on the message actually so we are creating user here as well and user there as well so this is the mongoose part so it will be user dot save okay so this is my user which will be my new user okay so this is the message dot value that we are getting from there okay so it means we have to create a new user so we'll save this thing first to use a variable and this user variable that i'm gonna now save it okay so save because so this will be of it and uh you have to take care about a sink everywhere like here as well we need to define a sink and make sure all the callbacks properly that you have used okay in here as well we have to define a sink over here so we can use in the parent should be async if you're using somewhere inside here okay so i'm by default i'm assuming we'll use of it somewhere so this is it i guess so i haven't sent anything over here we need to send message okay and send the message as well so yeah here we have defined the message this is my request dot body is my message okay cool so this is how we have uh created our user okay now i can close this thing or i can hit enter yeah like this or we have missed one thing i guess what if we get the error then what you will do so on error you have to do this okay you can console log the error if you want and this is it okay so again i want alt shift f uh after that you can use app.gate here somewhere if you want otherwise we can uh proceed with this thing only okay and this is my mongoose thing once again and this is my sequel eyes so see in sequels we can do one more thing so here i have defined this thing and now i can do one more thing here before this we can sync this thing so it will automatically uh create a something okay in your database otherwise the table won't be created and you can't upload anything over there so main purpose of this tutorial is to show you that we are sending a message from one container we are uh storing the user of from sending a message from here to here and storing this user to another container and on the request of on the success request if we get then only we are storing to our post hdb as well so this is how we are doing something in both the containers is how we communicate between the containers with the help of kafka actually you can create the topic as well you can delete the topic or lots of things we can do here as well but uh let's uh open the docker first so if you haven't installed docker already so what you can do is you can grab the docker from here local compass from here okay and after that i need to start the docker quickly okay this is docker it will start in a while not till then or what we can do is uh we can open the terminal quickly and check with the let me close this things check with the docker compose iphone iphone version i think okay i hope this is right so yeah docker compose and docker have an iphone version if you you have installed this thing then definitely uh you can like able to do this thing okay so you can see this little icon over here everything is running fine so we can proceed with the further things now so for the things means just docker compose up okay so locker compose and up i hope i i'm sure i'm missing something okay we'll get the errors no worries it's okay to get errors in uh while creating microservices this very real-time video guys okay it will take some time still then uh okay we got the average error quickly so let's go to the user.create method is in ah here uh in your postgres of course have one so where we are creating user1 so here quickly we need to define async so this is already async so of it is only in an async function the top level okay cool let's uh go one by one okay this is my sing async okay this is also async and this is not async okay so let's have saved this thing okay again we'll get errors now what is silly errors precisely so after 10 seconds okay i will call [Music] here if i go here you can see all the things are running fine okay now i don't need to okay so hopefully you can see select something is working over there so let me connect with my extensions okay so mongodb is running fine so i'll go here i'll search for mongodb colon slash localhost colon270177 slash app 2ys ah so let's go there let's see that in app2 [Music] hopefully this is working you can go here okay this is my mysql db extension you can see uh this is very nice looking gui inside post sql you can define your password postgres okay control minus command minus connect here and yeah this is connected so if i go here you can see in post as public tv will create users the user is created automatically because we have done the same thing over there okay so this is running fine now we will go to the thunder client inside that thunder client will call something from here and definitely will get the error but let's see so name email and password i'm sending from here in body port att if you guys remember let me show you uh this one 880 is the port of f1 so we will pass something from f1 here and once i send something over there i got the response back it means it should be created over here okay and over here as well which is hopefully everything is all right so connected no it's not getting created over here let me refresh this thing so because if everything is alright then we'll get the app to over here thing so let me quickly check with the name of db my db have to okay and let me check with the connection if possible but we can directly go here in app2 and we can view the logs from here and we can see the error over here we got our error wow which is good we got the error at least and this is can't find the model mongoose okay quickly we'll go there inside app2 and we'll check for the mongoose so am i gwsc mongoose so we might done the spelling mistake mon soha typo so as we know we have to make change here okay so it will okay we need to wait for 10 seconds i forgot this one now we can uh we can do this thing we can make changes because it started already so now we can do one thing again we'll go here we'll call something else over here let me see the ipk name of my father d-i-p-a key okay send something over there and hopefully this time it's created and otherwise you'll get the error not okay good so we got to use this from here as well so this is how so we got both the things because it was in batch something like that all right okay cool so thank you so much all of you guys for being with me supporting me all the time it's been pleasure to be here if you want something from my side please let me know in comment box i'll make more such videos don't forget to subscribe my channel and please smash that like button if you enjoyed already so thank you so much guys and have a great day bye

Channel: Mahesh Kariya

Views: 35,294

Keywords:

Id: wdsiQEa5mQY

Length: 34min 0sec (2040 seconds)

Published: Wed Feb 23 2022