Nest.js Kafka Microservice Tutorial

hey there today i'm going to show you how to create a microservice application in sjs using kafka.js as the transporter so we're going to have an api gateway and then two other an sjs applications that are all connected together through kafka now i have another video on setting up microservices with through tcp uh and so now we're gonna walk through and see the differences using kafka so the first thing we're going to want is the ability to run kafka on our machine and also i like to use the tool called calf drop which is a web gui for kafka allows us to view the topics we've created and see the messages on each topic so you can feel free to run kafka any way you'd like but we're going to use docker compose in this video so that we can run calf drop and kafka at the same time so i have this github repo here and i'll include the link in the description so we can simply copy the clone url here and then in your terminal you can run git clone and then clone the cap drop repository so once you've cloned the repository let's cd into cap drop uh and now we're gonna run cd docker compose kafka calf drop so we're in the catholic half drop folder and then we can simply run docker compose up and of course you'll need to make sure you have docker installed and running on your machine so let's run docker compose up and we should see kafka as well as calf drop are starting up in our terminal here now if you give it a few seconds when we go back to our browser here you should be able to navigate to localhost 9000 and you actually have a web gui now to view all of your kafka topics which will be useful for debugging your applications when you work with kafka so now that we have our kafka server running you're going to want to open up three different terminals or three additional terminals and cd into whichever directory you want to create your application in this is because we're gonna have three separate nest applications all running at the same time we want three different terminals so first we're gonna start off by creating our api gateway so i'm gonna use an scli to generate a new project and we'll call this nest new api gateway so this api gateway will be responsible for actually receiving http requests and then sending or emitting events to our different microservices so i'm going to use yarn to set up this package so we can let this continue in the meantime we can generate the other project so run nest new and this will be a new package we'll call billing so we'll imagine that our api gateway will allow for order creation so you can create a new order and a post route and rdp gateway uh and then in order to bill the user for creating a new order we're going to emit event to the billing micro service which will take care of charging the user so lastly we're going to have one more microservice so we'll call them that's new and we'll call this off so this microservice will host all the information about the users in our application so that for example when billing is trying to charge a user it needs to know which user to charge and get their for example stripe id if we're using stripe to charge the user so it's going to reach out to the users microservice and get that user so it can properly charge them so with the api gateway all set up we can see the intuit and we're going to have to install a couple of packages into each one of these microservices so we can run yarn add of course nest js micro services as well as kafka js so we can actually just copy this line here and make sure we run that we'll do the same thing here we'll cd into billing add nest js micro services in kafka js because this will be our transport and then lastly for auth we'll do the same thing so now that we have all the required packages we need we can go ahead and run yarn start dev in each of these apis so at this point we should have our kafka server running and then we should have three nationals applications all running and we can just disregard the error messages about the address already in use because we won't be listening over http in these apps after we change them so i've gone ahead and opened the api gateway up in vs code i'm also going to open up the additional apps as a workspace folder so you can see here i'm going to add auth and billing as folders to our workspace so that we can have all three projects listed like so so let's go ahead and start with our api gateway this is going to be responsible for actually emitting an event to our billing service so first of all let's go ahead and create a post route that we can use to initiate this flow uh so this will be for example a create order route and it'll take in a create order request so we can add a create order request.dtl and so we'll export we'll export a class create order request and this will take in the user id of the user that created the order and then we'll have a price here that we're going to charge the user so back in the controller we can specify that this is the body and we know this is the create order request of type create order request so we're going to dispatch this action to the app service and we'll have a new function in there called create order and we'll pass in the create order request so let's go into the app service and set this function up so in here we'll have the create order function uh and inside of this create order request we can pull out the user id and the price which is what we'll need and it'll be a type of create order request so in this function the create order uh we could have sort of course persist some information to our database uh but then what we want to do is we want to emit an event to the billing service down here for it to actually handle the charging of the user because uh anything to do with payments or billing is going to be associated with this microservice and it's a separation of our concerns so we really just want to admit an event with the user id and the price here so that the billing service can handle it from there so in order to do this we need to get a handle to the billing service now let's open up our app.module and in here this is where we actually set up our communication our catholic communication to our external microservices so we're going to use the clients module which we import from nsgs microservices and we're going to call register which takes an array of the different services that we want to allow for injection so the first one will give a name and this is going to be the injection token that we can use later on in our service so we'll call this billing service because this is the name of the service and then we specify the transport so in this case we know of course we're using kafka and then we have an options array here so in the options array we have a client object and we can specify the client id for the kafka consumer so in this case it's billing we'll call it billing and then we can specify the brokers that the kafka server is listening underneath and of course that's going to be localhost 909.2 by default which is where our kafka server listens now we also are going to specify the consumer object here and specify the group id for the building consumer so we'll call this billing consumer so now we have a configuration set up for the billing service we can actually go back into our out service and use this so let's go ahead and inject this and i will add the constructor here and we can add the inject param decorator that takes in the injection token which of course we know is billing service so this can be a private read-only billing client is what we'll call it and this is going to be of type client kafka so now we have a handle to this billing client we can actually emit an event here so let's emit an event and as it does in the tcp transport layer we have the same parameters that this emit takes and we have the pattern so in this case we'll have order created as our pattern which essentially will just be the topic and then we can specify a data payload now we need to actually create an event so i'll do that we'll have a order created dot event so this will be a class we'll export called order created event and the constructor so in the constructor we're going to take in a read only say we'll call it order id type string and then we'll add the user id type string as well and then finally the price everything that the billing service is gonna need to know in order to build the user so now that we have all these properties on the event there's one more thing we need to add so by default if we just provide a plane object as the payload here an sjs will simply stringify this and all will be well however if we use something else like a class an sjs will call the tostring method on that class so if you don't have this implemented in your class none of the properties on it will actually get transmitted so in order to resolve this we can actually implement our own tostring method on this class which will be simply a json.stringify with all of the properties that we have in our class so now that when nest calls to string in this class in order to serialize it we get all the properties being sent to the billing service so now we can specify a new order of created event and we can pass in the order id we'll just use one two three and then we'll pass in the user id and the price so now that when we create a new order we are correctly emitting a message to the building client using kafka topic order created and passing in the order created event here so now we need to set up the billing client to actually be able to listen for incoming messages over kafka so let's go ahead and open up the main.ts in our billing application and we can remove all of the boilerplate in the bootstrap method and as we've done in my previous video with tcp we're going to instantiate a crate microservice here with microservice options okay and so we're just creating the microservice right now instead of over http we're going to be listening over the kafka protocol so we pass in the app module here as the first parameter and then an object here with all the options so of course we can specify the transport which we know is kafka and then the options object as we've done in our client so we have the client object here we can specify the brokers we know that we're going to be listening on localhost 909.2 and lastly importantly we're going to add the consumer property and then we're going to call the group id and give it the same group id that we specified in the client now this is important for this billing service to be able to pick up messages properly we need the same group id that we specified previously in our client so finally we'll call app.listen here so now that we are listening for messages on the billing consumer so let's go ahead and open up our app.controller so we can actually add a route here for the order created event so we'll add the event pattern decorator here that takes in the pattern or the topic name so in this case we know it's going to be order created and we can call this handle order created so we're going to get the actual capcom message here that we'll call data and then we can call this dot app service dot handle order created and i'm going to pass in data dot value which is where the order created event actually lives so the value property is where the payload will be living the other information on this data property here will just be kafka related uh like the offset topic partition and so on but we're only concerned with the data payload right now so let's go ahead and open up our app service now and we'll create a handle order created function that takes in the order created event so we're going to have to specify this like we've done before let's be called order created event and i'm just going to go ahead and copy over the constructor from our other order created event because it's the same thing here so then we can specify the type here order created event now for now let's just go ahead and log out the order created event to see that our microservices are set up properly go ahead and open up postman and then send a request a post request to localhost 3000 which is where api gateway is listening for that handle uh order created and in our body here we include the user id the price and some extraneous property that i can remove here we don't need that so if we send off this post request to our api gateway a few times and then go back to our terminal here we should see that our billing client or a billing service is logging out the order created event properly as we'd expect now if you're not seeing this uh message here make sure you wait until you see this log message from nest that the microservice started successfully because sometimes it can take a little bit for the service to actually rebalance and and join the con the consumer group so now that we actually receiving the order created event we need to reach out to our auth service and get the user associated with this event so we can get their stripe billing id so as we've done before we're going to need to set up a handle to a different external service so as we have already done we will specify the client's module register function here and pass in an array and we'll just want a single service here so we'll call this all service okay and of course the transport will be transport.kafka and then as we've done before we'll specify the options object with the client object inside we'll give it a client id of all we'll list the brokers out here yes it's just localhost 909.2 we'll add a consumer object here with the group id we'll give it name off consumer so now we have registered this service in our app service as we have already done in our billing service we're going to inject this here so that we can use it so i'll have the inject param decorator here and we'll call and we'll include the off service injection token this will be private read only auth client this time and this will be client kafka so now we have a handle to the auth client we need to reach out and actually get a response back this time which is a little different because with the event pattern we didn't care about the response at all so in sjs with the kafka implementation in order to get a reply back a reply is actually its own topic so we send a message on a separate topic and then we receive a message back from the auth service on a different topic so we need to tell mesjis to actually subscribe to that reply topic and in order to do that in our app controller we will say that we want to implement on module init here on module init and then we will implement on module net and we need to get a handle to the auth client like we've done in our app service so we can copy this injection here and paste it in after our app service and make sure we import the inject and the client kafka here so then we need to call this auth client dot subscribe to response of and this is going to be the topic that we want to subscribe to the response of okay and in this case it's going to be called get user now this is the topic that we're going to use to send a message to the auth service and we want to subscribe to the reply topic and sjs will automatically do this for us the topic ends up being just the name of the topic and then reply but we just need to include the original name of the topic here which will be get user so let's go back into our app service now now that we're correctly listening for replies we can call this.authclient.send okay so send will actually send a message to a service and then wait for the reply back so we'll specify the pattern here which of course we know we just set up is get user and then we have the actual payload that we want to send so let's go ahead and set this up we're gonna need a get user request now so i'll create a new file here called get user request dot dto and this is going to be a simple class get user request with a constructor and all it's going to take is the user id that we want to retrieve now don't forget since we're using kafka serialization we need to implement our own tostring method here so we'll return json dot and just provide the user id so that nest will serialize it properly so back in our app service we can actually now create a new get user request and we'll pass in the order created event dot user id so importantly in sjs it won't actually send this message unlike events where it just fires and forgets immediately messages in case where we're using the send function here doesn't get sent until we actually subscribe to the response so we actually do want to subscribe here we get the user back so let's get the user back from the auth service and then we will log a message out let's say we will be billing user with stripe id and this user will have a stripe user id on it and we'll say for a price of dollar sign and then we'll pull the price off of the order created event and we'll add a few dots here so our message ends up became becoming billing user with stripe id and a price of price now that we're successfully handling this order creation and of course here this is where we actually reach out to the stripe api and build the user but for safe sake of example we don't need to actually implement that but we do need to go ahead and set up the auth client so now let's go ahead and finally set up our auth service to be able to listen for incoming messages so go ahead and open up the main.ts for auth to save some time let's actually copy over the existing one that we implemented for the billing service i'm just going to copy over this whole main.ts paste it in here and the only thing we're going to change is the group id here we're going to call it auth consumer which if you recall needs to be the same that we used in our app.module here so next let's open up our app.controller and instead of the event pattern this time we're going to use the message pattern decorator which similarly takes in the message pattern which of course is get user which will be the name of the topic that sgs is using so we'll have get user here we get the whole data payload and we will return now we're actually returning a message appservice.getuser and as we've done before we're going to pull the value property off of this data payload so now in our app service let's go ahead and implement the get user function which would be very simple we'll actually have the get user request as well so create it real quickly get user requests.dto and as we've done before i'm just going to copy over this request from our billing application so now we can specify the get user request here i'm actually just going to go ahead and create a simple in memory array here of users so i have a private read-only array with a couple of user ids and their associated stripe user ids so now in our get user function i'm just going to call this dot users dot find based off the user i'm going to compare the user.userid and see if that equals the getuserrequest.userid so this will simply just pull out the correct user from this array so go ahead and open postman back up and let's go ahead and send off a few different requests here so we have user id 345 and then we also have user id123 or you can even change the price here so we'll send off a few different requests so now if we go back to our logs we should see that our billing service is correctly logging out this log here billing user with stripe id and the price now we can see that we've actually correctly implemented a kafka js microservice architecture showing you how to use the event pattern and the message pattern to communicate between these microservices so i hope you've learned something in this video and it's been helpful if you have any questions please post a comment be sure to like and subscribe and i'll see you in the next one

Channel: Michael Guay

Views: 34,521

Keywords:

Id: JJEKPqSlXvk

Length: 22min 56sec (1376 seconds)

Published: Fri Jan 21 2022