Build Nest.js Microservices With RabbitMQ, MongoDB & Docker | Tutorial

what's going on guys today i'm super excited to bring you a brand new nest js microservices build we're going to be creating a brand new microservice from scratch this is going to be a real working example application that includes jwt authentication between our microservices we're going to persist data inside of a mongodb database and we're going to use rabbitmq as the message broker between our services ravidmq is great it's going to allow us to build a distributed application that can scale up to several different running containers at once and it's also going to allow us to build a fault tolerant messaging system that will allow us to replay messages that have failed so without further ado let's jump right in just a quick note i'm going to be separating the deployment out into my next video so if you'd like to see us actually deploy this application to aws make sure you stay subscribed and keep an eye out for that video okay so we're ready to go ahead and start creating our nestgs microservice app and to do this we're going to create a workspace with the nest cli which if you're familiar with a monorepo will allow us to have a single code base where we can store all of our different microservice apps and all of our common libraries where we can share code between them now as always i'll have this entire project in a github repo and i will include a link in the description so you can feel free to follow along so let's use the nest cli to create a new app so we'll use nest new and we'll call it ordering app then we'll go ahead and use npm to scaffold our project and install our dependencies so now we can go ahead and cd into our ordering app and now by default this ordering app is just going to be the basic nest application out of the box in order to actually convert it into a mono repo we're going to use the cli again and actually generate an app and now we can actually create our different micro service apps within this monorail so let's go ahead and create our first and primary app called orders which will actually be responsible for taking in the api requests for our orders and dispatching appropriate events to the different other services so we've gone ahead and created a new app orders and converted this project into a monorepo so we can go ahead and open it up in a code editor and take a look at the project now so you can see out of the box we actually have this nest cli that was created and if we open it up we can see that it has been converted to our monorepo and we can see the projects here so by default it did convert our original application into an app and then we have our other app that we just added the orders app and importantly the benefit of this is that they're all going to share the same common ts config that we have at our root as well as our eslint.js and they even share the same dependencies in package.json out of the box so to clean things up a bit we can actually remove this original ordering app from our application here then we can also remove it from the nest cli so let's go ahead and remove ordering app and then let's change the route here to orders because now our root application is the orders app and not the ordering app and then make sure you also change the ts config path here to apps orders dsconfig.app.json so now that we've created our base monorepo it's very easy to add additional microservices and common libraries to our app so let's go ahead and do that by finishing adding our other apps we'll go ahead and run nest generate app and then we'll add a billing application to our monorepo so we have our billing app and then lastly we'll add our final app called auth which is going to have all of our authentication so back in our editor we can see we have two new folders created with the default nest js bootstrap out of the box we get the app.listen at port 3000 and you can see by default it creates a ts config that will simply extend our root ts config so we get these apps created out of the box and we can see our nest cli has also been updated so that we know how to find these if we want to run them and we can actually do this by opening up a terminal and as we would we could always run npm run start dev and this would start our default application out of the box which would be our orders app which we first created if we wanted to start a different application like our billing app we just provide the name and now we'll start the billing application so let's close down this app and let's add one more part to our application and that's our common library so if we want to share code between these different microservices we'll do it by creating a new library so let's do that by running nest generate library common and then it will ask us if we want to use a prefix we'll use the default add app here and so now we should see a new folder created called libs and now inside of libs we have a common library and under here we have a source directory and this index.ts is simply exporting common services modules and other utilities that our applications can share and if you go ahead and look in our nest cli we can see our common library was added and you can see the difference here is the type is of type library because you wouldn't actually start this application but you would import this code into your existing apps now the first step in our project we're going to go ahead and add mongodb to our application so that we can persist data in our case actually persist the orders that the users are creating in our system so in order to do this i'm going to take advantage of some code we've already written in a previous video and so if you haven't seen this video about going through mongodb and sjs i'll include a link in the description so for now what i'm going to go ahead and do is i'm going to take those files from that project that allow us to work with and include it in our common directory so instead of common in the source directory i'm going to go ahead and delete everything we have so far and then i'm going to paste in a few files here and like i said this whole project is in github so you can easily get access to these database files here but basically all we have are four different files the first of which is an abstract repository which will expose methods that will allow us to create find one find one and update and upsert documents additionally we're going to have the ability to find documents and start a new transaction a new database transaction which we will actually delve into a little bit later and we'll go over this in detail so we have an abstract repository that we can extend and get all this common data access functionality we then have an abstract schema which is simply a schema document that specifies the underscore id property which all of our documents will have then we have the database module itself which will actually instantiate the mongoose module and pass in the mongodb uri from a config service which will take it from an m file and we'll go over this in detail shortly finally we have an index.ts file which just goes ahead and exports these files to the rest of our application so let's go ahead and create a new database folder inside of the source directory and let's go ahead and move those three files keeping the index.ts at the top here and then we can go ahead and update our path here to these files in our index.ts so that we're correctly exporting them so let's head back into our abstract repository and of course right now we're not going to have access to these dependencies because we haven't added them so i'll go ahead and open up a new terminal and we'll simply install mongoose to start now this monorepo is the same as any other project out of the box when we install dependency we'll have access to it in all of our apps and common libraries so next if we go into our abstract schema we can see we also need to install nestgs slash mongoose and finally if we go into the database module itself now we can see we need to import the nest js config package so let's install npm and sjs slash config and this like i said before will allow us to extract information from environment variables and so it'll be important that when we do use this database module in our application that we have this mongodb uri as an environment variable so now that we have this common database library let's go ahead and start using it in our orders app so we can actually persist orders before we do so we can feel free to go ahead and open up our terminal and start dev and provide the name of the app we want to start so in this case it's orders and make sure we say npm run start dev which is your application correctly started up and by default we will be listening on port 3000 if we open up our main.ts and we can verify this by opening up postman and sending a get request to localhost slash 3000 so let's go ahead and launch a get request and we should see hello world coming back from this api so let's first go up open up our orders dot module and let's set this module up so that we can use our new database module and actually persist data so first off we're going to need to actually instantiate our config module because if you remember our database module will be using it in order to get access to the mongodb uri so we can simply import the config module from at nsjs config so we call config module dot for root and we can provide some options here let's go ahead and set it to global so that the config module will be globally available in our entire orders application and we don't need to keep re-instantiating it and then we can go ahead and provide a validation schema so this will essentially make sure that we have certain environment variables defined and if they're not defined we will throw an error when starting the app so in order to do this we need to use the joy validation package so we can simply go ahead and npm install joy go ahead and let that install okay with that installed we can go ahead and import star as joy from joy up top here then we're going to go ahead and define the joy dot object where we're going to provide the environment variables we expect to be defined so of course right now all we have is our mongodb uri and we're going to specify that this should be a string and that it is required to start up our app now next we need to actually specify the file path for the dot m file we will define because we want to have a different m file for each of our applications each of our applications in our architecture is going to be running independently of one another in its own container and it needs its own set of environment variables so to do that first we can actually just define a dot m file in our root here and then we'll go ahead and open it up and define our mongodb uri we'll set it equal to a connection string and this connection string will leave empty for now because we haven't actually set up mongodb in our architecture but we can go ahead and set the file path to the m-file by providing m-file path and then saying to our app look in the dot apps directory apps orders dot m for that m file now that we have the config module correctly set up we can simply import our database module now if you notice vs code auto imported the database module from the libs folder and that will work but you can also provide a path here so by default this path will be slash app and then we can say common so now this way our file paths aren't hard coded and we can rely on ts config to resolve uh these file paths here now that we've imported the database module we're ready to actually create our first schema and actually create a repository to add some crud functionality to our orders app so i'm going to go ahead and create a new schemas folder inside of our orders app and i'll simply create an order schema.ts and i'll decorate this with the schema decorator which will import from we'll import this schema from nsjs mongoose and so we have this schema decorator here and we're going to specify the version key to false because we don't want to deal with versioning in this app so exporter class called order and it will extend our abstract document from our lib common folder and of course we'll prefer to use the app slash common import here if you recall the abstract document is simply a class with the underscore id property which is the default identifier now inside of this class we'll go ahead and specify a few properties here and so make sure we import prop as well from nest gas mongoose and then we can specify some different properties so we'll help us name of the order then we'll specify the price of the order which will be a number and then we can say we will have a phone number maybe for the user in order to notify them later on after we have correctly built them so next we're going to go ahead and export the order schema which will be schema factory create for class and we'll pass in the order itself and we go ahead and import the schema factory as well from nest js mongoose so now that we have the schema all set up we can go ahead and create our orders repository which will allow us to manipulate data inside of the database so we'll create a orders repository and this of course will be an injectable class here from nasa jazz common and so we'll export a class called orders repository and it'll extend the abstract repository which we have set up in our common database library here the name of the schema we're passing in so in this case it's the order that we just created and then we're going to have a logger that we're going to have to declare inside of the repository and then we need to go ahead and provide a read-only logger here so that our abstract repository can log and we'll import the logger from nestjs common and just simply provide the name of the class so in this case it will be the orders and then in the constructor we can use the property injection inject model which will be imported from nesgus mongoose so let's go ahead and import the inject model from an sjs slash mongoose so we have this inject model and then we're going to provide the name of the order schema we created and we'll get access to the order model here which is a type model type order and we import model importantly from mongoose so let's go ahead and import model from mongoose so now that we have the model itself we're also going to go ahead and use the inject connection decorator and we'll import this as well from nest js mongoose so go ahead and use inject connection and this will allow us to pass up the connection to the abstract repository which will be important for when we want to track a database transaction so let's go ahead and label this correctly as a connection which we import from mongoose so now that we have these two properties injected we can pass them up to the parent abstract repository and now we've done all this work we get all the crud functionality from the abstract repository for free so lastly let's go back into our orders module and we will go ahead and actually instantiate our orders repository here and then underneath our database module we will call mongoose module which you might expect will need to import from nest js mongoose so i'll go ahead and import the mongoose module from sgs mongoose and then we'll call for feature and actually register our order schema so we do this by providing an array of objects with each schema and so in our case we need to provide the name of the schema we're looking for in this case it's the order and then we'll provide the actual order schema itself which we created using the schema factory so now we have everything in place to actually persist orders we can take a look at our back end and as you'd expect we have an error being thrown because we have this config validation failing since we haven't defined our mongodb uri so let's go ahead and get mongodb up and running so that we can connect to it okay so we're ready to start up our app and mongodb in order to do this we're actually going to use docker compose now docker compose is a tool that allows us to start up multiple docker containers at once in a single application so we're going to do this in our application by starting up each of our microservices independently and all of our dependencies so we'll start up mongodb along with our microservice apps so let's dive into this first we're going to create a docker compose.yaml now this docker compose yaml will be a list of instructions to docker on which images and containers to start when we want to start our application up now we want to start up mongodb to begin with now we could just use the base mongodb image from mongodb however we're going to need to start up a replica set in mongodb in order to use the database transaction functionality and in order to do that we can open up our browser and let's google for bitnami mongodb now bitnami is a image provider and they have an image for mongodb but they also have one that allows us to create a replica set very easily and make use of advanced functionality like database transactions so i'm on this github page for bitnami docker mongodb i'll include a link in the description but what we're concerned with here is the docker compose replica set so let's go ahead and open this up and if we open this up we can see a list of services that we want to start up so in this case we have a mongodb primary node a secondary node and an orbiter now we don't really care too much about what's going on behind the scenes we just need to be able to start up this replica set and we also have a volume which will allow us to persist data between application restarts so let's go ahead and just simply copy this entire yaml file and paste it in to our docker compose so now we should have uh three services and we can see that we're pulling the image from the bitnami mongodb here and we're also specifying some environment variables and lastly we specify the volume that we want to use in this case mongodb master data with a local driver so that we persist this data now the only thing i'm going to add here is for our mongodb primary i'm going to go ahead and add a port section so let's go ahead and specify that we want to use port 27017 and we want to map this to our machine port of 27017 so that we can access the mongodb on this normal 27017 port so now that we have this all set up let's open up our terminal and in our same directory we should be able to run docker compose up and of course you need to make sure you have docker running and installed and if you don't i can include a link in the description for how to get up and running with docker so if we go ahead and run docker compose up you should see that our containers are starting up they will go ahead and run through the initialization process so now that we have the ability to run mongodb in a replica set in our application we want to actually be able to run our own app inside of this docker compose in order to do this we need to specify a docker file for each of our microservice apps so that we can use them in our docker compose here so let's go ahead and do just that we'll go ahead and start with our orders app so let's go ahead and open up our orders directory and let's create a new docker file now i'm going to go ahead and paste in a list of instructions for how to build this docker image now i have a separate video where we delve deeper into each of these instructions and learn more about how docker works in sgs so if you're interested in that i'll include a link in the description but from a high level all this image is doing is providing the list of instructions for how to build our container so you notice here we're copying over our rootpackage.json running npm install copying over all the projects running npm build and then finally at the bottom here we're actually starting our application using node in the dist apps orders main folder now one drawback of building our images like this is that all of our microservice apps will share the same package.json and the same dependencies so you may have a scenario where you have one app that has a lot of dependencies which other ones don't need in that case you may want to consider having a separate package.json for each app but in our case our apps have limited dependencies and we can share this package.json just fine so now that we have a docker file with a list of instructions of how to build our app let's go ahead and create a dot docker ignore for docker which essentially will specify files that we want to ignore when building our image so i'm going to paste in a list of files here and these files are just versioning and metadata files so dot get dot get ignore docker ignore itself but then also build dependencies like node modules environment variables and other miscellaneous files that we don't need in our image we're not going to include the dot m files because we're actually going to inject these environment variables in the docker compose itself so let's go ahead and do that by opening up the dockercompose.yaml and now we can actually specify our orders service and so what we're going to do now is specify the build section next we're going to specify the build context so this is essentially where in our directory do we want to start this build from well it's going to be the root directory where we already are then we're going to specify the docker file that lists the instructions for how to build the service and of course we know this is the one we just created so this will be in dot apps orders docker file now we're going to specify a target for which part of the docker image we want to use in this docker compose now if we take a closer look at the docker file we can see it's broken up into two stages the development stage and the production stage production stage will only install the production dependencies which is not what we want when we develop we want all of the dependencies including the development dependencies so to do that we can specify we want to target the development stage and lastly we can specify a command that we want to run the image with and essentially override the default command now this command would work but we want to be able to refresh our app when we make changes easily as in when we normally use the nest cli simply override this by providing the command npm run start dev orders so that our application will restart when it detects file changes now let's go ahead and specify the m-file which i talked about earlier and of course this will be in apps orders dot m and so this will load the environment file that we specified here and inject all these environment variables into our container next let's go specify a section called depends on which will simply be a list of services that we need to start before we start this one so in our case this will be the mongodb nodes so we want the mongodb primary secondary and mongodb orbiter to be able to run before this one starts then we're going to specify a volume section so we're going to go ahead and mount our entire application into the user source app directory which if we remember is the working directory for our image and the reason why we do this is because we want to be able to detect if there are any file changes and if there are we want to reload the application finally we're going to provide one more volume here and this will be user source app node modules and this is a bit of a hack to specify that we won't we don't want to mount our own node modules inside of our project here but we want to maintain the ones inside of our running container so that we don't include any node modules that are just part of the machine that we're running on so we have our volume set up and now we'll specify a port so of course we know that this application will be running on port 3000 and will expose port 3000 on our host machine as well so now we should be able to go ahead and start up our orders application as part of our image so let's go ahead and run docker compose up now i'm going to run that dash build and then dash v to reset all volumes because we have a new image in our docker compose so you can see it's going ahead and building orders based on the docker file that we have specified so after we start up our images you should have seen the orders container flash by and if we scroll back up a little bit we should be able to see our orders container and of course we still have the same error about the mongodb uri being empty because we still haven't provided the mongodb uri so let's go ahead and do that in our dot m file we will now provide the mongodb uri so this connection string will be mongodb and then we go ahead and provide the username and password now by default in bitnami the default username will be root and then we can see that our password here is password one two three so we'll go ahead and provide the password as well and then we specify at and then we'll specify the host now since these containers are all running inside of docker we can actually use the name of the service itself to connect to it so in this case it will be mongodb primary and then of course the port 27017 now we have our connection string in place i've gone ahead since restarted my containers and we can see them starting back up now we can see our orders application successfully starting and if we give it a little bit we should be able to see it starting once finishes yep so here we see our net application has successfully started for our orders container so now we have a mongodb full replica set running as well as our order service all in one application using docker compose so we're actually ready to start developing our orders app and allowing the creation of new orders so let's get started by opening up our orders application and opening up our orders controller now just so you remember whenever we make a change to our orders application like for example just putting in a comment here we should expect to see our application fresh and restart so let's go ahead and start by opening up our orders controller and let's get rid of this default get route here we'll go ahead and create a new post route that will allow us to create an order so we'll call this async create order which will of course take in a body that we'll call request and this will be of type create order request which we will shortly define and we're essentially going to just return this dot order service dot create order and pass in the request now we need to go ahead and define the create order request let's go ahead and create a new dto folder and then we can create the create order request so now we have this dtl we can define let's create a create order request and now we want to apply some validation and you probably guessed that we can use class validator so let's open up our terminal and install class validator so that we can easily validate our dtos additionally make sure we go ahead and install class transformer which is required for the class validator package to work properly now we have this class validator package we'll go ahead and provide three properties we'll have a name of the order the price of the order and the phone number of the user that created it which will be a string in our case so we can provide some simple validation to these properties for the name we should expect it to be a string and it should not be empty the price should just be a positive value and lastly the phone number actually has its own validator called is phone number which will validate that this is a valid phone number with a country code so now we have this validation out of the box we need to open up our main.ts file and make sure that we actually apply the global pipe so we'll call app.use globalpipes and then call new validation pipe from nest js common so while we're here we also want to make sure we're not hard coding our port because this will make it difficult when we want to deploy our application in a different environment so let's go ahead and pull the config service from the app we will call app.get config service let's provide the config service dot get and provide the port here now we'll actually need to provide this value in our dot m file with this simple we'll just provide a port of 3000. additionally let's open up our orders module and add another validation here number and this should also be required so back in our orders controller let's go ahead and import create order request so now we've correctly imported it we can get rid of this get import we now can go into our order service now we can actually persist the order let's create a new async create order function which takes in the request of type create order request now all we want to do right now is simply persist this to the database and that is all so let's go ahead and inject the orders repository that we created earlier this will be private read only orders repository type orders repository and all we want to do is return this dot orders repository dot create and we'll go ahead and pass in the request so now if we go back to our terminal we should see some issues that class validator is not defined well this doesn't really make sense because we did install it the problem is that whenever we add a new dependency we need to make sure we start up our application with data build and dash v to make sure we force the containers to rebuild to restarting our containers we should see our orders application starting up successfully again so we're ready to test this api out and before we do so let's just go into our orders controller and provide the controller with an order's name so that we have the correct name for the api and then we should be able to open up postman and launch a post request localhost 3000 slash orders now i have a body set up here with the name of t-shirt a price of 39.99 and a phone number with an area code importantly so if we send off this request we can see we get a 201 created back and if we were to for example send a request with no price for example we can see our validation working properly okay so now that we can create an order let's add a route so we can retrieve existing orders to make sure our database is actually persisting orders correctly so let's go ahead and add a get route here we'll call it get orders and it will simply return this dot order service dot get orders and we can go ahead and order service create a get orders function and all we're going to do is reach out to the repository make a call to find with a empty filter query so we retrieve all orders make sure we save our controller here and let's go ahead and open up postman again and launch a get request at our orders api so i'm inside postman i'm going to go ahead and send a get request here and then we can see a number of orders that i have created coming back in an array here so now we're ready to start connecting the micro services in our application we're going to start by initializing our billing application as a rabbitmq microservice so to go ahead and do this we're going to open up the terminal and add a new dependency so this dependency will be at an sgs slash micro services so make sure we go ahead and install this so after we install an sgs micro services we're also going to need amp qplib and the amp qp connection manager to establish a connection to rabbitmq and then what i want to go ahead and do is i want to go ahead and in our libs folder in our source directory let's go ahead and create a new folder called rmq to stand for rabbitmq and what i want to do is create a new rabbitmq module that will encapsulate all of our rabbitmq related functionality so let's go ahead and create class called rabbitmq module and we'll decorate this with the module decorator from nestjs common and now let's also go ahead and create a new rabbitmq service now this rabbitmq service will be an injectable class that we'll call rnq service and like i said before this service will allow us to encapsulate common rabbit mq functionality so the first functionality we're going to add is a method called get options which we can use for each microservice to call when they're initializing it so we can provide a single place where we can figure all the options for our rabbitmq microservices so we're going to need to get the name of the rabbitmq that we want to initialize and then we're also going to add a property called no ack and set it to false by default now in rabbitmq we have the ability to say that we want to manually acknowledge a message before removing it from the queue by default an sgs will automatically acknowledge all these messages so if we want to handle this acknowledgement manually we need to specify it so so let's go ahead and return an object now we can also specify the return type on this method and say that this will return the rabbitmq options from sgs microservices and then we're going to specify the transport well we know of course the transport will be rabbitmq and then we have an options object here so to in order to provide some options for our rabbitmq microservice we're going to go ahead and inject our config service so that we can get access to environment variables so let's go ahead and import that and now we're going to provide a list of urls so this is going to be the urls where rabbitmq is listening on or our brokers so it's an array and then we'll call this.configservice.get it'll be a string and then we'll call it rabbitmq uri so like before with mongodb we're going to make sure that this environment variable is available in any service where we're using this and so now we can provide the actual name of the queue that we are creating so let's call this.config service.get this will be a string and then we'll provide a template literal here where we'll say rabbitmq q will be the queue that has been passed into us and we'll call this q so the queue that's passed into this options function will be the environment variable that we're extracting lastly we'll provide that no ack property and then we'll say persistent true so that we maintain our list of messages so now that we have this common functionality to initialize a rabbitmq microservice we can go ahead and open up our billing application main.tf and firstly we're going to go ahead and get access to that rabbitmq microservice by calling this dot app.get rmq service and then we will provide the name of arm cue service i'm going to go ahead and change the import here to be from app.com and you can see here we're getting a complaint that app.common has no exported member arm cue service and that's because in our index.ts we need to actually go ahead and make sure we export it from rmq arm cue dot service lastly in our rmq module we'll go ahead and make sure that we actually provide the arm key service here and then we can list it as an export we'll go ahead and list that as an export so now back in our main.ts we have access to the armq service we're going to call app.connect microservice and now we're going to pass in the arm cue service dot get options and now we're going to provide the name of the queue that we want to create so in this case it's the billing application we want to create a billing queue for messages to listen on so we'll pass that name in and secondly we're going to set no act to false by keeping the default because we want to manually acknowledge messages we receive and in case of a failure we want to replay a message so we're going to leave those options as they are and then instead of calling app.listen we'll call app.start all microservices and this is all we need to do to start up a rabbit mq microservice make sure we also import the rmq module here from the app.common and if you recall as we did earlier we're going to need to go ahead and export the rnq module so let's go ahead and do that we'll just copy the rmq service and change this to module so now that we have the rmq module imported you'll recall that we need to create a set of environment variables for our billing application so let's go ahead and create a dot m file the first environment variable we need of course is the rabbitmq uri so we'll leave that empty for now until we have rabbitmq actually running but we can also specify rabbitmq billing queue because as you recall in our main.ts we passed the name billing in which corresponds to this environment variable so we need to make sure we have this available and we'll call this q billing so now we actually want to start our billing application we're going to go ahead and do the same thing we did with our orders application by creating a docker file in here so that we can containerize this app and run it as part of our docker compose so let's go ahead and open up our original docker file and we'll simply copy everything over into this one we're just going to make a change down here when we start the application to make sure we go into the building but other than that the steps to create the application are exactly the same now let's go ahead and open up our docker compose so we can actually set up our billing application so this is going to be almost exactly the same as orders so we'll go ahead and actually just paste our existing configuration for the orders application however we're going to go ahead and change the name to billing and then we're going to change the path here to the docker file to be in the billing project we're still going to target the development stage and now when we start the application we want to start the billing app so make sure we change that we'll change the path for the environment file we'll keep our volumes exactly the same and then we can remove this port here because we won't be using it so while we're here we can also easily add a rabbitmq service and we can do this by creating a new service called rabbitmq and we'll provide the name of the image so in this case we'll use the default rabbitmq image hosted on the docker hub repository that's all we have to specify then we'll specify the port here that we want rab mq to be exposed on so this will be five six seven two in the container and the host machine so this is really all we need to get rab mq running quite easy and so now that we have revit mq the ability to run it let's open up our billing and file we have the rabbitmq uri and now we can specify the uri which is going to look like this ampq colon slash this will be the host so since this is docker compose we give it the name of the container which in our case is simply rabbitmq this will be wrapped mq and then of course the port we specified was 5672 and this connection string here should actually be am qp now additionally we need to instantiate the config module in this billing module because rab mq is going to use it so let's go ahead and call config module for root here where we'll specify that we want to be globally instantiated and then we also want to provide some validation as we've done before so we'll use our validation schema here we'll go ahead and import joy by calling import star as joy from joy and then we can call joy.object and then inside this object here we of course know that we have two environment variables so let's go ahead and provide them we have the mongodb queue uri which will be a required string and then we have the actual name of the billing queue which we want so let's go ahead and provide this by labeling it as a required string as well now let's go back to our terminal and stop all of our containers since we've installed some new dependencies next time we start we're going to run docker compose up dash build dash b so go ahead and let these containers finish building and running so in our logs we should eventually see the billing application come online and you should see the nest mic service successfully started message so now that we've gone ahead and set up our billing microservice let's go ahead and actually use it inside of our orders application so we can communicate with that service to do so i'm going to go ahead and open up our rabbitmq module so we're going to go ahead and create a dynamic module so to turn this into a dynamic module we're going to go ahead and create a static register method okay and this is going to take an object which specifies the name of the service we're trying to register we're going to define this as rabbitmq module options which we can go ahead and quickly define above as an interface here so m armit and q module options which will simply take a name that is a string so this function is going to return a dynamic module from nest js common and now we can go ahead and return a dynamic module so in this case we're going to return the rabbitmq module that's the module that we're returning now we want to provide an import to right here and importantly in the imports array this is where we actually want to register the rapidmq service that is being passed in here so let's go ahead and call clients module dot register async now clients module is the module that we're going to go ahead and use to register a rabbitmq service and we import it from sjs microservices above so this is going to take an array of services that we wanted to find so let's go ahead and define the first service by opening up a new object and we provide the name of the service here and then we go provide an options object which will provide the information for how to connect to this service now that we specify the name of the service we're going to use use factory here to be able to inject the config service when we define how to connect to this service so we need to get access to the config service i'm going to import this from sjs config and go ahead and return an object here so inside this object you just specify the transport that we actually want to use for this microservice so of course for our case we know this is going to be rabbit mq so we go ahead and define that and then we have the actual options object which defines how to connect to this server so i'm going to go ahead and define the first property which is urls which will be the urls that our rabbitmq server is actually listening on so of course in our case we know that we can extract this from the config service we'll call get give it a type of string and we know this will be the rabbit mq uri which will of course be defined as an environment variable and make sure we change the config service here we have a typo so this should be config service so now we have the list of urls now we need to find the actual name of the queue that we're trying to register here so this will be config service.get and as we've done before in the rabbitmq service when we're actually defining the name of the queue inside of the bootstrap method we're going to use this same exact pattern here so to actually register the queue we can just pull it out the config service and this will be the name here of the service that we're trying to register this will be pulled from the environment variable as well so now we have everything we need to actually register this microservice in our application we're going to go ahead and of course inject the config service that we are using here so we have access to it in the use factory and then lastly in our dynamic module we want to re-export the clients module here so that the consuming module that imports it will have this client's module with the registered rabbitmqservice that we just created so now let's go ahead and actually use this dynamic module inside of our orders app so that we can register the billing microservice and actually use it i'm going to go ahead and open up my orders.module here so now we can actually import the rmq module and i'm going to go ahead and add the arm cue module from our app common import above and then we can call that static register method to go ahead and actually register the billing service so all we have to do is provide the object here we just need to provide the name of the service here so let's go ahead and create a new folder called constance and then we'll create a new file called services.ts and i'll simply export a const called billing service which will be the name of the rabbit mq service that will be defined in our m of course so at the same time we're going to go ahead and also open up our dot m and make sure that we have firstly our rabbit mq uri defined which of course we know should be amqp poland rabbit mq and then give it the port of five six seven two now we also need that rabbit mq billing queue to here which will be the name of the billing queue that will be used in our rabbitmq module so now we can actually go ahead and reference this constant value we'll use the billing service here and now we have this microservice registered inside of our orders module now we can actually make use of it by opening up our orders.service and we can of course inject this billing service right inside of our code so we can actually communicate with that microservice now to do so we're going to use the inject decorator here from nest js common and then we're going to provide that same billing service constant that we used in our register call to rabbitmq module so we have that constant value that we're injecting here and i'll go ahead and call this the billing client which will be of type client proxy from nesgus microservices so now that we have access to the billing client and can actually communicate with that billing service let's go ahead and refactor our create order function now in this function what i want to do is only if we are able to successfully create an order and i want to emit an order created event to our billing microservice so that we could go ahead and build the user so in order to accomplish this we can take advantage of database transactions in mongodb so if you're not familiar with database transactions they essentially just allow us to ensure that we only perform some functionality if our database calls succeed and if not we don't perform that functionality so to see this in action let's go ahead and create our database transaction by calling this dot order repository dot start transaction and if we take a quick look at what this is doing in our abstract repository all it's doing is starting a session and then starting a transaction on that session and returning it so let's go ahead and open up a try catch block here so firstly we can actually just say in the catch block if any error were to occur we're going to go ahead and call session dot abort transaction which will abort the current active transaction in this session and cancel any database calls so we don't actually persist the order if something were to go wrong here for example we couldn't communicate with our billing service so that this is this entire transaction here is atomic and we don't persist the order if we aren't able to bill the user so we'll also throw the error here if we do get an error in the try block what we're going to go ahead and do of course is actually try to create the order so we'll call this.orderdepository.create we'll pass in the request and then we're going to pass in the session here so now we've actually tried to create the order let's go ahead and emit the order created event now this billing client will return us an observable and we want to convert this into a promise so i'm going to call await last value from import this from rxjs and then we can call this doc billing client dot emit so we're going to go ahead and emit an event to our billing microservice and we're going to call this order created next we can actually specify the data we want to send through and in our case i'm just going to go ahead and pass the exact request that was passed in to this function or the create order request itself so if we are able to successfully send this message to the billing client then at that point we can call session.commit transaction which will actually go ahead and commit this database transaction persisting the order to the database and finally we'll go ahead and return the order so with this in place let's go ahead and open up our billing application in our billing controller we will go ahead and set up this to be able to receive this event so this is quite simple to do we can simply decorate a new controller function here with the event pattern decorator which allows us to specify the event we want to listen to so in our case we know this will be order created and we'll go ahead and call this handle order created now inside of this function we can get access to the payload from sjs microservices which will be the actual data that we passed in in our case the create order request itself we can also get access to the context here which is going to be a set of information about this request from robin mq so this will actually be of type rabbitmqcontext from sjs microservices so with this information i'm going to go ahead and call this billingservice.bill and we'll pass in the data and so let's go ahead and open up the billing service now inside of here it will be quite simple all i want to do is i want to create a logger that will be specific to this billing service so i will create a new logger from sjs common and pass in the billingservice.name i'm going to go ahead and create that bill function which takes in the data and i'm just going to log it out so we're going to call this.logger.log and say that we're billing based off of the data we have received so with all this in place we should now be able to open up postman and of course like we did before we can send a request to localhost 3000 slash orders so this is our orders application and we can place an order with this body here where we provide a name a price and a phone number now if we go ahead and send this request to our orders application we get our order back persisted in the database but now if we look at our logs here we can see in our billing application we actually are logging out this billing statement here with the request that is coming from the orders application so we are able to communicate between microservices and you can imagine this billing service we would be billing the user based off of this information here let's let's go ahead and all i'm going to do is just save this billing controller again i'm just going to save this file and restart the billing application now if you notice when we restarted the billing application we actually logged out this request again without having to send another request so if i just save again you can see the application restart and again we're replaying these messages off of the rabbit mq and so why is this well the reason this is is because if you look at how we set up our rabbit mq service here we set no ack to false by default so what this means is that we need to manually acknowledge that we have processed a message so that rabbitmq can take it off of the queue and we don't we keep replaying it and so this can be useful if we want to handle message failures but we also need to make sure we do acknowledge our messages when received so let's go ahead and take care of this in the rabbit mq service i'm going to create a new function here called ack which will take in the current rabbitmq context and inside of this function here we're going to go ahead and create a new cards called channel which we'll call context.getchannelwrap okay and then we can get the original message from the context by calling context dot get message and lastly we can call channel dot ack and pass in the original message so this is how we can actually acknowledge that the message has been processed and taken off of the queue so now back in our billing controller we can go ahead and inject the rabbitmq service and so i'll change the import here to be how we've done it before from app.com and then we can call this.armkeyservice.ack and pass in the context so only if this function did succeed and no errors were thrown then we will acknowledge the message and it will be taken off the queue so now if i go ahead and save this file and our application is restarted you can see we are no longer logging the message out because we have acknowledged it and it has been taken off the queue okay so for the next part in our project we're going to actually add authentication to our system so that we authenticate the user before they're able to create orders and we also authenticate our microservices so that we make sure that all messages receive have authentication on them we're going to use json web token authentication for this approach and i have created a video already on how to implement json web token auth in sjs using passport.js i'll include a link to that video in the description where i actually go into detail on how to create this authentication mechanism but for now to save time i'm going to copy over code from that existing video into our auth package so that we can make it take advantage of it and not have to rewrite it all so in this off folder i'm going to go ahead and paste in that code and of course this whole package will be in github so you can take a closer look at it i'll go ahead and remove our default source directory and rename this source now in addition to this we're going to need to go ahead and install some packages so that we can take advantage of this off library and implement authentication so let's go ahead and install nestjs.jwt and sjs passport we're going to go ahead and install the crypt to be able to hash and unhash passwords we're also going to install cookie parser so we can implement our json web token auth as secure http only cookies we're going to go ahead and install passport itself passport jwt and passport local lastly we'll go ahead and install some development dependency dependencies here so npm i dash d and then we'll go ahead and install type slash cookie parser type slash passport jwt types slash passport local and go ahead and let that install next we should be able to open up our main.ts in our auth project and this bootstrap method should be completely empty for now so next let's go ahead and open up our off thought module and walk through what's going on here so we have our database module from our common app library as you've seen before and we also have a user's module here so if we go ahead and take a look at the users module in our auth project it's a simple module that will create a controller and a new mongoose schema here that will allow us to actually create and persist users so we have a controller set up with a simple post route to be able to create a user we have a user service which will simply validate the request persist the user after hashing the password so we have some simple validation to make sure that the email doesn't already exist and we simply validate the user as necessary by comparing the password in the database so back in the auth module we'll keep on going we of course import the rabbitmq module itself so that we can actually make advantage of the rav mq service and then we have the config module as we've seen before so we make this global and we have a validation schema where we need to define some environment variables the jw2 secret expiration and of course the mongodb uri which we will set up soon and have the end file path which in our case will be in the auth project next we have the jwt module itself where we define the secret which comes from an environment variable using the config service and then we provide the expires in using the jwt expiration environment variable finally we have our jwt strategy local strategy and all services providers which use passport.js to implement authentication and lastly we have the auth controller itself which will host our routes to be able to log into our application so by default we have our first post route here which will allow us to log in over http so in our case this will use the local strategy which will simply take the email and password and validate it using the user service so if the username and password are valid we can see we use the auth service to log in all this is going to do is essentially create an authentication http only cookie for this request and set it on the response now finally we have the jdbt strategy which is what we will be using for rabbitmq which will be able to pull the authentication off of a request the jwt that is passed to our auth service is valid and if so it will return the user associated with that gwt so with that brief overview of the auth service let's go ahead and create that god m file now amongst other variables i'm going to go ahead and assign a new port to this service because this service the auth service will actually be a hybrid application we're going to be listening over http so that we can still log in using post requests but we also want to start this as a microservice so we can send messages to the auth service and validate requests a port of 3001 and then we're going to provide a jwt expiration here of 3600 seconds and then we'll have a jwt secret so you can provide a jwt secret value here and this can be any value you want any long random characters so we have the jbt secret and then of course as we have already done we're gonna need to provide the mongodb uri and finally we'll go ahead and add the rabbitmq uri while we're at it and the name of the rabbitmeq off queue which we'll simply call off so with our dot m information complete let's open up our main.ts and start building this out we're going to go ahead and of course create our app using the nest factory dot create method where we pass in the off module here next we're going to go ahead and get the rabbitmq service by calling app.get and giving it a type of rabbitmq service and passing that in and we'll change the import here to be from app common so that we have the rabbitmq service we can call app.connect microservice pass in rmq options here and then call rabbitmqservice.getoptions now we'll provide the name of the queue in our case we know it's off now we also want to set no ack here to true because with the off service we don't really want to manually acknowledge messages because we're going to be using the request response based message pattern instead of events we'll also go ahead and set up global pipes here so we'll call use global pipes and pass in the validation type so we can make use of class validator next we'll go ahead and get the config service as well so we'll call app.get config service here and with that we can now start our microservices by calling app.startall microservices and then we'll call app.listen so that we still listen over http on configservice.getport here so this is how we've essentially created a hybrid application that will list on an http as well as the rabbitmq that we've defined here so with all this setup we are ready to actually add our auth application to our docker compose file so of course as we've done before we're going to go ahead and create a new docker file so we can go ahead and copy this from an existing app so we'll just take the one from billing here thing we'll have to change is the directory where we start the application so in this case we'll change this to off and start it from off and so now that we have this docker file defined let's open up our docker compose and we can add a new service in here so let's go ahead and add a auth service now we'll specify the of course build section first so the build section will have a context where we provide the root context as always we're going to specify the docker file itself which will be in apps off the docker file next we'll specify the target which as before will be the development stage then we set the build section up we'll provide a command here to run and this will be npm run start dev off let's go ahead and specify the ports and now this case it'll be on port 3001 as we've set up in our m and speaking of m we can go ahead and provide the m file here which we know will be slash app off dot m finally let's go ahead and set up the depends on section which will set a list of services that need to be started before this one does so in our case this will be all of the mongodb services and then rabbitmq itself let's go ahead and set that up lastly we'll go ahead and define our volume section here so as a number four we'll go ahead and mount our entire project in the user source app directory and then we'll go ahead and mount the user source app node modules here so now that we've set the auth up let's go back up to our order service here and let's go ahead and flush this out so in addition to depending on mongodb our orders app should wait until billing has started up it should wait until auth has started and it should also wait until rab mqs started similarly billing should also wait until rabbitmq starts and it should also wait for off the start with that complete we should be able to stop our current application and then call docker compose up dash dash build dash v and go ahead and let our new auth project build and our application restart so you can see our application has started up here and if we go ahead and re-save our auth app we should see in our logs the auth application restart and what we're looking for here is to see that the nasa application successfully started as well as our microservice successfully started for our auth application so auth is up and running which is great to see so now we should be able to open up postman and launch a http repost request at localhost 3001 slash auth slash login and of course this will correspond to our auth controller here where we have auth login and we can see i have a body here where i have an email and a password now if i go ahead and send this request we can get a 401 response because this user does not exist in the system now if we open up our users controller we can see how to create a user it's simply all slash users within email and password body so i'll call auth slash users and set a request to create this user now i already have a user with this email so we can see our validation works correctly so i'll use test2 create the user we can see we get the user back with the hash password and now if i change this route to login and send the request we can see we get a response back but importantly if you look under the cookie section we can see we're getting a json web token cookie back and so now at this point we have a json web token we can use to make further requests in our system like our orders application so let's go ahead and set up authentication in our orders application using this json web token flow okay so in order to implement authentication in our system for both http and over our rabbitmq service what we're going to go ahead and do is create a new auth folder inside of our common library then i'll go ahead and create a new auth dot module here so this would be a normal nest.js module and we'll go ahead and export a class called off module now i'm going to go ahead and implement a nest module here and that's because i want to use the configure method to be able to apply some middleware whichever other module imports this auth module i want to implement our cookie parsing so that we are able to parse incoming cookies and look for a potential json web token so i'm going to call consumer.apply and then call cookie parser here which will take cookies and add them to the recurrent request object so i'm going to go ahead and apply this for all of the routes in our system so that we are extracting cookies from every route additionally i'm going to go ahead and create a new services.ts file here which as we've done before we'll export the current service we're looking at so in this case we're going to export the off service so we'll go ahead and export off and this will be our injection token that we can now use so in our off module probably guess what we're going to go ahead and do we're going to add in imports right here where we import the rmq module and we will call register and give it a name of auth service because we want to be able to communicate with the auth service here and then we're going to go ahead and re-export the rmq module so that any other module that uses this one will have the auth service available so now that we have this auth module i'm going to go ahead and create a new jwt off guard in this off folder and this guard will be a normal injectable class from nestjs and so we'll go ahead and explore the jbwt auth guard which will implement can activate from nest js common we'll go ahead and add a constructor where we of course follow that same pattern using the inject decorator here and inject the auth service so that we can communicate with the off service that we've set up so i'll go ahead and call this private auth client which will be of type client proxy from nestgs microservices then we'll go ahead and implement the can activate function which will implement this interface here now importantly we get access to the execution context so we can actually see if this is going to be an http or a rabbitmq quest so let's go ahead and first do that i want to extract the authentication value or the jwt from the current request depending on if we're http or in rpc mode so let's create a new variable here called authentication and we'll call get authentication with passing in the context we of course need to implement this method so we'll put our new method called get authentication which takes in the current execution context and let's go ahead and define the authentication value here which will be of type string next we're going to go ahead and say if the currentcontext.gettype is equal to rpc which will be the case if we're communicating over rav mq then we'll say the authentication will be equal to context dot switch to rpc dot get data which will be the data object being sent and then we'll pull the authentication value off of that data object otherwise if the context.gettype is equal to http then we're going to pull the authentication off of contacts dot switch to http dot get request dot cookies dot authentication because if you recall our authentication over http is implemented with cookies and cookie parser will automatically add this to a cookies object on the request lastly we can now check to see if authentication is currently provided and if it's not then we'll go ahead and throw a new unauthorized exception and say no value was provided for authentication otherwise we'll go ahead and return authentication so now we actually have the authentication or jwt based on whether or not we're in http or in rpc mode lastly we'll go ahead and return this dot off client dot send now unlike emit send will send a request to a rabbitmq microservice and wait for a response back so we'll specify the actual message pattern we want to use here in this case we'll call it validate user that we're sending to the authentication service and we will provide an object which has the authentication property here and pass in our authentication jwt value now we actually want to pipe this rxjs observable so that we can add some additional functionality here in particular we're going to use the tap operator from rxjs which allows us to specify side effects on our calls so what we're going to do is we're going to get the response back from the auth service and with that response i want to add the user to the current request or data object depending on if we're using http or rpc so i'm going to pass in the res and context to this function and finally we'll go ahead and add a catch error operator from arcgis which will catch any errors thrown in this observable chain and throw a custom error so in this case if there are any errors thrown like for example the auth service through an unauthorized exception we want to rethrow that here and say that our guard cannot activate the current route so let's go ahead and implement this add user function below we will add a new add user function which takes the user of type any and gets the context which is the execution context so as i said before we'll check to see if the context.gettype right now is rpc and if it's rpc then we will call context.switch2rpc getdata and we'll actually add the user to the data object so we'll do it like so else if the context is http so get type equal to http then we will simply attach the user to the request object so we'll call get request and attach the user to the request object here so to test this functionality out we're of course going to need to open up our auth controller and make sure we define this route here for rpc calls so we're going to go ahead and specify a message pattern here which is similar to the event pattern we've used earlier so we'll just specify the actual pattern that we want to match here in case it's called validate user and all we're going to do in validate user we want to return the user back to the calling service if the authentication is succeeded so what we're going to do is we're going to call use guards here and pass in the jwt off guard that is from our auth package so if that succeeded then we can extract the current user from the given request and simply return the user back to the calling service so if we take a closer look at this jwt auth guard it's going to implement the jwt strategy now if we look at the dwt strategy that will extend the passport strategy we can see we're extracting the jwt from the request so in our case we know we provided this uh in our jwt auth guard we passed that authentication object here so we're pulling the authentication jwt off the object and we're validating it based on our jwt secret and if this is valid then we call the user service to get the user based off of the user id on the token payload so if this will succeed passport will actually automatically add the user to the current request or data object for us so what we need to do is we need to modify our current user decorator here and add one more bit of logic to say if our context.gettype is equal to rpc then we call context.switch2rpc.getdata.user because by default passport will attach the user to the data object in rpc just like it does for http on the request object now our auth controller we are able to extract the user no matter if we're using http or rpc and then return it so let's go ahead and try this out by opening up our orders controller now in our create order function here we can actually use used cards and pass in that gwt auth guard now i'm going to go ahead and import jbt off guard from the common lib so that we want to actually change the import here to app slash common so to go ahead and actually import this correctly make sure we open up our index.ts in our common library and we'll go ahead and export star from off slash auth.module and we'll do the same thing for our off jwt off guard here so now in our orders controller we are using that jwt auth guard now to make sure this works properly make sure in our orders module we go ahead and import the auth module and this should also be coming from app.com so we'll go ahead and add the off module import here and remove this one lastly make sure we change our import from cookie browser here let's import star as cookie parser from the cookie parser library so once we've imported the auth module in the orders module we should be able to apply this gwt auth guard let's see if it works by using the request decorator here from nestjs common so we can actually extract the request object and let's log out the request.user because if this auth guard succeeded we should now have the user on the request object now finally before we can test this all out we need to make sure in our orders m-file that we're actually defining that auth queue which is what we'll use to communicate with the auth service because if you recall in our orders module we go ahead and import the off module now the auth module is going to use the auth service now if we remember to register it we need to have the auth queue defined in our environment file so in order to make sure this is the case open up our dot amp in the orders application and we'll define the rabbitmq auth cue here and set it equal to off with all that in place let's go ahead and restart our application so we reload that m information once your application is started up successfully we can open up postman and let's send a another request to local 3000 slash orders with our example order here so we'll go ahead and send this request and we can see it succeeded now if we open up our terminal here we can see in our orders application where we logged out the user on the request object so we can see that our auth actually worked properly and then our auth microservice validated the user as we'd expect and we can still see our billing application billing the user here so now that we have authentication working for a http request let's do the same exact thing for our rabbit and q service so guards can apply equally to rabbitmq services so if we open up our billing controller see our handle order created event here we can actually add another used guards here and apply the jwt auth guard and this will be coming from app.com so we'll add the jbt off guard here and now we can actually protect our rabbitmq routes to make sure that we have a json web token being passed to this route now just as we did in the orders application we'll go ahead and have to supply the rabbitmq auth queue to the billing application so that it can be used properly and as you'd expect we'll have to go ahead and import the auth module so we'll go ahead and add the auth module which we can import from auth common here so with this route protected with our jwt auth guard we can now see that if we send a request to create an order we can see that it still succeeds but now if we go look at our terminal we can see we have an exception being thrown in our billing application because no value was provided for authentication when we sent this event moreover if we actually resave the billing application we can see as it restarted it also has this error re-thrown over and over again and this is the same as we've seen before we weren't able to acknowledge that this uh event actually succeeded because an error is thrown in our jwt off guard so we are replaying the message over and over again every time our application restarts which is a good thing because this is a valid failure and we want to handle that so to make sure we're providing authentication to our billing service in our create order function in our order service we'll go ahead and add an additional parameter here called authentication which will be of type string and now when we emit this order created event we can also add a property here called authentication which will be equal to the off it gets passed in here so now in our orders controller we're going to go ahead and pass the request.cookies.authentication [Music] value because since this is http we should expect that cookie parser has added this request authentication gbt value to our cookies object now back in postman if we send a request we can see our whole flow completes successfully i'll send a request and we can see billing now correctly billing the user and additionally we can see that on the data or request object itself we have the authentication jwt value passed and as well as the user has been attached to the actual data object which is great to see so our jwt flow is working properly and we have protected our http and rpc calls so with that we've implemented a full working microservice architecture in sgs with jwt functionality using rabbitmq i hope this video has been helpful for you and if you'd like to see ones in the future make sure you subscribe and i'll see you in the next one

Channel: Michael Guay

Views: 64,912

Keywords:

Id: yuVVKB0EaOQ

Length: 84min 40sec (5080 seconds)

Published: Mon May 23 2022