A Complete Guide On Nest JS Queue

in this video I'll cover everything you need to know about n gsq what it is how it works behind the scenes and a quick demo on how to set up and use it in your nest application but before we start let's quickly understand what a queue essentially is a queue is a data structure that follows the first in first out principle meaning that the first element added to the queue will be the first one to be removed think of it like a line of people waiting for a service where the person who arrives first is served first Nest provides a package Nest bull which is a rapper on top of bull bull is a popular high performance no gs-based Q system implementation now bull is built on top of redis and uses it to possess job data so in order to use bull we need to have red installed on our system bull Q works on the producer consumer pattern producer adds job to the C Q with specified payload and optional config the job is serialized and stored in Red's database making it available for processing jobs are processed by worker processes these workers can run on the same server or be distributed across multiple servers when a worker starts it connects to the radi server and begins listening for jobs on the specified queue when a job becomes available the worker dqes it process the job payload and executes the specified processing logic if a job fails during processing you can configure bull to handle R rise with back off strategies fail jobs are moved to a designated failed queue and you can inspect and retry them as needed that was all about bull q and how it works next let's see a quick demo on how to use it in our application now in order to use bull CU we need to have ready install on our system now either we can install it locally or we can use a Docker image now for this tutorial I'll use the docker image so head over to Docker if you have installed Docker that's pretty good if not you can download it for Windows or Mac then head over to command terminal and just type doer hyphen V if it is installed you'll get the version now once you have installed Docker you can install the radius image so for that you can just type Docker pull and the uh image name which is radius it installed the radius image on your Docker desktop now for my case it is already installed so that's why it pulled immediately but if it is not installed on your system it will take some time now once we have R let's open the docker desktop now we can run the container uh via command line as well but also we can do it via Docker desktop so I'll head over to desktop and just I'll just click run now we have optional settings we can Define the container name I'll name it as redis and for ports default is 6379 so we don't have to change it I can just click run so our radius image is running now since we have setup red on our system we can focus on our Nest application here I've created a sample Nest app and added a user module with a route to allow users to upload files on S3 now in controller you might have noticed that I haven't awaited this call to use US service because if in case user uploaded a bigger sized file and I don't want user to wait for the response until the file is uploaded to S3 instead I'll immediately send 200 with a success message and image upload can happen in background so far everything is fine then why do we need bull Q here the reason is simple file upload is a resource intensive task which means that it will take your server resources like CPU and memory in order to process and upload file on S3 or any other hosting service but a single request won't make any effect consider getting thousands of concurrent requests where users are uploading multiple files of let's say 10 MB each all these requests will asynchronously handled in the background which will eventually need more resources and hence your server will shut down when limit exceeds in order to resolve this issue we can add this request to Bull Q bull will store these shops in redish which will be dced and processed by worker processes this way instead of processing the file upload in current request response cycle we'll handle it asynchronously now let's add bull to our application let's first install the required bull packages so we'll install the bull package and it's rapper for sjs once the packages are installed we can head over to app module to register the bull module config now in ports we can add the bull module and press enter this will Auto Import the bull module from njs bull then we can add the for root method this for root accepts and config now if I quickly head over to Nest document documentation we can see that these are the properties that is accepted by this an object now we can see if we want to configure the radius connection we can use the radius property let's register that so you can just type radius and it accepts an object that has host and Port Now by default Port is 6379 for host since we are running redes on Docker we have to define the IP of the docker container so to get the IP we can open the command terminal and just WR Docker PS this will list all the running containers now we can see that our red container ID is this one we can just copy this and type Docker inspect and can paste our container ID now we'll get this object all we need to care about is this IP address we can copy this IP address and this is our host we can just paste it here and we have configured our redis connection now once we have configured our Red Connection let's proceed to register our Q head over to user module and add Imports property next let's add pool module and import the register que method this method accepts object as an argument let's add the name property and Define the name of the queue let's name it file upload now optionally if we want to register multiple cues we can add a comma and add another object and we can need name it something else that's another Quee that's how we can register multiple cues but for this tutorial we don't want to uh register multiple cues so I'll remove it so this register Q method is used to instantiate or register a queue so as we learned previously that bul uses radius to purist job data so whenever your app is started or restarted bull will check if there are any unfinished jobs in the previous session for this queue if yes then it will executes those jobs and that's it we have registered our Q now if you have multiple radius instances and you have registered multiple cues and you want that your different CU should use the different radius instance let's head over to app module and register a new config for this case I'll provide a different red instance let's say my different instance is pointing is running at this IP now we can see that there's no difference in these two configs so for the alternate config we can Define a name let's say I'll name it alternate config now if you see that by default our Q is using the default config now if we want that our Q should point to this alternate config we can add another property called config key and we can add our config name which is alternate config and now our Q is pointing to the alternate config next let's add a job to our queue in my user service I'll add a Constructor now in order to inject Q I'll use the inject Q decorator which is imported from Nest bu package now this decorator accepts the Q name so I'll pass the Q name here which is file upload and then I'll provide the variable with which we'll access the queue this will be of typee Q which will be imported from bull package now next instead of directly calling upload to S3 I'll add this job to the queue so in order to add that I'll use my file upload variable and then call the add method now this add method accepts an object now this object could be anything we can use it to Define our payload structure which in this case will be this one so I'll just copy and paste it now since this is is an object I need to provide the keys as well now we can remove this if you see the return type of add method it's it returns a promise so we can await this call now we have added our first job to the queue now optionally we can also name our job so before the data we can pass a string to name our job let's say this job is to upload images now whenever we create a named job we need to create a processor as well to process this job otherwise bull will complain that there is no processor to process this name job now jobs do support additional options as well so after we have provided the data add a comma and add another object to provide the options if you press control space or command space on Mac we'll get to see all the available properties so we'll have attempts back off delay and tons of other properties as well let's use delay for our case and let's add 3,000 milliseconds so what this means is that after a delay of 3 seconds this job is available to be processed by worker processes next let's quickly check if the job is added to the que so after this line I'll add a console log let's build our application so I'll type too build hyphen T to define the tag and Define the name of our image which is Nest app let's say and use dot to include all the files now once the image is built we can run it in a container to do that we can run Docker run hyphen P to define the port which is 3,000 it should map to the 3,000 Port hyphen D to run it detach mode and then name of our image which is Nest app and this will run our image in a container so if I open do dock a desktop and open containers our app is running now if I open Postman and hit this end point I'll get status as 200 with a message now if I again open DOA desktop we can get the log the job is added to the queue next we can add a consumer class and worker processes to process these jobs let's open vs code again and add a new consumer class in the same user directory I'll add a new file called user. process. I'll export a new class and name it as user file upload now to Define this class as a consumer class we need to decorate it with processor decorator which is important from The Nest bull package and we need to provide the name of our queue which is file upload now in the consumer class we can define a method let's say handle file upload now to Define it as a workup process we need to decorate it with process decorator and optionally we can Define the name of a job which which this worker process will listen to so previously we created a name job called upload image so this worker process will listen for this job also this method now will accept an argument called job which is of type job which is imported from Bull Pack package now for now let's console log this job what we have in the job. DAT now before we hit the endpoint we need to register this consumer class as a provider so that Nest bull package can pick them up so in user module in providers I can addit this user file upload let's import it let's build the image again let's hit the Endo got this job is added to the queue and also we got the data now the reason we got the data previously as well because we added the job two times so those jobs got executed next let's update our Handler method now this method should call the pl to S3 function let's import that so I will call uplo to S3 that will Auto Import now we need these four parameters that we can get from this job data let's extract those and that's it we are done with our worker process this decorated method is called whenever the worker is idle and there are jobs to process in the queue this enter method receives the job object as its own argument the value return by the Handler method is stored in the job object and can be handled later on for example in a listener for the completed event next let's explore the event listeners event listeners must be declared within a consumer class bull generates a set of useful events when a queue or job State changes occur now here I have defined two events on Q active which will be called when the job is in progress and on Q complete completed which will be called when the job is completed both of these events will be imported from The Nest bull package now if I head over to Nest documentation we can see the list of all the available local and Global event listeners now let's run our application and see this in action I'll build the image first and then run it in in a container I'll hit the endpoint again and then let's see the logs so the job was added to the queue and after 3 seconds that the delay we added the job is processed by the worker process and we got our logs so we got the log when the job was in progress and when it was completed and there you have it I hope this video clarified any doubts you had if you enjoyed this video and want more content like this make sure to hit that like button and subscribe for upcoming videos on web development I'll see you in the next video

Channel: Programming with Lakshya

Views: 3,370

Keywords:

Id: ohVmmS0nLIw

Length: 16min 10sec (970 seconds)

Published: Mon Jan 15 2024