How to create Queues using Redis and Node.js | FIFO, LIFO, Delayed, Prioritized and Repeatable Jobs

hello guys and welcome to another video today we are going to learn how to create a q system using redis and node.js so for this video we are going to use this library called bool that basically is an implementation of queues directly with various commands so all the heavy work is done by this library and we just connect to their api and the idea of this project is that at the end we can have a q system where we can send emails to users and the reason why i choose to create this type of project is because first is a common use case on uh basically a lot of web development projects and also because i think it's a really easy implementation and if this is your first time creating queues i think is going to really help you understand at least the basics and then you can later create your own cues but before we start diving into the code i would like to show you guys what are we going to be using and what do you need for this project so if we take a look at the package json here we have an express the express dependency and the reason is because we actually have in our entry file an express app and we already define an endpoint and this endpoint is the one that we are going to use to send the emails so basically uh you need that then we also need boot and bull board bool is this library you can go ahead and see that in their github page how to install it you can use npn or john and also if you're using typescript you can install you're gonna need to install also the types pool and and this pool board is just a front-end application to inspect the cues and you can see they have different ones the task for is uh an official uh library from this guys that created the bull library as well but i'm using bullboard because it's free with transfer you need to pay i think i haven't tried yet but i i saw there that you need to pay for that but bulwark is a great library i have used this for production and i think it's great so you'll need to install that and also you will need node mailer i guess you probably already know what dom mailer is but if not basically it allows us to send emails so you need that as well perfect so also one thing that you guys need is to have a redis server running in your computer uh for this particular project i'm using talker and the reason is because i'm using windows and i had a lot of troubles using the windows version in here so i decided to use docker it solves the problem for me you can use docker as well or if you're using a linux system you can just install radius without a problem uh also one thing that i might be doing in later tutorials is to use docker for all of the projects because i think it's easier for you guys just to download the project and run the project if you have docker so yeah that's a great thing about docker so now that we have that ready we are going to run the project so for that we use compose up if you're using docker if not you need to start redis and also you need to start your backend so in this case we are going to run our application import 5000 and now the application is running perfect okay guys so now it is time to start actually working with queues and see how all of this process goes so creating a simple queue is actually easy we just need to basically use this line of code and we will have a q system but there is a couple of things that we need to pass there as well that they are not mentioning here so to create use i like to have a folder called qs and because we are going to create an email queue so i'm just going to name that email.cue that yes okay so we need to import first bool from bool and now we define our email queue so we create a new email queue is equal to new queue at this case is bull and we pass as the first argument the name of the queue in this case i'm just gonna name this email and then as you can see we have options here so these options later we are going to see that we have a lot of a lot of options inside our queue that we can pass but the most important one is to connect to already server so in order to do that we pass here readys and radius can be passed either with an object we pass the hosts the password and the port but in this project i'm using an url so we can just use the url inside here so let's take a look at the docker compose here we are passing a custom environment variable called redis url this reddish url is just connected to this radius instance that we have here so we can use process.emb dot redis url and with this we will have our queue created okay so one thing to know about queues is that we have three main different roles we have a job producer a job consumer and we also have events listener but events is something that we're not gonna work in this project maybe later i will create a video just for this type of uh for the events okay so a queue can have many producers many consumers and many listeners but for the email queue we are just going to work with one for purdue one producer one consumer and that's it so a producer is just a way to add new jobs into a queue you can think uh the producer is adding a new item an array with data so later our consumer can execute the function or can execute a function or process a heavy work based on that data okay so then we have listeners just really quick to understand what a listener is are events that happen in our queue so for example we have a completed event in here uh that gets executed once this consumer is done with one job uh we can also have an error we can have a progress and i think there are more events that i'm not sure right now but let's take a look at this jobs live cycle to see how everything works so we have the jobs the job that gets added our producer adds a job so it gets on this state then it will move to the weight state right this delay state is another type of job that we're going to see later and how we can add it but for now let's just see that when we add a job it gets added to the weight stay if we have more jobs getting executed uh it's going to wait when it's free we are going to move to the active and once the job is done we either move to complete completed or failed if it is fail and we have a retry logic or something we are going to how to retie this job but if not the job is going to be finished and that's pretty much uh all all it is on this we have the producer that creates this and then we have the consumer and then we have the events on this part okay so now let's start creating or producer so to create a producer we are going to create that inside our email queue so we're going to have a function called say new email and here we receive data that can be of type any okay so inside here as you can see we call our queue and the add function so mlq the add and we pass data and then we also have some options in okay and the reason this is the actually the reason why i'm using a function to create this producer for this producer because i want to have uh some options that i don't want to be passing every time i call this producer so i'm just going to have a function that have everything and we pass just custom data on this function and adds a new job into the queue but for now we're just going to leave it like that later we're going to see the options in here okay so now let's go ahead and create our process to add the consumer so the way that i like to do it is creating also a new folder called processes and in here because this is an email so email for the email queue i'm going to create an email.process.cs and this is just a function as you can see our process is just a function with the job argument so we are going to define the email process and this takes a job and this job is of type job from bull in this job basically it has the data that we are going to receive from here so when we add a job this data gets added and we can read that data from the job.data i think that makes sense so for now i don't want to do anything so i'm just going to print console.log i'm going to paint hello or actually i'm going to print the data of the data and this needs to be async so now that we have our process there we can pretty much export this or export default email process great so now it is time to add our email process here so we do that using our name of the queue process and here we pass our email process a is coming from this folder over here or file sorry and perfect now we can export default or actually we can export the same new email and with this we will be able to add cues into uh or add jobs into our queue and execute something in this case we are just doing a console log it doesn't make sense but later we are going to see how we can send the email from here so uh if you guys remember here we have this freddy's commander thing that basically will allow us to see the data that we have inside the redis application so let's just go ahead and do localhost and this is in the port 8081 okay let's refresh we can see that we have the radius we don't have anything yet because we need to actually execute the or call our producer in order to see something here so let's close everything let's go into our app and inside here i want to pass the same new email right that it is coming from email queue here we pass the data in this case breakdown body so we get the the body from the request and we do a wait right so hey this is async okay so with this we are adding uh a new job and we are going to console.log something so we should be able to see the console log here and we also should be able to see also the data in the queue so for this i actually have the endpoint ready to execute here on postman so we are calling the uh the localhost 500 we have 5 000 sorry and we are calling our endpoint and here in the body we're going to pass this data but basically later we're going to use season emails but for now we should be able to see this data uh in the queue so let's just click send we get the status okay we get the console log from our producer and if we refresh we can see that we have this email queue we have a completed state with the value of one so this is uh the job that we were trying to execute as you can see the data that we passed is this one over here is our requested body and we only made one attempt right now let's just try to fail on this on this uh on this process so we can just throw an error something bad happen okay so we wait until our server restarts okay so now we have our server there let's just click again send and we don't see any console log okay we refresh now we see that the queue that we just added so this number two we get uh the reason why it fails we also have the data that we pass in here the temps that we made uh and we also have that it would it is in the failed state so now if we take a look at the job life cycle this one the first one went directly to the completed event in here and the second one went to the fail event now because we don't have an auto retie uh system basically uh we just fail and we finish but we can pass a retry system and doing that is actually really simple let's just go into our email queue here where we are creating our or adding new jobs we call the okay let me see it is not auto completing for me okay so it's not working because it is attempts okay so now we have that and we can at least do two attempts for this email queue okay we wait until restart okay so now we click send again all right so we can refresh and now you can see that we made two attempts to do this job but at the end it still failed so it failed uh number three and number two great great now it is time for us to actually do something with the process so we are going to send an email so we can see how everything works and see how it actually helps using cues okay so remember we have node mailer here is the one that it is actually going to be doing everything in this part so let's take a look at the old mailer documentation really quick and literally we are going to copy everything that we have in here i don't see a reason why not so first thing is to actually require this we are not going to use require we are going to use import so we import from our mailer okay okay so now we have non-mailer there and inside here we are going to copy this uh entire function so okay copy and paste that in here we add some format okay so really simple let's see what we are doing in here first of all we are creating a test account how we this is an option that we have for example when we are doing this type of testing so this is great but you can connect your gmail account if you want or any smtp server or whatever you want to use in here but uh we just create the transport in here we pass the user the password and in here this is the part where we need to change so remember that in our endpoint we have this uh information already in here we have for from or to our subject the text is uh the plain text body but we are not going to be using that and then we have our html but in this case i don't want to use html because i don't want to pass html in here i just want to pass them a simple message but we're going to change that later so in here we can just get our job data because our job.data is going to have that information in here now let's just make this a little bit bigger so we can see better so we are printing the message id and then we are trying to preview uh the email because the only thing that we can do here is preview the email we cannot actually uh receive an email from from this test account so for this we are not going to do it like that we are going to return the actual uh message url so we can see it later in our queue in our radius commander ui so they save this okay so now that we have this there is also one thing that i want to add here because if not we are going to get errors so reject and authorize things yeah i'm going to pause i pass false in here great okay so now let's uh go into our endpoint here and it's an email and here i'm gonna pass basically this request.body but for the html part i wanna do a custom message that says hello uh or not hello i'm gonna pass the body the message i'm gonna close that okay uh or actually i can do this uh i can do the message and i can do first body press up body so we actually don't send a message there right so with this we will be passing this data here we will be receiving that data in here to send the actual email so here instead of having html we have message right so now let's try and send this let's receive the status okay as you can see it was really quick we didn't have to wait for that email to be sent and we also cut the console log of this message and now we have number four we have that uh it was completed uh as you can see number four was completed the data that we received and we the return value because we are returning here url we can open this and see how it looks okay so welcome to your new account you can see the subject here we have the the email that it was it was being sent from and the receiving so i think that matches this data that we send here um too and the subject and the message great now let's try something with the attempts here okay let me make this screen here okay perfect we can close this also not mailer because we have pretty much all we need so uh with the retries okay so as you can see here we are making two attempts and i actually want to make five attempts right i'm sending this email but i'm going to throw errors on purpose okay so let's just here do if the job that attempts mate is less than that you say 2 i want to throw a new error this is something or server is down right if not i wanna continue send the email so here we're going to see that we made two attempts and the third one was same so let's just save this okay so now our app is running we can click send again now we get the status okay and we are going to get errors probably okay so yeah okay so we got uh two attempts that we made uh then we had a failed in this job that server was down and then we got the result at the end it works at the end perfect okay guys so now we can pretty much delete this error there and now we like to go and connect our board dashboard so we can see the queues in an interface uh so doing uh the connection is really simple uh remember we need to have the board library here installed and inside our email queue we just need to require the set queues and the bool adapter okay so we do it like this import from a bull board okay we need set use and we also need the bull adapter so probably after creating the queue we can do set queues here we pass an array new adapter and we pass the a or q with this we just added our cues into the dashboard and we just need to create the endpoint to access uh to the dashboard so as you can see here we just call router from bullboard and we use uh we passed adding inside our application express application so we can do uh yeah import on google board and here we pass router right and maybe here we can use our admin use and we pass our router so now we save this we wait until our server restarts okay so now server is running again now okay before we do it let's just go ahead and see if this is actually working is inside admin use okay so now we can see let me just make this full screen we can see that we have our latest queues that we run we have our completed queues with the return value the data that we send options that we have for the uh for the our job and then we also have the failed jobs uh here we can see that the error that we get the data that we pass options when we creating the job and we can either retie this queue as you can see here we can delete it we can clean everything retry all uh here we can clean all so this is great when we are actually working with cues and let me just try sending uh or sending on yeah creating a new a new job so let's click send so as you can see it's gonna be moved to the active and when once it is done it's going to move to the completed state as you can see this was the last email that we sent here we can see the date of the time and everything so yeah this is perfect okay guys so now let's take a look at some of the options that we have uh with bull as well so in this case we can have a red limiter for our cues doing that is really simple when we are uh creating our q we just have these options here uh when we compare the limiter with a maximum of jobs per i in unit of time so in this case will be something like limiter here and we pass just a max in this case 1000 and the duration could be 5 seconds all right so you can play with this it's an option that we have you also have named job so in this case we can name our jobs in here so for example when we have multiple jobs and multiple also workers we can define as you can see here uh the name of the of the job and in here they browse the name of the job in the process so in this case this is optimus coder view as you can see we can do pretty much any type of work here we're transcoding files so that would be adding the name here for example we can pass this email it doesn't make sense but it's an option that we have now it gets the data moved to the right and then we have also our options here but this is something that you can play with as well now moving on we also have some we have different job types so in this case we can change the configuration here to make the job to be instead of first aim first out the order uh and the job gets executed to be last in first out as you can see we can pass that uh here in the options uh just next to the attempts we can pass for example here that's in first out you can really pass through so that's an option that we have now the delay uh job that we have we can add a delay to the jobs in this case i think that is super simple it gets delay five seconds this queue right so for example we just passed the delay option here and if we add for example five seconds okay let's wait to the server to restart and we take a look here we have a top here called delayed and if you guys remember in our life cycle like when the job gets added it will move to the delayed state here so by just adding the delay here our app is running there so let's just click send and if we take a look at the delay type we should be able to see this is delay for less than 10 seconds and once that time pass we move to the active and then move to the completed um tab here so that's one type of the uh of job that we have then we also have priority that we can pass in here same thing we just passed this option here i already i don't know why it's not autocompleting for me or you can password it is on this right that's one thing that we have and then we also have uh repeatable jobs and these ones uh you can use it for different type of use cases that you might have but in this case we can pass the repeat option here for our job so in this case let's just say that we pass repeat and we have every right so every 10 seconds and be a maximum of a hundred times so every 10 seconds we repeat this this job you can use it to certain reports or something to users but i guess for reports you will use something like a chrome job where you pass instead of passing the every unlimited option you pass a crown specification you can pass chrome and you can pass the chrome specifications uh here we we get that the job repeats once every day at 3 15 a.m so they added here let me just go back the guys from bull added here a tool to see how chrome jobs work so you can basically just go ahead and you want to play with this thing and pass the specific value here so you can just pass that and the job will repeat uh uh we'll repeat based on that specification um here there are some considerations that you need to have but to stop or remove a room a repeatable job you just pass this method over here okay guys thank you so much for watching this video i hope you had learned something new uh if you enjoyed the video i would really appreciate a like to the video a subscription as well that will really help me i love on growing the channel also any questions that you might have let me know as well i'm gonna leave a link in the description for this project so you can download this for free also uh the links for each one of the libraries that we use uh yeah thank you so much again for watching and see you guys in the next one

Channel: Coding Venue

Views: 14,740

Keywords: rabbitmq, kafka, celery, jobs, ack, bull js, bull.js, redis queue, bull, background job, redis queue commands, redis queue delay, redis queue dashboard, redis queue cluster, redis queue fifo, redis for queue, redis queue gui, redis queue node js, redis queue monitoring, redis workers, email queue, queue email, send email queue, redis docker, priority queue, delayed queue, repeatable queue

Id: b7DJEAJZsG0

Length: 33min 51sec (2031 seconds)

Published: Tue Mar 23 2021

Reddit Comments

[u/cjrutherford]

Be sure to clean up your cues or expect reddish to just crash if you're in a resource strict environment.

[u/Plexicle]

Very cool— love Bull and Redis.

Obligatory: you don’t need body-parser any more. It’s been built in to express for years now.