How to build an Order / Message Queue System on AWS Step by Step | SQS, Lambda

[Music] what's up everyone welcome back to another episode of aws tutorial and today i'll show you how to build a message queue system on aws but before we get started let's talk about what exactly a message queue is and how people use it in the real world using a message queue is a pretty common practice for things like calling center or delivery orderings because it typically preserves the ordering of the messages so for example for calling center in a bank so every day there are a lot of people calling in to ask questions about a bank account or other things but you only have so many agents to take the phone calls so typically the call will enter a message kill system and then they'll place you on hold and the first person that calls in would get a phone call answer first and then the people who call later will get served later on and that's what the first in first out means and the same thing happens for restaurant delivery systems and today we're gonna build a message queue system for a pizza online ordering system so right now let's imagine that you are a small business owner that sells pizza to the whole town and because your business is booming you need to open two pizza shops instead of one to serve the whole town they have the exact same menu use the exact same ingredients and make the exact same pizza so whenever people from your town use your app to order the pizza online they don't really care about which shop the pizza is coming from because they taste the same so whenever they place the orders you'll just enter the message queue system and then your two pizza shops will take the order from the cable system and then process them make the pizza and then deliver the pizza to the customers based on the address and right now let's imagine that unfortunately today one of your pizza shops is broken down let's say the oven is broken in shop number two so that one cannot make pizza anymore and the beauty of using a message queue system is that it provides you a reliable system that can keep your flow going even though one of the shops is down because right now all the orders are taken by shop number one so you don't have to make any changes and your businesses can still operate the same way except it may be slower because you only have one shop to make pizza instead of two but the advantage is that you don't have to make any changes to the system in order to keep processing all the orders from the queue you just have to focus on shop number two to fix the oven and then bring it back online so that you can make your pizzas faster to to satisfy your customers but at the very minimum right now you can still have one shop that process all the orders and in today's tutorial we're gonna simulate this whole process this whole scenario that i just went through so without further ado let's get to it all right so right now i'm on the home page of the aws console and step number one is to create a message queue and we're going to use sqs on aws so i'm going to type in sqs click on it and then we're going to create a queue and now you're going to see that you have the option to choose which type of the queue you're going to use you have a standard and then you have fifo five will stands for first inverse out we're going to choose fifo because it guarantees the first in first out delivery and then and then it will ensure that each message will be processed once and only once for the standard one i think it will try to do the same thing but if there's no guarantee so we're going to choose the fifo queue instead and then we're going to give it a name let's call it pizza ordering q i think for fiverr it has to end with fifo at the end and then for the settings visibility timeout um so what the visibility timeout means is that once the message is picked up by a consumer this is the time that it will remain unvisible to other consumers so after this time and if the message is processed successfully by the consumer it will be deleted from the queue but if there's a failure in the consumer side when it process it you'll just go back to the queue and then other consumers can consume it so let's just do one minute why not and then message retention time so this is the time that the message will remain in the queue if no consumer picks it up and process it we don't need it to be four days maybe like uh one hour or i'll just do one day and then we can just keep everything as default encryption let's encrypt our message and then hit create queue alright so it's done and now we're ready to move on to the next step which is to create a lambda functions for our backend application and the two pizza shots but before we can do that we need to create an item roll for them to use so i'm gonna go to imro click on row and then create a new row that's just lambda next i'm gonna attach two policies to it one is the sqs full access and then the other one is the cloudwatch log for access and then hit next you can see that we have two policies here give it a name i'll just call it pizza delivery blender wrong hit create all right so it's done and now we're ready to create the lambda functions so i'm just going to go to lambda so the first one that we're going to create is the application backend lambda that takes in the orders and then we're going to create a pizza shop number one and pizza shop number two let's call it application back end we're going to use node.js and then permission we're going to choose the row that we just created i believe it's that one hit create and then let's go back to functions and create two more for the pizza shops pizza shop one same thing we're going to choose the row that we just created and then let's create another one okay so it's done before we make any changes to it i'm gonna make a few changes to the lambda functions so the first thing is the timeout because the default timeout for lambda functions is only three seconds i'm going to increase it to one minute because that's what we set as the visibility timeout in our sqs so i'm gonna click on configuration general configuration hit edit i just change it to one minute zero seconds hit save and then the second thing is the concurrency i'm going to change the reserve concurrency to be one just so that it will only process one batch of the message at a time otherwise one lambda or one shop will just take in all the messages all at once one and then lastly i'm going to add the sqs as a trigger to this lambda to this pizza shop i believe that is the one i'll keep the patch size as 10 and then hit add and then i'll make the same changes to pizza shop number one okay so now we're ready to move on to the next step which is to write the code simulate process let's do the application back end first i'm just going to write the code in the lambda console because it's pretty simple so the first thing we're going to do is we're going to import the aws module and then i set the region to be usb1 because the sqs is end is in us cs1 and then i'll land this also in usc's one and then i'm going to define sqs clients next we're going to define the url file as sqs and it goes like http s that's qs and then it's the region of the queue which is us is one let's see yes us is one and then.amazon.aws.com slash and then it's the account number and then lastly is the queue name so i'm just going to copy that whole thing all right and then we're going to define a couple constants one is the number of pizza shops we have two and then let's simulate the number of orders that we're going to have from the people in the town let's just do 100 and then we're going to define the function here we're first going to define a variable called order id we're gonna start from one zero zero zero and then we're gonna define an array called records that we're gonna have for the orders and then we have a for loop where we're gonna build the params for each of the order the first attribute we're gonna have is the message body remember that we have to stringify that because that is what the client is expecting we're going to have an attribute of order id in the message and then we're gonna have to audit detail i guess what pizza the customer is ordering i'm just gonna do it random number here uh from 0 to 20 right typically there may not be that many pizza types let's just do 10 and then just do the timestamp in terms of when this order is placed all right and then the next attribute is q uil which is where we define here and then next for five fifo cubes we need to specify a message to duke id we are just going to use the older id as the little id and it needs it to be in the string format or string type and then lastly we need to have a message group id because that's how we can have two consumers to consume the fifo cube messages otherwise it will just be processed one by one one at a time so we basically have two groups because we only have two pizza shops and we're gonna do it in a alternative way just to i mod number of pizza shops and that should do it and we're gonna finally add the param into each of the records that we're gonna process later on and then we have the order id increment by one and now we're gonna send all the records into our sqs and we're gonna do that in a for loop as well we're gonna do an await because we're gonna have a an async function send message if that's an error we're just gonna lock it out okay so that is it for the application back end and obviously we're just hard coding the number of orders and the order ids and all the details but in real life you're going to build this as an api and then you have the front end the code api to place the order depending on what the real user is using it or what they want with it i have a different video on how to build a serverless api i'll include link down below so you can check that out if you're interested so i hit deployed to save the changes and then now we can write out the code in the pizza shop to receive the message and process them and the same thing i'm just gonna edit it in the lambda console so the first thing we're gonna do is let's just cancel doc log out the event body just to see how it looks like and then we're gonna extract the order detail from the sqs message we're gonna parse event records and then we're gonna take the first one it's always gonna be just one message in the body and then we can just cancel dog log out what the order is and then we're gonna print out something like making pizza and then let's wait for five seconds because if not he's just gonna process everything in lightning speeds and we're going to define this function we're going to take the milliseconds um i believe we just do a promise i think and then reserve we're going to do set timeout resolve in milliseconds okay i think that should do it oh he's safe let me just add something like pizza shop number one making pizza hit save again i'll copy the same thing to pizza shop number two and change it to pizza shop number two hit deploy and now we are ready to test this whole thing out so what i'm going to do is i go back to the application backend lambda imagine right now people are using your app to order pizza from your shop so this is the api that handles all the orders and then when i hit test it's going to send all the orders to the queue system and then jump number one and jump number two would process them and make the pizza and do the delivery so i just do test i don't need this yes it means hit test it's an error ah timeout three seconds but some of them was sent successfully if we go back to the queue the message queue go back here we should be able to see some of the messages got sent successfully and some of them were being processed in flight means it's being processed let me fix the timeout and do the send again so do configuration one minute should be enough go back to the code his test again alright so they were all sent successfully now if i go back to the queue and some of them are waiting to be processed and some of them are being processed by the pizza shops right now and i think it's going to take a minute to finish because we pause each execution by five seconds and we have 100 messages to process so i'm gonna pause the video and then come back in about two minutes okay so it's been a few minutes i think it should be done processing all the messages right now so if i hit refresh it should be zero and zero right now yep so it's done so if i go to shop number one and number two they should they should both have the logs for the processing so you go to monitor view logs in cloud watch i'll do the same thing for shop number two so that logs here if i click on it i just see something like the order id the order detail and stuff and then pizza shop number one is making pizza and if we go to shop number two we should be able to see the logs as well and there you go you can see the order details and pizza shop number two is making pizza so both of the lambdas are processing the messages from the queue separately and they process different messages from the queue and as you can see here each of them take about five seconds to process because we put the lander to sleep for five seconds for each of the execution and i think the same thing happened here as well five seconds and now let's simulate the broken down of the shot number two so we can just go back to functions and simply just delete jump number two okay now we only have one shot being operational right now and we did not make any other changes and now let's go back to the application backend and we're going to hit test again it should send 100 orders to the message queue again all right so it's done and now if we go back to the sqs hit refresh we should be able to see some of the messages are being awaiting to be processed and some of them are being processed by consumer number one right now and i'm gonna pause the video again and then come back in maybe five minutes because we only have one shop processing the orders right now all right so it's been a few minutes um so we should be done processing all the messages so if i hit refresh all right so it's all done and now if we go back to pizza shop number one go back here and that is the new log so if we click on it we should see all the messages and each of them takes about five seconds to process and you can see the same thing here the record that looks like this and then the order detail that looks like that and it says pizza shop one is making pizza and as you can see even though pizza shop number two is gone pizza shop number one can still process all the messages without any problem or any modifications to the system and that's how you can have a reliable pizza ordering system and this is it everyone i hope you've learned something and if you like this video i hope you can give a thumbs up and i'll see you in the next video

Channel: Felix Yu

Views: 836

Keywords: aws, sqs, message queue, lambda, ordering system, message queue system, fifo queue, fifo queue multiple consumers

Id: SPyDoTkRolY

Length: 20min 20sec (1220 seconds)

Published: Sun Oct 03 2021