Step functions for long running tasks

do you want to learn how you can set up long running tasks with Amazon's step functions in this video that is exactly what i'ma teach you how to do hi guys my name's Sam with complete coding but our aim is to make you into the best developer that you can be in this video we're going to be looking at Amazon step functions where you can create long running tasks which act like flow diagrams to provide different functionality and experience to different customers this can be really powerful as it allows you to do things like adding delays of hours or days before you do tasks or it can allow you to do complex logic like checking attributes or checking if a customer has say checked out their basket after 24 hours you can do loads of things with this and I'm gonna introduce you and show you how you can set up an example flow just like one of those two starts out creating our service step functions we're actually going to jump into our browser and have a look inside draw dot IO in here I've just made a little bit of a diagram to help us work out what's going to happen inside this step function the first thing was going to happen is we're going to get an API hit with a sign up we're going to take this sign up which is going to have an ID and it's going to have an email address and it's going to add that to dynamo dB we're then going to wait for 24 hours and in those 24 hours the user could either play some games in which case we will add a play field to this record or they could not do anything that's all in which case the record will be the same after that 24 hours we're going to read that same database row and check how they played any games if they have that perfect we don't need to do anything else so we're just going to close the step function if they haven't played we want to send them a reminder email reminding them that they've signed up but haven't played any games yet and once that is sent we are going to close this is a relatively simple step function and these can get a lot more complicated but it's definitely not a trivial one so it's a really good place to start to learn using step functions so now that we know how we're going to lay out our step function what we're going to do is go into our AWS account and start setting this up so if you go into your AWS console and search for step functions you'll get onto this page we can click in the top left here and click on state machines and at the moment we have no state machines and no step functions we want to create a step machine and we're going to start authoring with code snippets we have the option to do an express one or a standard one and we're going to go with standard for now and then finally we get down to our generate code snippet this is where we are going to be at all we're going to use to generate the template that we're going to be putting into our service configuration at the moment it has a comment and a start at it has two states one that says hello and one says world and you can see that represented in this diagram what we're going to do is change this diagram to represent the diagram that we have already have so to do that what we need to do is go into our generate code snippets and what we need to do is find out what our first task is our first task is to take the email from the database and add it to the database sorry so what we're going to do is generate a code snippet and we're going to use a lambda function to add that data to a database if you click on invoke function you then get some config here as you can see on the right hand side there is some Jason that we can copy this points at a resource which is lambda it has a function name which is the a RN and then it has a payload so what we're going to do is copy all of this go into our code snippet and inside States it is an object and we're actually can delete the first two that came with us and paste what we've just taken if you click format Jason it just nests it very nicely and then we can change some things in the snippet we've just pasted so the first thing we want to do if we look on our diagram is add email to DB so let's call this add email to DB and we can copy that and we want that to be the start of our process so that is our first process here where it's going to invoke a lambda we don't know the function name yet so we'll have to config inside service and that will in putting everything once that has finished we're then going to call the next state and that's what we need to do next so after we have added the emails the database we need to wait for 24 hours unfortunately there isn't a code snippet for waiting so what we need to do is do that ourselves and we're going to create a state called wait for 24 hours and inside this it has a type of unsurprisingly wait we have a duration which is seconds and as we are waiting for 24 hours if you open up a calculator and do 24 hours times 60 seconds times by 60 minutes 8 6 400 as well as the seconds we need to define what happens next and the next thing to happen after we've waited for 24 hours is we're going to read the database so read deep DB so got an error here saying that next state doesn't exist on the lam dream vacation that's because we need to add wait for 24 hours as the next state here now that we've waited for 24 hours we need to read this database entry what we can do with this is actually just copy the ad email today space invocation and paste it there we can change the name to read DV and after we've read the DB we need to go to a choice called has played as you can guess we now need to make that has played stage so here we can actually generate a code snippet and if you scroll all the way to the bottom there is a choice state this is quite complicated but we're gonna do is copy it'll paste it in and then delete the things that we don't need so we need to check that something equals something else so to do that we're not checking a numerical equals and we're not using any ands or anything like that so we can delete all of those so here we have a variable of dot type string equals private and what we want to change that to is change it to dots payload dot played and we want it to equal the string of true we also don't want this not a statement so we can delete that like that and hit format jason if we add that comma and delete that comma there we go so now the choice if the choice of payload got played equals true we have one option that we can go to and that will be closed and if that is not true we have the default of send reminder email the last thing we need to in this choice is update the title to has played which is going to resolve that issue we now have the clothes and the send reminder email to create so we're going to start with the send reminder email this again is going to be another lambda so if we go into our templates invoke a lambda and copy this to the clipboard we can now change the name from invoke lambda function to send email reminder we don't have the function name just yet but once this is finished we're also going to go to close and finally we need to find this close state so close is an object and inside that object we have a type and that type is a string of s you see ceed succeed so that just means that this function this step function has succeeded and to close the step function if we now hit format we make sure this is all in the correct format we can change a little bit start with a comment and change it from this to something like a step function to send reminder emails after 24 hours we now have all of this code and we need to move it into our service project to do that you can select all of the code but at the moment this is in Jason and all of our code is in Amal so we need to go into our browser and search for a converter to convert Jason to llamo there's loads of these online open up one of them you can paste in your JSON file and you get all of your yanil out we now can go back into our code and we can now store that yanil somewhere we're going to create a new file called step function dots Yamma and paste all of our code in here oh one sec copy all of that and paste that in here there we go and we can delete that comment at the start and there we go we've now got all of our code in here we just need to update a couple of things which is the function names so to reference a function name we can do a couple of different things the way we're going to do it is we're going to use some cloud formation function joins which is going to join a couple of attributes substituting in things like the account ID and the function name so this is FN colon colon join and this is an array the first parameter in the array is what you're going to be using to join the object the strings so we're going to be joining them with a colon the next parameter in the array is all of the things that you're going to be joining so that's another array and we're going to start with ar n : aw s : lambda : e you - west - 1 and if you see this is very similar to this part here so AR n aw s lambda and our region is EU west 1 the next thing that we need to add into this is a reference to our account ID and the way we can do that is with an object which has a key of ref and a value of aw s : : account ID so that is going to be substituted for the accounts ID that we want and finally we need to have the function name at the end so that is going to be another string of function : and then our function name our function name is going to be self : service - self : provider dot state - function name so for this first function this is the add email to DB we're going to call this the function of ad signup this may all look really complicated but it's a way of turning variables such as the account ID into a string that will look something like that the good thing is once we've written it once we can copy the whole row like that and use it everywhere else that we use a function so when we read the database we can replace this and the only thing we need to change is instead of it being add sign up it's gonna be get sign up and finally the last one at the bottom paste that instead of it being add sign up it's going to be send reminder email now that we have all of that we can go all the way back to the top and we've completed this step function now we need to go into our server list channel and do a bit of setup so that this all works the first thing that we need to do is go down to our plugins and add a new plug-in that plug-in is going to be service - step - functions and we're also going to copy that go into our terminal and install it so NPM I minus D to install it as a dev dependency and paste that in there so that is the first step to getting these service step functions configured as well as that we need to go into our code and add step functions and this is part of this service step functions plug-in we can define these step functions which is what we have done here and we need to add a little bit more configuration to this so in here servlet step functions we want to say that we want to validate it first so what that is true we want to then have an object containing different step functions or state machines state machines and at the moment we have one which is this reminder email step function so inside this reminder email we need to define a couple of properties first being the name and I'm going to copy the reminder email now we need to define how we want this to be triggered so events and this is very similar to how we do it with lambdas so HTTP to create an API endpoint it's going to have a path of sign up with reminders the method is going to be a post and we're gonna say yes to cause so cause equals true so that's going to set up a new endpoint that allows us to trigger this step function with an API request and finally we have the definition and it's this definition is what we've written inside are the step function so to add that to our service Yama file we can actually just reference it and the way that we do that is we add a dollar curly braces and we're referencing here a file that file is on same directory aval and is called step function dot y Amal and that will make sure that the definition is the one that we have stored in that file this is a really nice way of splitting up your code so that not all of it is in a single one file which is often easier to read the last thing that we need to do in here is actually create a new dynamo database so that we can store these user signups to start that we're going to add a new thing into our table name so we have the original table name and we're going to add a new one this is going to be called something like signup table name and we can call it something as simple as sign up table name and actually I'm gonna switch these four using hyphens to be in accordance with the one above us we can now copy this variable and we want to be able to access it from our lambdas so in our environment variables we can say that signup table name equals self : custom dots sign up the table name so that now is pointing at the same value as this and the last thing to do is scroll down to the bottom of our code to where we have our resources we have a bucket and a dynamo table and we're going to create the another table so we can copy all of this down to our billing mode and paste it in below this needs to be just formatted just right so make sure that that my table name aligns up so we need to change this from my DB table name to my email signup table the table name instead of being self dot customer table name is now self dot custom signing up table name and we can get rid of this one attribute and save that format this all nicely and there we go this is now a new dynamo table that we're going to create when we next do a deployment so if we jump back into our code and look at our diagram we now have the signup it will save it to a dynamo DB by calling a lambda and we now need to make that lambda so that it does that functionality inside our code if we open up the lambdas at the moment we have common code we have endpoints but these aren't going to be endpoints these aren't api's that we're calling they're just lambdas so we're going to create a new folder inside lambdas and call it step function lambdas in here we can create a new file and this new file is going to be called add sign up dot J s this actually relates to if we go all the way to here this add sign up that we had inside our function this add sign up is relatively simple so the first thing we need to do is import Dynamo from common slash dynamo we need to get the table name so Const table name equals process dot n dot signup table name because that is an environment variable that we set up here and now we need to create our handler so exports dot handler equals an async function which takes as all lambdas do takes an event inside this event we're going to console.log it out just for clarity then we're going to get the ID or get the email off the signup so const ID sorry const email equals event dot input dot sign up dot email now that we have the email we can go down and we need to create a UUID no way now that we have the email we need to create an ID so if Const ID equals and we're going to generate an ID using UUID UUID is an NPM package so we have to install it so npm install - d - save it to our dev dependencies actually we need to save this to our live ones so it's minus X and then youyou I D at the top of our code we also need to import that so import and in here we're going to be important the v4 version as UUID and we're importing that from you you ID like that so this is going to generate us a brand new ID now we can do a weight dynamo dot right and we need to write some data which is going to be email and ID and we're gonna set played to be false this is because as soon as a player creates an account they haven't played anything as soon as they do we'll have a different lambda which will change this records to be played equals true but that's really from video as well as that we need the table name just like that and once this is succeeded we need to return an object with just the ID this means that when we do a get later on we already know what ID we're looking for this is great and it's set up so now we need to go into our service yeah Mille and scroll down to our functions to set this one up so at the bottom of our functions we're going to copy the first two lines of any other function and paste them in here format them so they're in the right place and here we can change send email to Adam sign up and this adds sign up here needs to match up with the function name you have at the end of your step function configuration so it's add sign up here and it's bad sign up here our handler isn't going to be endpoints slash send email is going to be lambdas slash step function lambdas slash add sign up dot handler so that's the first of our three functions to make so we need to go into our step function lunders again create a new file and this one is going to be the get sign up file we can actually copy the whole of the add sign up as that gives us a good place to start we can console log out the event and now this time we're gonna say that ID we're gonna get from event dot input dot payload dot ID as when we do our ads sign up we fire this payload off so we can access it here to get the ID we can now say that the row equals dynamo dot get and with a get we can get rid of all of this because get just takes an ID and a table thing so we can set that up and finally we want to return the row which is the row that we just got which may or may not contain played equals true or false as we've got rid of generating an ID we can also get rid of that row up there just to save a little bit of code now we've got the get sign up again go into service copy this Handler and we're going to change it from add sign up to get sign up we've now got two of our three functions done we've done add sign up we've done any get sign up our choice will be handled by our step function code so now we just need to create the send email lambda so create a new file called send reminder dot J s and now go into our step function code go down to the bottom and I've actually called it send reminder email here so we're going to change this file do we call the send reminder email is now with this code it's going to be a little bit different to the ones we've got before so I'm going to start with our handler so exports dot handler equals me sync event just like that we're going to get our email from the payload so Const email equals event dot input dot payload dot email and when you're doing this it's often easy to start by adding a console log of the event so that when you get to trying to work out how you get parameters off you know exactly how they're going to come through now that we have an email we need to have a message and that message is going to be pretty simple it's going to say hi we saw that you signed up for our gaming platform but haven't played yet we hope you play soon so that's a very simple message you could obviously make this a lot more comprehensive but that's all we need for now now that we have that message on the email we can actually take some code from one of the endpoints we've created before which is the send email what we can take is we can take the parameters and the try catch and go back to our send reminders and paste all of that in so in our parameters we're sending it to an address and for us that is going to be the email that we're sending it to the body has a text data field which we can set to the message that we've just written the subject we can hard code to remember to use the gaming platform and from so you meet need to have set up Amazon SES before and the way you can check that is if you go into your Amazon console go to AWS and search for s e s it's a simple email service you can click on email addresses and you have an email address just there if it says it's verified then you can use it but if you click on here and there's no email addresses or it says that it's not verified then you can verify a new email address this is just something that Amazon of ism has implemented to stop people from creating spam email addresses and sending out lots of junk email once your email has been verified make sure it's copy it and we can then go back into our code and use that as our source so that is the email address we're going to be sending from we need to then make a little change to this try-catch because it's not an API we don't have responses so we can just return but if there's been an error we don't want to return we want to actually throw the error like that the last thing we need to do is define this SES so at the top of your file do import star as AWS from the AWS SDK and then in const SES equals a new version of the AWS SES class like that and that will allow us to send our SES emails the last thing we need to do now is set up our send reminder email inside service so again copy the gets signup format it and then change it here and here and these need to be set to the send reminder email and that those three parameters all need to line up with the parameters you used at the end of these join functions inside your step functions we can now go into our terminal and run SLS deploy and that will compile all of this and build our new step function now that has finished deploying we can see that we have this extra bit of output at the bottom which is our service step 4 action outputs here we have a URL so what I'm going to do is I'm going to copy this head over throughout console to a postman instance and paste in our URL need to change this to a post request and then set up a jason body in this body we need to have a sign-up object and in that object we need to have an email and the email that we're going to go with is going to be the same one that you've used inside your server your sending email so if you go into your send reminders that email they're so copied up and you have to use this for now until you've got your account verified to send to any email address if you want to find out how to do that check out the video I linked up in the top of this page and that will tell you how to get your account verified for sending emails now that we have the email added in we can hit Send and at the bottom we get an execution a RN and a start date this is exactly what we expect from a trigger of a step function we can now go into our console and go into step functions again in here we've got our reminder email and if we scroll down we can see that we have on the 10th of May at 12:56 a record if we click in here we can see that it has all succeeded it has played it sent a reminder email and then it's closed if I actually go into my emails it's a bit bigger I can see I've got a reminder to use the game platform it's there and there we go that is the email that we have automatically sent out zero minutes ago so we've just received it this is really powerful and if we did this again but quickly went into dynamo which is what I'll do so if I go into able us go into dynamo DB and look inside our tables we see arts sign up a table and it will have a couple of emails in here for make this a little bit bigger and then go back into our terminal into our postman instance sorry if I clear all of these alts so there's nothing in there at the moment I hit Send quickly go into here refresh this change that to true and hit save when we go into our step function management we go back to our reminder email there'll be a new one which is running and at the moment is waiting so obviously it's not gonna wait for 24 hours because we've sent it to wait for 20 seconds it's then read the DB and what you can do is if you go into this you can actually see the outputs and you can see that because we changed played equals true it goes into this logic and it can say that played is true so it doesn't send our reminder email this is really powerful as you can do loads of logic you can have processes that run for long times and you can separate out the logic inside your code in this video we have used Amazon's step functions to build up a flow where a customer can sign up it hits a step function API which then adds them to a database we can then wait for up to 24 hours and then after that time we check if they've been back and they've played a game on our system if they have that's brilliant they're already using our system but if they haven't we can then go down a different route and actually send them a reminder email to go and make sure they use the platform that just signed up for this has loads of applications and you could use it for anything from a designing your own email funnel to reminders for when a customer signs up for something or even something completely irrelevant to customers such as data processing you could have any time a shipment of your product arrives wait a certain period of time before updating some data in your back-end systems if you've learned something new in this video make sure to give it a like as it really helps the channel by making YouTube more aware of it and suggesting it to more developers just like yourselves and if you haven't already please make sure to subscribe down here and turn on the Bell notification so that you can get notified when our next release a video the next video is going to be on API authorizers so hopefully I'll see you in that video

Channel: Complete Coding

Views: 4,100

Rating: 5 out of 5

Keywords:

Id: 0fDjptXVcsc

Length: 46min 31sec (2791 seconds)

Published: Sun May 10 2020