Automate AWS DynamoDB Scheduled Backups using Lambda, Serverless Framework & GitHub Actions (CI/CD)

hey everyone i hope you're all staying happy and well and thanks for tuning in to another tutorial today we'll be going over how to create an automated scheduled service that performs a backup of your dynamodb tables using aws lambda the serverless framework and github actions to handle the deployment pipeline as always if you find any of these videos helpful please make sure to subscribe as it really helps out the channel an incredible amount but for now let's get started now the first thing we want to do is head over to our im management console and we'll begin to create a new user and this user will generate programmatic access keys so that we can feed them into our git of actions deployment pipeline and this allows us to deploy our service to the aws cloud so we're just going to name this user here and we're going to call this dynodb schedule backup user now it's a bit long but it's okay and we're gonna check off programmatic access hit permissions and then we're gonna click attach existing policies and we're just gonna search up the following i am full access so we're going to check that off we also need amazon s3 full access most of these are just um are required for the serverless framework to actually perform the deployment um and some of these are actually for our application itself um but anyways the next one we need is cloudwatch logs um full access so we've checked that off we also need cloudwhich events and this is for our application because um we'll be using oh well while the serverless firmware behind the scenes will be using cloudwatch events to handle the scheduling of our lambda function so we'll check that off and of course we also need cloud formation as this will help our application to be deployed and last but not least we need aws lambda um yeah i ate over slam the full access and that should be all so we're just gonna quickly uh skip ahead and check that we have everything um and looks good so we can go ahead and click create user and awesome so once again these are this is the only time we'll be able to see the secret access key so we're going to make note of that we're going to copy it we're going to head over to our repo here we're going to click into settings we're also going to click into secrets and we're going to add a new repository secret so this is a secret access key so we're just going to paste that in really quickly we're going to generate a new repository secret and this is going to be the access key id and we're just going to copy that here and we're going to paste that in as well and awesome so now our repository is set up for github actions now we can actually go ahead and head over to our code and write up the deployment file for github actions so we're going to head over to vs code here and i've actually already pre-written um the configuration files so as to save us some time i go over a bit i go over it a bit more in detail my other videos so feel free to check those out as well but basically the idea is that we're going to create a folder in our root directory called dot github and within that folder we're also going to create another folder called the workflows and then finally in this folder we're going to create a workflow file called main.yaml and i'm naming it main because it will be only triggered on the main branch and we'll see how to configure that in a second so this is what i have pre-written already but basically just to quickly go over it this is the name of the workflow file this section tells us that we this job will only be triggered when we push to the main branch and of course here this is our list of jobs we only have one job here called deploy and basically here we're checking out into a repository um we're using node 12.x here um i believe this should be 14.x but we'll leave it like that um and then we install some dependencies and then last but not least we actually run the serverless github action which allows us to ultimately deploy our application to the aws cloud using the serverless framework and as you can see here this is where um we actually retrieve and use the um the secret access keys that we have just uh copied into the repository so awesome that's pretty much it for setting up the github configuration file now let's move on to setting up the serverless.yaml file and to do so we're going to start off by creating a new file in our root directory called serverless.yaml and we're going to start off by naming the service so the first thing we want to specify is of course uh under the service field and this is being this is going to be our service name i'm just going to name this uh dynamo schedule backup feel free to name it however you wish and the next thing i want to specify is the provider and this is going to be the cloud provider in this case we're going to be using aws so we'll specify it as such and the runtime we'll be using today is node.js 14.x which is currently the latest version supported by aws lambda i believe and next we're going to specify a development stage since we're just in development i'm just going to keep this as dev and we also want to specify a region um generally you want this to be the region that is closest to the clients that you are serving but since this is ultimately a backend application it doesn't really matter um but i guess it would make more sense to keep as a region as your dynamodb table to ensure low latency but we're just going to use us west one here um and the next part is pretty important um we are actually going to specify a global set of i am role statements and this we only need one which basically allows us our lambda function to have permission to create the dyno db backup so basically we're going to this the syntax is as follows we're going to name the effect and we're going to say that the effect is that we're going to allow the following action and the action will be um dynamodb because this is the resource we're accessing and the um the operation itself is going to be called create backup um and yeah that's pretty much it and then of course we also want to specify the resource that we actually want to um enable this action for and um generally you want this to be the arn or the amazon resource name of the actual dynamodb table so as to make it very specific and granular so your security permissions are really restrictive which is generally good practice and it will follow this format all the arns for dynamodb tables will follow this format so everything up to here is the same region will just be whatever region you specify here or whatever region you're dying with db table is and of course this is your aws account id and table stays the same and then this will be your table name and if you notice earlier my table name is actually important data um just for the sake of naming it anyway but for this tutorial um just to keep things simple and quick i'm just going to use the wild card which matches all resources and that's just represented by a single asterisk um yeah please don't make note of the good convention of or best practice of using the actual arn for your dynamite db table um but yeah that's that's pretty much it for the provider section and the next section we can actually do is write up the function section which will allow us to initialize our actual lambda function so we're just going to call this backup db here this is not the name of the actual lambda handler so that's fine we'll see in a second how to name that but the next thing we want to specify is actually where this function is going to be located so i'm going to put this under the source folder and i'm going to call the file backup.js later on and our function name will be handler so i'm going to name it as such and of course we're going to write specify a name for the actual lambda function itself that will appear on the aws lambda console i'm just going to call this backup um oops backup dynamodb and of course you can also include a description i'll include that for the code source code mentioned uh in the description below but i'll leave it out for now for the sake of time and also we want to specify the memory size this is a really simple job i don't of course if your database table is really large you might need more memory and perhaps a larger timeout time but because my table is pretty much empty it only has a few things entries in it i'm going to keep it as the minimal option possible which is 128 megabytes and of course we also need a timeout and this is just going to be set to five seconds i don't think we we'll need this but let's just keep it like that and next but not least we also need to specify the event triggers and this is going to be under the events field and because today we're going to be using the cloudwatch events um schedule trigger we're going to reference this as the schedule field and there are two options that are provided by the serverless framework which is really awesome and one option is to use the rate option or we can specify the schedule under a cron expression and um for the rate format it's going to follow this format where um you're going to specify the value and then the unit um as such um so for example um we if you want to have it to run like once a day it's going to be something like this rate one day and there's a small detail that if it becomes plural for example two days then we need to make the units plural as well um and of course just as a as an example for completion um if you want to write a cron expression so that it runs for example at um perhaps like 5 p.m every day then it's going to be something like um like this i believe and of course please do make note that this is all going to be in utc time as per lambda specifications um so definitely make note of that um but yeah for the sake of simplicity and for demoing um this application i'm just going to use this as perhaps wait three minutes um just so i have some time and i'll be able to show you that the application actually worked so we'll leave it like that but definitely feel free to change the schedule to however however frequent you need it to be and yeah that's actually pretty much it for most of the specifications the last thing that we need is just to specify an environment variable for our lambda function and we're just going to call this uh dynamodb table oops dynamo uh yeah sure download tv uh table name i guess or demo db table name um anyway yeah but yeah as i mentioned earlier my table name is just important data so i'm just going to hard code it here and awesome so that is pretty much it as you can see the serverless configuration file is pretty short but this will allow us to get the job done to not only deploy our lambda function but also attach an event trigger that is basically scheduled right now for every three minutes and and yeah so now we can actually move on to writing our lambda function and before we begin we're just going to quickly install all the necessary dependencies i'll be using today and i'm going to be using node and yarn here so i'm going to do yarn add and the dependencies are really simple just two of them and that includes the aws sdk as well as dot and and this helps us access environment variables in a node um so we'll just give it a second to install awesome so both of them have been installed and we're going to close our terminal and now we're going to go ahead and in our root directory we're going to create a folder called source and we're going to create our file as such i believe i've named it back up here so we're gonna create backup.js and now we can begin writing so the first thing we wanna do is actually import the um aws sdk all right awesome and then we're gonna initialize a new dynamodb object so we can access the um aws sdk function specific for dynamodb um i'm not sure why it's not highlighting it for me but it should be like this awesome okay anyway uh and next but not least we want to actually export our um handler function and this is going to be asynchronous function normally we would take an event trigger but because this is just going to be a scheduled job and there's not nothing too complicated here we don't actually need to um pass in the event argument here at all so we'll leave it empty as that and next but not least we also want to uh write a try catch block um for error handling and inside the try block we're going to actually um create a new date object and this is going to be how we um basically um name the backup um just so because we're running once a day we want to um sort of be able to uniquely identify each of our backups and um the way we're going to use this gonna is that we're gonna pass it into a parameters object that will then pass it into our function call um to create the backup so we're gonna first start this by creating a new params object and the first thing we need to specify is the backup name and so this is just going to be a string interpolated javascript string and we're going to use the um we're going to get the full year first and then preceded by our dash and then date.getmonth and we're going to get the month as well and because javascript months are zero-based index we need to add one um to make sure it's accurate and then we're gonna put a dash as well and then we're just gonna write the date awesome and put a comma and the next field that we need is actually the table name um so we know which diamond db table we want um because this is we're just handling one table at a time for now um we're just going to access this in our environment variable as we have declared earlier in our serverless.yaml file and i believe i named it download db table name awesome okay now we're gonna put a semicolon and uh now we can finally call uh make the function call to perform the backup and we'll do it as um or just name the result on backup res um i guess we can log this for debugging purposes as well but we're going to make sure we await on this and then we're going to call dynamodb um dot uh create backup and then we're going to pass in the parameters and of course we need to wait on the promise so we're gonna do uh dot promise and that is pretty much it now um the last thing i guess we can do is just log um the backup res to see what the result actually looks like and of course we need to implement the cache block uh we can just quickly include a um a log of the error message um but feel free to um implement your own um retry uh mechanism or whatnot depending on your use case or how important you want this backup to be um but for now we're just gonna log the error and nothing else and of course um to conclude this we're just going to return from the lambda function and as you can see it's really simple it's not much code but this will basically do the job but for now we're going to quickly deploy this and we'll be back to see if it actually worked and so we're back and as you can see all the steps in the deployment pipeline have succeeded so now we can actually head over to cloudwatch logs and under the cloudwatch console on the left side we can click log groups and if you just click refresh you should be able to see the log group created for your lambda function so we're going to go ahead and click into this and i've actually allowed this to run for a few minutes now so we should see a couple of logs and yeah awesome okay so it looks like it's been running for at least nine minutes and so we should expect to see three backups created successfully uh one at 11 38 um another one at 41 and 44. so we're going to head over to our dyno db console and we're going to click into backups and evidently we can see three different backups all created uh today um i guess the naming convention is a bit poorly chosen here but if you were to actually run this once a day then the naming convention here would work um so if we look at the time more specifically we can see um they're all run at or they're created at the um correct times 38 41 and 44. and there you have it that pretty much wraps up this video about creating a scheduled service that is automated that helps you back up your dynamodb tables using the serverless framework and aws lambda as always if you learned something or found this video helpful please do consider hitting that like and subscribe button down below as it really helps out the channel and series and as always i hope you're all staying happy and well out there but for now i'll see you in the next video

Channel: Jackson Yuan

Views: 1,262

Keywords: serverless, aws, aws lambda, aws dynamodb, dynamodb, amazon dynamodb, node.js, backup, coding, cloud computing

Id: v2TrrBr1OEY

Length: 15min 9sec (909 seconds)

Published: Sun May 02 2021