AWS DynamoDB Streams to Lambda Tutorial in Python | Step by Step Guide

alright guys welcome back to AWS simplified the place where I teach you everything there is to know about AWS and today we're going to be talking about DynamoDB streams and we're gonna do a step by step guide and i'm gonna show you how to detect change on a dynamodb table using a lambda function so let's get started so firstly let's do a quick architecture overview so we're going to start with a DynamoDB table and I'm gonna name this table game events and the tables content is going to look like this so we're gonna have a player ID we're gonna have a dates we're gonna have a game round ID and a score and the primary key is gonna be the player ID and the range key is going to be the date and from there we're gonna add a lambda function and I'm going to call it the game events process or lambda function and I'm going to code it up so that it can detect change from the dynamodb table and this is what the input is going to look like so this is what a change event looks like so there's a records array and each of these objects here is a change event so just taking a quick glance at this so there's an insert here so this corresponds to an insert event and if you recall from last time there's three types of events the insert modify and remove so if we look a little bit further here you can see in the DynamoDB in the new image section we have all the corresponding attributes of the newly inserted row that corresponds to this event so you see the player ID was this value the date was this value the game round ID was this value and the score was this value okay so then from there we need to actually attach some permissions to our lambda function so that I can interact with the game events table so we're going to go to I am and we're gonna create a new role and I'm going to call this role the dynamodb stream lambda role and the role needs a couple permissions that are associated with it so the four main ones are these four here so described stream get records get chard iterator endless streams these are the absolute minimum that you need in order for this to hook up and work correctly if you don't add these things aren't going to work and for me I want to do some and do some testing to make sure this is actually working correctly so I'm adding some extra permissions for cloud watch so create log group rate log stream and put log events after we create the policy and associated with this role we're gonna attach that role to the lambda function so the lambda now has all these permissions associated with it from there we're gonna go ahead and hook up the trigger and the trigger type that I'm gonna set up for this exercise is the new and old images trigger and if you forget what this is please take a look at my old video where I explain this in detail but basically it shows us what the before and what the after images of the change look like and then from there I'm gonna come along and start poking the database and triggering some of these change events so we're gonna do three types of change events the first one's going to be the insert so I'm going to be inserting this row into the table I'm going to verify that we we handle that insert successfully and then I'm going to be doing a modify and in the modify case I'm going to be altering the score so I'm going to be modifying the score to this new value here 175 and then finally I'm going to be removing the row and making sure that we get the delete event so those are the three types of operations that we're doing and this is what it's going to look like now let's head over to the console and get started alright guys here we are in the console page so the first thing I said I was going to do is create the Donny 1tb table so let's go ahead and do that so I'm gonna come over here type in dynamodb I'm going to select create table and I set my table name is going to be called game events and my partition key is going to be player ID and my sort key is going to be date if you don't know what these mean I'm gonna be doing a future video describing the details about dynamodb so I look forward to that so I'm gonna go ahead and click create and this is actually going to take a few moments to successfully create so I'll come back once it's all done already so that took about a minute or so but we can see here that we now have successfully created our game events table okay so we're going to go to I am because we need to create a role for a lambda function in order to access the DynamoDB streams so I'm going over to I am and I'm clicking on roles and we're going to create a role so just a reminder we need a couple dynamodb stream permissions and we also need some cloud watch permissions so I'm clicking on create role and choose the service that will use this role so it's a lambda function and we're going to say next permissions and there's actually a pre-built policy so I typed in DynamoDB and yes so this one AWS lamda DynamoDB execution role so if we click on this guy to expand it out and scroll down we can see that these this role has the exact permissions that we need so described stream get records get shard iterator with streams and then this stuff for cloud launch so this is exactly what we need one thing I forgot to add here is that you need to click the checkbox beside the policy name and then click the attach policies button before proceeding to the click Next at the bottom so make sure you do that before proceeding and I'm gonna click Next I don't care about tags so I'm just gonna skip that and role name so I'm gonna call this gdb stream lambda role and description the boilerplate is fine so I'm going to click on create a role alright cool so now our role is created so let's head over to lambda now and we can actually create the function and associate the role with it so going to lambda and clicking on create function and we're going to do this one from scratch so I'm going to call this one the game events processor and from there we're going to be using Python 3.6 for this exercise and here's where we select the role that we just created and I do not want to create a role I want to use an existing role and this should have our role and there it is the dynamodb stream lambda role that I is created and we're gonna click on create function and this will take actually was very quick this time so our lambda function is now created right so it has everything that it needs so we need to code up our lambda function now so let's go ahead and switch to sublime text where I'm gonna be doing that alright folks so I switched over to sublime text and we have two panes here so on the left hand side I'm going to be writing the Python code that's going to be the core of the lambda function and on the right hand side I just have some references so I so I have three different references I have an insert event reference so this is what an actual insert event will look like on a murder on our DynamoDB table I have a modify event that will look like this and you can see there's a little distinction here so we have a new image and an old image so this is the what's there now and this is the reference of what was there before that's the old image content and in the remove events all you have is the old image because there's nothing new anymore okay so let's scroll back up here and let's start coding this up so the first thing I did just has a quick little reminder I just wrapped the entire function in a try-catch it's very important if you take a look in my previous video I explain why it's important but make sure to do this if you're creating a lambda function and it's gonna be hooked up to DynamoDB streams okay so let's start going here so the first thing we need to do is we need to iterate over every record right because there could be multiple records in the array so iterates over each record and I'm going to say for record in events records now here's where we handle the event type so I'm just gonna say and all events say so we're just gonna code up three quick functions one for the insert one for the modify and one for the remove so let's just write a couple of statements here record events name equals inserts that I can handle I'm going to write this function afterwards we're gonna do the same for modify and then we're going to have modify Handler and you should probably be else ifs and we're going to do the remove and then once the invocation is done I'm just going to print a delimiter to help me out and actually going to do that at the top too so we can figure out where we're starting and where we're ending and also in the exception case as well all right cool so now we need to actually code up these three handler functions so the handle insert handle modify handle to remove so let's go down here and actually start doing that so we're gonna define handle inserts and it's going to take a record and let's just say we are handling the inserts event ok so there's three sub steps here the first one is we need to get the content from the new image so if we go over here we look at the inserts so there's a new image key so we need to grab all this stuff from then from the new image key and then we need to parse out the value so it's kind of got some awkward I like that I'm going to be notation so you have the key here and then you have están noting that it's a string type that's in comparison to n where it corresponds to number and then we need to actually grab the value all right and then we'll just print it out so three steps let me just say 3a so get image contents three B will be parsed the values and three C is just print it out okay so let's go through this one by one so first thing is get the new image so we're going to say new image equals record DynamoDB and then new capital and new image so now we have a reference to this object over here now that we have that we can actually parse the value out so I'm just gonna be parsing out the player ID here for this exercise so let's say new player ID is equal to new image so we need to grab the player ID and then you can see the S key here too it's actually grabbed the value so we say s and so now we have a reference to the player ID now we just need to print it out and let's just say new row added with player ID equal to the player ID right and finally let's just wrap it up down here and say done handling inserts that's cool all right so that that's good for our insert so let's do the modify now so let's just scroll down here so we can take a look at the modify event so as a reminder you have the new image and the old image in the modify event so the new image corresponds to what is there now what is the most recent thing and the old image is the reference object so what has changed so you can see here in this case the old image contains a score that's 175 and the new image contains a score that's 150 so that's telling us that you know the score changed here so that's exactly what I'm gonna try to capture with this function so we need to do handle underscore remove this should actually say handle underscore modify but I come back to correct it later and then record scroll down make this a bit easier and from there we so we have three sub steps again so 3a is parse old image and score and then 3b since it's a modify event we need to parse the new image and its score and then we'll do like a quick little comparison to say something changed check for change and I'm only going to be checking for change on the score field but in your function you may want to do something a little bit more comprehensive and similar to the other one let's just say and lean modifying events and then at the bottom done handling modify events all right and so in this sky we are going to what did I say parse out the old image in the score so let's grab the old image and so we need the old image key so it's case sensitive so be careful and let's grab the old score now it's equal to old image and then we need score so going back over here so we need score and then the key is end because it's a number type so then we put it in all right cool so now we have the old score and let's do the same thing for the new image so new image equal to record DynamoDB new image and then new score it's going to be equal to new image score and this is exactly like the old score and then we're going to write a quick little if statement to check for change so let's do that so we'll say if old score is not equal new score and print something prints scores changed and let's show the old score versus the new score so the old score is equal to string old score and then we'll say the new score is equal to string new score right so now if there's a change event and it's X change here we're gonna print the old and the new and then finally let's move on to the remove excuse me this should actually have been handle modify okay so this step is gonna take similar to the first one except that we're not gonna have an old image we're only gonna have a new image sorry excuse me it's the reverse so we're only gonna have an old image and we're not gonna have a new image so let's do the same kind of boilerplate that we had before so handling remove events and then done handling events and so three steps here so 3a is parceled image like we did before 3b it's going to be parsed to the values and then 3c is just print it out okay so let's grab the old image record DynamoDB old image case-sensitive cool and then we're gonna grab the player ID out of that so we'll play rindy old image and the player ID and then s because it is a string type and I'm just gonna print out hey this thing was removed something like that so let's actually save row removed with the player ID equal to old player ID all right so hopefully this kind of gives you some insight into how these functions actually work and how to kind of parse out the values so from here I'm going to copy this function and I'm gonna go back into the console and paste it into our lambda function so let's go back over there now alright folks so we're back in the lambda console so I'm going to remove the existing code that was given to us by default and I'm going to paste in the code that we just wrote and just for information I'm going to be making this code available for you so if you want to try this yourself it's gonna be below in the description section in the video so you can hook this stuff up all on your own so after it's pasted in I'm gonna click save all right so now it's saved so the next step is to add the DynamoDB to lamda trigger so you can do this in one of two ways you can add the trigger over here from the lamda perspective and this will work perfectly however I prefer to do it from the dynamodb side so there's absolutely no difference I'm just showing you one way so dynamodb go to the tables and then our game events table and then triggers going to create a trigger existing function and then here's our processor going to leave the batch size as a hundred this is by default and we're gonna enable the trigger right away one thing to keep in mind is that if you don't have the right permissions for your lambda function you'll get an error as soon as you click create so this is kind of a good way to test whether or not you have the right permissions attached to the role that you've associated with the lambda function we're gonna click create and you can see that mine work perfectly so let's go over to the item section and we're gonna create an item and I have something prepared to add in so this is what I was originally going to add if I come over here and we change this to text there's a neat trick where you can just paste in a JSON and it'll kind of follow your it'll use your keys as the column names and we're gonna click Save so again as a reminder that should have triggered a insert event and now I'm gonna change the score to 175 change that to 175 click on save so that should have triggered a modify event on our lamda and then the last step is to do the remove so deletes are you sure you want to delete yes I am sure and again that should have triggered the remove event so now let's go over to lamda and see if there are any metrics to indicate that this was successfully invoked going to monitoring and there's our invocation it's weighted one in a vacation and this may be kind of confusing because we we did three separate operations but the way lamda processing works on Dinah would be streams is that it may group together multiple change events into one lambda invocation and that's what the batch size that I was talking about earlier refers to if you set that really high then you can group together many many events and when a vacation if you said it really low than you group together only you know so few events in one indication so let's go over to cloud watch now and verify that the metrics are actually there and that we have actual logs going to logs going to game event processor going to the stream all right cool so it looks like let me change this to text so it's easier to read okay cool so let's see what we got here so the first step was printing the event so this was the event this line here and then handling insert event and then cool so new role added with player ID equals this awesome and then it was done handling and then it went to the modify event which is down here and then it worked perfectly so I was able to detect the old score was 150 and the new score is 175 and then down through the remove event so we can see that the rows are moved with player ID blah blah blah so it looks like things worked here pretty seamlessly and thanks so much for watching and if you have any topics you'd like me to cover drop me a message below in the comment section and don't forget to Like and subscribe thanks so much folks and I'll see you next time

Channel: Be A Better Dev

Views: 27,707

Keywords: AWS DynamoDB Streams, DynamoDB, DynamoDB Stream, DynamoDB Streams, DynamoDB Stream Lambda, dynamodb streams python, walkthrough, dynamodb streams lambda, dynamodb stream lambda, dynamodb old image, dynamodb new image, aws dynamodb stream, aws dynamodb streams, aws dynamodb stream python, aws dynamodb streams python, dynamo lambda, dynamo stream lambda, dynamodb trigger, dynamodb, aws, amazon web services, dynamodb trigger lambda, serverless, serverless nodejs, aws developer

Id: RhLUyJxS8Tk

Length: 21min 53sec (1313 seconds)

Published: Sun Aug 18 2019