AWS Lambda : load JSON file from S3 and put in dynamodb

hi welcome to lambda with Python tutorial series in this video I am I'm going to show you how to process a JSON file the requirement is like this we get JSON files into s3 object and there should be an automated process which takes adjacent file and it stores that information into DynamoDB this is the overall requirement we're going to implement in this video let's get started you first let us have a DynamoDB table have already created one table with name employees for this employee the primary key is employee ID and remaining data can be anything next let's go and create history bucket where we are going to place this JSON object you I've previously created a bucket with named Jerome Jason this is a bucket where we are going to place JSON files and it's automatically processed by lambda function next we need to set up lambda so before setting of lambda let's go and prepare I am role for this lambda this lambda requires access to s3 bucket it has to read data from s3 and it has to place that information into DynamoDB so I am role for this lambda should contain permissions to s3 and it also should contain permissions to DynamoDB Plesk it also needs to contain permissions to AWS cloud watch locks because all the logs it's all the locks it publishes to cloud watch lock so this particular lambda function must have access to s3 access to DynamoDB and also access to cloud was locks let's go ahead and create I am policy and roll you first let's go and create a policy create policy to service you one services cloud watch logs select actions to keep the example simple I am giving complete access for that resources again to keep it little simple I am giving it as all resources we need additional permissions we give permissions for logs CloudWatch logs we do permissions for s3 here let's go and choose s3 actions compute actions I am giving their resources so ideally we need to pick a specific resource like we need to pick a specific s3 bucket but anyways this a demo I am giving access to all resources here I need one more permission which is going to be dynamodb right and again I am selecting all actions here coming to resources and I am saying all resources review the policy name it you you you and create the policy now create a roll I want this role for my lambda function so in a SS service ready to go and select lambda next go and set the permissions we just created a policy select that policy and attach this policy to that role your name I am given again same and create a road yeah we are done with a creation of a role and creation of a policy the next step is let's go and create a lambda function you select lambda create a function let's create a function from scratch name it run time is going to be Python 3.6 roll I am going to choose an existing roll here yeah that's a role we created now create a function the function is created for us go to the editor and start start writing the code for that so this function is going to be called when and JSON file is operated to s3 so in this function we need access to three and dynamodb let's import yes three and DynamoDB here first minute import border three you which is Python module for AWS SDK from bata tree so we'll get client of yes three you you okay from using this s3 client we need to get data from that si object first let's check what what data we are getting in this event object accordingly let's go and implement our logic you so print it so wish not directly predict event object we have to convert this event object into string and print it we have function string STR and to that function passes even object so first we we are going to check what this even object contains and based on that structure from that event I am going to take the file name and I am going to take the object sorry the s3 bucket name and accordingly we are going to build a logic let's first save it let's also add a trigger so then adjacent file is uploaded into s3 it has to trigger this lambda function so from this trigger trigger section select s3 a configure disaster a trigger the bucket is going to be Jerome Jason this is the one we created and even type is going to be object created whenever object created in this bucket that has to trigger the event for us I don't I don't want to give any prefix and suffix let's keep it as dot Jason so whenever there is a file with dot jason extension uploaded into this s3 bucket it's going to trigger this lambda function so add it save it now let's upload object into this s3 I have an object with me on the desktop you yeah this this is JSON file object which contains some data in it this is what I am going to I am going to upload into sa bucket let's go and do that so add files so on the desktop I have data dot Jason open that file and upload so this triggers our lambda function the file got uploaded so this would have triggered the lambda function to see the output let's go to cloud watch logs and check for the data so under this logs right under cloud watch locks they should be a log group created with our lambda function name yes three jason DynamoDB open that now there's entry let's open this also yeah say this one so this is what we are getting from that evil object so from this event object we need to take the bucket name plus we also need to take the file key such that we can download that object and process that file and store that information into dynamo dB content properly let's go on check some online Jason viewers you place this data here format it but this is the actual object so from actual object we have records that's the main property in this object it is an array so in this case it's going to be one single entry let me show it in viewer it will be nice too it's nicely formatted see the main object the main properties records in this we have only one record so under records we have something like event version source along with that we have the property called sas3 under that we have a bucket under this may have a name and likewise we have one more property polis object it has two object it has a file name there so from this object we need to pick the key of the file and we also need to create the bucket name so let's go and implement the logic according to that yeah so open your lambda function you bucket name equals in this case right even isn't dictionary and from that dictionary dictionary is nothing but again JSON here so from the dictionary the main object is records you okay so under these records we have it is say array so under this array we need to pick the first array in that you zero so inside this array inside this array we have yes three inside this st object we have bucket you inside bucket we have name you so this returns the bucket name from this event object let me cross work once again they have records so under records we are picking the first entry in that array that's zero so from the first entry we are picking up yes three s3 so from s3 we are taking the property called as bucket from bucket they're picking up this name which returns the bucket name for us likewise we also need to get the file name now you you you so almost the process is similar so from event because from records first element in that you so from first element it's s3 from s3 it is object and key you you you you let's also try to print it and let's make sure we are getting the right information and let's proceed with the further logic print those values here so print bucket print Jason file name save the function go back to your Esther bucket delete this object reupload it uploaded this must have triggered a lambda function let's go and check logs here okay let's see there is one more entry created here let's pick up that one yes looks we are getting the right information so this is the bucket name and this is a file name so we are getting that bucket name and file name from this email object the next logic is take data from this s3 bucket and correct to dynamodb and place this object into dynamodb let's import just a client so use both oh three and call client and get the reference of s3 here you you so let's see a function on this s3 client which gets object from s3 let's open both the three documentation for s3 you open this first link such for a function that this is how we got the client search for a function here we have get object here so it takes bucket name and the key so we don't need to pass as information here so use this client now we can take out this information and take out this because we call it to God the logic to get bucket name and file name there so client dot get the object adopted bucket close bucket lane here and pee you equals the file name you this returns back JSON object for us we got response from this s3 get object so this object contains the body we need to pick up the body from this object so let's check the document we are calling this cat object we're passing bucket name key and let's see the response response type is dictionary and it's giving some JSON object with lot of information from this JSON object we have to pick up the body so in this property we have the JSON file actually okay so from this you on this object we have to call readmitted this is a streaming body in Python so from the streaming body way to get this reader okay so from this we need to convert it into a dictionary name this one else Jason filereader okay so from this we need to use one more module here which is Jason module or Jason so jason has a function called as loads so pass this reader it converts this reader into a file for us so finally we converted this JSON file into this dictionary so we can pass this dictionary to DynamoDB table and purchase it let's import DynamoDB client here so from DynamoDB they have to get a table the table name the table MSE employees reference that with table so on this table we call you item let's go and check document for DynamoDB bata three damn you yeah so we got dynamodb resource here just a moment it has to be resource not client and from this resource by getting a table by passing the name using this table name you can say put item good item this is a function we are calling here and I need to pass the JSON data otherwise the item which is a record in dynamic dynamodb table to this item so come back here you so that information is in this dictionary see this is nothing but this object where we are downloading a file from s3 okay it's a response from yesterday from that response we are getting the body from that body we are trying to get the reader and using this Jason loads we are converting this we are reading data from this file and converting it as a dictionary and we are passing the dictionary as an item by calling this table output item so I think this is a logic let's go and execute this function and see the output let's say this function plot the JSON file here advils session data loaded okay let's go and check your watch logs here you let's go and check the DynamoDB table yeah and file content is inserted into this DynamoDB let me show the file here see employee ID is three-straight the age is 26 name is hurry and location is USA so that's all about this demo thanks for watching

Channel: Java Home Cloud

Views: 75,336

Rating: 4.929481 out of 5

Keywords: dynamodb, aws, boto3, json, s3, load JSON file from S3 and put in dynamodb, lambda tutorial, aws lambda tutorial, aws lambda process json data, java home, javahome

Id: Y18HF5ALXew

Length: 23min 12sec (1392 seconds)

Published: Sat Feb 03 2018