AWS Lambda Get CSV from S3 put to Dynamodb | AWS Lambda | AWS Lambda CSV

as part of this demonstration we will see how to trigger lambda function whenever a CSV file is uploaded to s3 and persist that information to dynamo dB for this demonstration I have created one sample CSV file I named it employees dot CSV I have three employee data in this file I am going to upload this file to s3 and that's going to be processed by lambda function and persisted into DynamoDB table we are right now in AWS management console let's navigate to DynamoDB create a table for persisting our information I'm naming this table employees it's going to have one primary key its name is ID type string you can skip remaining details with default values the next step is let's go and create a lambda function I am opening that in a separate tab create function so let's choose create otherwise author from scratch you I'm naming csv underscore s3 and score DynamoDB runtime is going to be Python for role I have on existing admin role created for lambda function let me use that in real-time scenarios we need to create a specific I am role and policy and give restrict to otherwise specific access to lambda function so he'd great function that's right cold let me maximize this window I do not need Jason module I need both three module using Berto three let me get the client for s3 I'm storing s3 client object into yes three underscore client variable with help of yes three client I need to get the file which is uploaded to s3 so I'm going to use get object method that will show this information in both the tree documentation this is the method it takes lot of arguments but we are interested in passing bucket and the key so bucket the next value is key now the point is from where I should get the bucket name and the file name which is nothing but the key so usually when we upload CSV file to s3 we're going to configure an event which triggers this lambda the information about the CSV file uploaded to s3 will be part of this event object before I go and extract bucket name and key from this object we also should understand its structure you replace lambda yes free event so let's go to a SS documentation which helps us to understand the event object structure we are going to get part of lambda function we have main object in that records which is an array so from event so when it access records from records in this case we're going to process only one record at a time I'm hard coding that information to zero after that I am navigating to s3 under s3 bucket and name yes three so under s three we have bucket and a bucket name so this gives bucket name correct so likewise I should get the file name yes three Kyle named so we need to get that information in similar fashion so from object records the first record then yes three under s3 this time it is object followed by tea object followed by key which gives us a file name I'm going to pass that info here so bucket name and the file name so whatever it returns I am storing that in response variable from Beaudry documentation we have to understand the response structure of this method so response type is dictionary so from this dictionary I am interested in the body let's grab that information response of body that gives streaming body data so from streaming body I should call read so first before I proceed with before I proceed further logic I want to show you what we get part of data alright now I want to test this code so one way to test this code is every time you want to test upload fire and on that s3 configure an event which triggers this lambda function so that's going to be a little lengthy process so a SS lambda has a concept of testing within this function click on this so we need to choose the event template this time it is yes three put event see that so this is our sample event which simulates the actual event we're going to configure once we are done with the code so in this event I need to replace only bucket name and the fire okay let me first cancel this minimize this one and I want to take you to s3 dashboard in order to show which bucket we will use I am going to use this bucket and I'm going to upload employees dot CSV file or testing our code the bucket name is this so test I will call this one as CSV CSV and change the bucket name and the key employees dot CSV okay so this is a mock data this is mock data when I call test this mock data is passed to this event object from there I am getting bucket name and the file name so the bucket and the file already exists when I test this it should work say test response is in binary format and it is in single string to get it as a plain text I need to call D chord utf-8 so when I call decode utf-8 the binary form will go off and we'll get information as a plain text yeah see that now we got that information in plain text so all data in CSV file is coming as one single multi-line text the next part is I need to split this multi-line text for newline I mean to say I'm calling split I am splitting the file sorry and I am splitting the text wherever there is a new line let me store that information in employees let's print employees and check you right so we have three entries for three different employees so we have to loop on this for EMP in employees let me print and make sure we're getting data in proper format save it testicle so it seems pretty much good for me yeah we are getting data from yesterday now the next job is so persist information into dynamodb so add it to dynamodb I need to get the client of dynamodb here let's go to Bordeaux three documentation for dynamo click on table coppy discord let me maximize this window but a three resource dynamodb that's okay and the table name I need to give let's do one thing let's open dynamic dashboard in separate window yeah we already have this employees table if you go and click items ID is primary key in that this name should be employees using this object I should call put item and insert employee data into it let's take help from DynamoDB methods we have put item there it takes item object so here otherwise after this comment table dot put item item equals it is a dictionary i D dydee right right now I'm hard-coding name name their followed by location location but I need to remove all these hard-coded values why would I should pick that information from the employee so again we need to split employ wherever we have comma right you MP dot split wherever we have comma it's going to split each line wherever it has comma and we're going to get a list of entries with ID name and location let me call this employee data employee data of zero which is ID employee data of one which is named employee data of two that the cross-check index zero is ID name location i D name location okay let's save this cord and test it go to your dynamodb refresh it okay so why we are getting only one he's some bug in the code let's fix that seems the problem is this one so it should not be double /n it should be a single slash Ian let me rerun the code SAVE and test so we still have some issue index out of range so I see the records are getting inserted properly but index out of range probably this is because of empty line in the end of CSV file so probably this is causing the issue let me take out that and reupload the file so go to s3 so upload add files employee dot CSV upload it for testing that may delete all entries I think it is already saved test it do we still see okay out of ever let me cross-check what they're getting fine so what is wrong then you you okay let me check this I think this is causing the issue the end we are getting probably empty line that is causing issue I will do a simple fix to this what I'm going to do this I'm going to wrap this cord inside try and accept blocks so whenever I have an exception let's assume that is end of the file so print in the file so save for retesting let me delete all so go back make sure we are saving it and test again invalid syntax lambda 25 this I can use as yeah save it test it now I do not see any exceptions and I do see the data in dynamodb now I need to automate this right so I need to automate okay the bucket is in Tokyo let's do one thing let's create a new bucket Java home CSV demo CSV demo one two three okay keep it in by create the bucket open that go to properties events CSV upload all objects create event Suffolk's dot CSV send it to lambda function yeah so say this now anytime we upload CSV file to this bucket that triggers this lambda function as part of this lambda function event we get the information about which bucket which file and so on which automatically picks that information read CSV file inserts data into dynamodb let's test this functionality so let's add other records here let me save for five six okay you fine save this file go to s3 bucket hit upload add files employees dot CSV uploaded it's successfully uploaded if the other thing is fine we have to get data so this is all about the demo and the code is working fine and it's completely automated

Channel: Java Home Cloud

Views: 16,661

Rating: 4.8943896 out of 5

Keywords: java home, devops, java home cloud, devops tutorial, devops by javahome, devops by java home cloud, aws, aws tutorial, aws by javahome, aws lambda csv, aws lambda tutorial, aws boto3 tutorial, aws boto3 s3, aws boto3 dynamodb, aws lambda python tutorial, aws by java home, AWS lambda CSV example, aws boto3 lambda tutorial

Id: lRD_oHoZHlI

Length: 22min 33sec (1353 seconds)

Published: Fri Jul 12 2019