AWS IoT: Storing IoT data into DynamoDB Database

hello all welcome back in the last tutorial we have seen how to publish messages to AWS IOT broker and how to send messages from IOT broker to your email as well as SMS clients so this tutorial I am trying to show how to store your data into a cloud-based database so AWS provides a cloud-based database which they call as DynamoDB it is a no SQL database and it has several advantages okay so first thing I will quickly show you dynamo BB then we will see how to store data into DynamoDB so you log in to your AWS console and search for dynamodb and you will get the console of nano buddy okay so again every aw service it has a console which looks similar okay so since I don't have any database now this is how the welcome screen looks like now basically Amazon they call each database as a table okay so on the left you can see tables so if you want to create a new database you will go to tables and click table and choose create table and you need to give some name to your table okay let's use the table or the database to store some temperature data so I am putting it temperature database okay now another thing that you need to provide is something called a primary key so as I mentioned before they use a special technique to store data into database so each row in the database should be uniquely identified for that they will be internally using a technique called hashing okay so to identify each row uniquely ews requires something called a primary key that as a user you should provide that primary key can be your string type binary type or a number type okay any three of them so in most cases we will be using a string type because they will be able to apply hashing on string and they'll be able to store it again when we are using temperature things like that we need to associate each temperature value with some particular time and date so we can be able to provide this information in a string format and using that string a double use will be uniquely able to identify that particular role in the database okay so primary key I am just calling it time so these are the names this is the name of the table this is the name of the primary key which is used for uniquely identifying a particular row in your database so once you have done this much choose create now when you use dynamodb again first year it is free but with certain limit so there are restrictions on what is the maximum size of your database and how many times you can read and write from the database so if you cross that limit you will have to pay a W so keep that in mind so when you develop IOT for testing purpose make sure after you don't you don't run your application like or the night or something because that can encode charges because you are making too many read and write from the table okay so he already created the table again okay so this is how it looks like now we will go to the details later at the time being what we are interested is to put some data into this database so you go to items and here you can see like times because I specified that a spy partition key okay so if you want to create an item you can go to create item and you can specify the item now if you notice here at the top there are two option tree or text you can either enter your item value using the tree format or the text format but when you choose text format you can immediately see like this is following JSON format you can see the curly bracket and he's saying that attribute each time and we are supposed to give some value to this time and that should be a string type okay so let's give some date maybe today's date for when 2020 and lets me use some time also you know 1:45 okay so that is the time after that whatever data you want to store in the database so I want to store temperature now as I mentioned before everything in usin should be in the attribute value format and your attributes there should be strings and they should be put inside double quotes to indicate the Airstream your values can be strings numbers even Eddie's things like that okay no not much restriction on what is the value but the attribute should be string type so if you want to enter temperature you need to put temperature in double quotes and if I okay and you can save it and now you can see that got entered into the table now the beauty about this no one SQL database is there is no restriction on the number of columns you can have in the table it can be variable length what I mean is each item in the table can have different number of fields for example here I have only one field called temperature but when I create a new item say again time that is mandatory because that is our primary partition okay so let's put a value okay and here let me just put you can save it we need a comma here see if you look at this item it has only temperature attribute there is no pressure attribute but if you look at this attribute it doesn't have temperature but has only pressure attribute so this is one advantage of this known as ql DynamoDB so the number of fields for each item can be different again practically there is no limitation on number of fields that you can have but for our IOT application mostly every item will have say number of attribute guess so this is a board dynamodb and as i mentioned before this is a separate service um areas this is not part of IOT but we are going to link IOT application with this DynamoDB database that's what we are going to do so first thing we have to modify a publisher code because as you notice in our current publisher code we are just sending the temperature value which is the current temperature with the attribute temperature now practically in dynamo DB you have two key one key which has to be used for uniquely identifying arrow but the problem is you can have multiple temperature with the same value so you won't be able to uniquely identify each row that is one thing and if you simply store the temperature values in the database it doesn't make sense because it's not giving any any special information so usually you need to store temperature and associate time or something like that okay so that's what we are going to do so I'm slightly modifying my Python code I'm going to get the current system time for that we can use the date/time pitch so if you want to get the current time okay so here we are reading the temperature okay let's hear at the time something can say a simply date time or date time now okay so when you use this function it will return the current date and time now we need to form the message okay so let's look at the architecture this is how the message should look like maybe we can put the time stem at the beginning so curly bracket within double quotes x term so we have curly bracket within double quotes we can say I'm camp and followed by a colon and we need to add the actual time staff so we need the string format string Plus after that we need a woman okay after each attribute we need to put a comma so we need oh ma after that we can have temperature again that should be in double quotes so I need to put a single quote here the temperature for temperature value and let's try to run and see whether it's giving okay it looks fine now let's see whether it is really coming in JSON format so let me go back and let me open my eye oh t platform and use my dummy client to check whether I'm getting the data properly or not so I'm just relieved in this table we will recreate it later so I'm going to Rio T pour back and let me test it now yesterday I remember we created a rule whenever I get some temperature I need to send an image now I don't want to keep on getting image so first thing I am going is I'm going to act so this is the email this is the rule which was sending email to me for the time being let me just disable it otherwise I will keep on getting emails so let me go back to test and try and subscribe to a topic let's put this one temperature of take subscribe and let's okay descending but here he is saying something is wrong so we have curly bracket okay so this is a string so we need to put double quote across these value also to indicate this is a string because this is not a number you can see there are numbers but there are blank spaces as well as corner so altogether they should be treated as a string so let me go back and here we need to put double quotes okay so single quote double quote single quote us or you can put a slash double quote and as single chord the chord think you caught oh we can also look at should it stop and rerun okay no he is not complaining because it is in JSON format okay so we have the time stamp this entire thing is the time and we have temperature and this is the temperature value time stamp is a string and temperature is a number floating point number okay so we are getting it next we need to put this information in to be so the steps are exactly same as the same way you did for S&S we need to go to act we need to go to rules you can add it to the same no problem because same message is coming it can go to s honest as well as to database no problem but for the time being let me create a new rule and let me call it IOT temperature so here like yesterday select star from the name of the topic take the entire message from topic or topic is temperature drop special condition we are always taking and add action and choose insert a dynamo a message into a dynamodb table and the configure action and table name okay if you created a table there in dynamo DB it will automatically come here because I deleted it there it is not coming here okay so let's click on create a new resource which will actually open up dynamo DB and you can create it it will know so let's call it in furniture table and I'm ready that's why we created this one so you need to give the attribute aim there which you are going to use as the primary key so it is time stamp it is a string of care we can say create so he created it may take a few seconds okay now let's come back here and click refresh okay and you can see that table is already here and he automatically filled these parts time stem string but important thing is this key value okay this so this this is just the name of the key okay so the name of the key is always timestamp now he's asking what should be used for actually hashing for finding a particular row okay what value should be used for hashing to uniquely identifying a row that is actually the value of time scale okay so actually confusing why it is there but you need to write like this the name of the key is timestamp and actual value use for finding a particular row or for hashing if the value of x down so you need to put this donor similar to our excel so this is indicating the value coming in touch okay so that's it and these things we can see later time being it's enough now here again you have this roll thing you have seen last time which is for managing the security thing the last time we created a roll already so that will be already listed here but this time let me create a new role again why we are doing it we will see in later tutorial what this role is actually doing and where this information is stored when we discussed about a double yes I am we will discuss it so for the time being just create a new road so let me call it my IOT DynamoDB role and choose create role and that comes here for to see sort of automatically attached and choose create action so that comes here now you choose create rule so now you have two routes but this one is disabled only this one is enabled now let's go ahead and run our code once again okay so he's sending data now let's come to our table here and look at item and you can see whenever he is sending a message that he is getting updated here okay so each time you refresh the table yeah you can see that this information is getting updated here now you will notice that he is putting that entire payload okay the content of that entire message here because that's how we asked in the SQL whenever a message comes give that entire message to dynamic that's how we brought our SQL query we're here so that entire thing is going here from that he is using the timestamp part for finding a unique row then he is touring that entire thing as payload now if you want to store only this temperature part here we have to do something more so the easiest thing to do it is the following okay so you can use this technique provide that you want to store only one attribute value in a separate column so when you create the table now I cannot edit the attribute of this table when you create a table let's create a new one okay my new create table and here again let's keep our x by stamp there's an option add salt key okay so when you add this there will be an option for one more key so this is like a secondary key and you can use this key for sorting the values in the table ascending or descending order okay so we can make advantage of this and you can give em furniture as this thought key and our temperature is number type it's a floating point number so keep number type and let's keep a new table here so now we have two tables here this is our old table and this is our new table now let's go ahead and change the action here so okay so let's refresh refresh and choose our new table and here you can see time stamped all the time stamp is already filled now sort key already came here right so the key value here is infra Chur and remember to click this option update roll yeah otherwise nothing might work and choose update okay okay so now we are going to insert whatever item is coming to this second table I guess code is still running low okay so let me rerun my code and now if I come to the items here you will see see this is that additional key the sort keys because using this you will be able to sort in ascending and descending and since I said like use temperature as my sake that inserted into a separate column so the temperature values are here the timestamps are here and the enter message is here yeah so this way you can have it in a separate column provided it's only one item so if you want to store temperature and pressure in two separate columns we have to do something more that we will discuss later now one thing is if you want to download this file to your local computer you can just click here and go to actions and choose export to CSV so he will download it as a CSV file to your computer and you can use it locally for if you want to do any processing maybe we are not interested the end of a lot you can delete it and you can have time and temperature values in this file fine okay so one thing remember as I mentioned before for safety purpose we are just testing things once you have done testing once you understood things just go ahead and delete the table so that in future if you accidentally keep on sending data it doesn't get stored in this table and they Tobias charges you okay so that's all for this tutorial see you in the next tutorial

Channel: Vipin Kizheppatt

Views: 7,041

Keywords: AWS, IoT, DynamoDB, Amazon Web Service, Internet of Things, IoT Database, database, data base, storing iot data, storing iot data in dynamodb, storing iot data in database, dynamo db, dynamodb database

Id: ktQoA4LV__w

Length: 20min 58sec (1258 seconds)

Published: Tue Apr 07 2020