Ingesting Data into Amazon Timestream with AWS IoT Core

hi my name is philip zaha and i'm a specialist solutions architect for iot at amazon web services i will talk about aws iot core and amazon timestream integration that means how you can ingest data from your iot devices into amazon timestream what i'm going to cover is the architecture to ingest iot data what an iot rule is and how it works how to create a database and table in amazon timestream create an iot rule ingest data into aws iot core with a data generating script do some simple queries and do some basic troubleshooting there's also a brief story behind this example setup you have sensors and buildings which measure humidity pressure and temperature apart from the measures the sensors will also report the building name and room number where they are located as well as the device id let's take a look at the architecture to ingest iot data into amazon timestream in general on the left side you have your iot devices they are sending data to aws iot core a topic rule then injures data into amazon timestream now let's take a look at the topic and message payload your devices are sending data by using the mqtt protocol or via https these data are published as a json object to a so-called topic in our example sends was published to the topic dt slash sensor slash device id the device id is the id of a particular sensor in this example the device id is sensor underline 0 2. in the message payload you can find the measures humidity pressure and temperature and the dimensions device id building and room dimensions and measures are terms from amazon timestream an entry in amazon timestream contains dimensions and measures dimensions are attributes that describes the metadata of the time series and measures are attributes that describe the data of the time series and here you can see the iot rule which ingests data into amazon timestream an iot rule consists of a sql statement a topic filter and a rule action with a sql statement you select humidity pressure and temperature from the topic dt sensor slash hash hash is an mqtt wildcard the wildcard is used here to receive messages published by all sensors values extracted by the select statement are stored as measures in amazon time stream in the rule action for amazon time stream you define which dimensions should be stored each dimension consists of a name value pair in this example we use the values from the message payload and now i will show you how to implement this architecture in the first step i will create a database and a table in amazon timestream here you can see the timestream section of the aws management console i chose create database and then standard database i will use iot for my database name leave encryption settings as they are and create database now i can create a table here under tables create a table i will choose the database are you teachers that i just created use cinder data as my table name and i will pick some arbitrary values here for the data retention like one day for memory store retention and five days for magnetic store attention create the table and that's it i have created a database named iot and a table named sensor data within that database now i'm going to create an iot rule i'm using the iot core section of the aws management console rules are created under act rules create a rule choose a meaningful name i will use to time stream rule as a name and define the sql statement to select data from our messages we will select humidity pressure and temperature from the topic dt slash sensor slash hash next i'm going to add an action in this case the time stream action configure action i need to choose a database i will use the database that i just created iot and the table sensor data now i need to define the dimensions i will use the device id and get it from the message payload another dimension is the building we will also get the value from the message payload and we will also have the room in our timestream database also from the message payload in the next step we need to define a role or to create a role which grants aws iot permissions to write into amazon time stream create a row i will use the name iot data to time stream role create the role a policy is attached automatically that allows the iot service to write into amazon timestream then i choose add action and finally create the rule and here you can see the newly created rule to timestream rule for ingesting data from iot core into amazon timestream we need to generate some data to populate the amazon timestream table data are generated by a script the script is implemented in python 3 and uses boto3 the amazon web services sdk for python to publish messages to iot core to set up the environment for the script you need to install both 3 and set up authentication credentials you must associate the permission to publish to iot with your credentials installation of portal 3 and setting up credentials is covered in the border 3 documentation i will run the script on an aws cloud 9 ide in my setup i have attached an instance profile to my aws cloud9 instance which allows me to publish data to aws iot but you can launch the script in any other environment which meets requirements to run it now let's start the script it's called sensor data you can see here activities and messages are published to aws iot core to verify that messages are reaching aws iot you can use the mqtt test client which is built into the management console here we have the aws iot management console you find the tab client on test and then we will subscribe to a topic dt slash sensor slash hash subscribe as you can see messages are arriving at aws iot that looks good so far and now i will continue with some database queries the amazon timestream console has a built-in query editor that i will use for some queries how many rows do we have in the cinder data table now i run the query that yields over 40 000 rows as we have entries in the table that means that our setup with ingesting data to aws iot core and then write the data with an iot rule to amazon timestream is working now let's find out which buildings do we have in the table select distinct building from iot sensor data you can see here three buildings named doppler kumu and day one and now we can also have a closer look at the data for example from the building day one select all data from iot sensor data where building is day one run it and you can see here a lot of results you can see here the sensor id building name the room number a measure value and a measure name and also the time when the data were ingested into amazon timestream these were some very basic queries to verify that the whole setup is working if you want to visualize your data you can use for example grafana which is a analytics and monitoring solution please take a look at our getting started video how to visualize your time stream data by creating dashboards in grafana finally some words about troubleshooting in case your setup is not working as expected it can have several reasons one option to find the root cause is to use amazon cloudwatch which is our monitoring service when you enable logging for iot core logs are written to amazon cloudwatch logging can be enabled for iot core under settings as you can see here you can use cloudwatch insights to query your logs in the cloudwatch console under logs select insights and define your query i want to know what the iot rule is doing to ingest data into amazon time stream i am choosing the log groups for iot that is aws iot locks v2 and add a filter to the predefined query i will use a filter for the rule name our rule was name to time stream room and i will run the query and here you can see the results you can see also a lock level here within the results and this is only one example how cloud watch insights works you can search other events for example if your devices are ingesting data properly into aws iot and much much more yeah with that we are at the end of this short quick start and you have learned how to ingest data from your iot devices with aws iot core into amazon timestream thank you for your attention

Channel: Amazon Web Services

Views: 7,485

Keywords: AWS, Amazon Web Services, Cloud, AWS Cloud, Cloud Computing, Timestream, time series, IoT, database, SQL

Id: 00Wersoz2Q4

Length: 12min 58sec (778 seconds)

Published: Wed Sep 30 2020