AWS IoT - Device data to dashboard in 10 minutes - A demonstration

in this short video we'll demonstrate how you can quickly and easily create an AWS iot thing and store and visualize the data it generates we'll do all of this with just three AWS services AWS iot core Amazon timestream and Amazon managed grafana if you'd like to follow along all you'll need is a laptop or some other device with python installed and an AWS account my name is Chris Green I'm an iot specialist Solutions architect in the AWS worldwide specialist organization here are the high level steps we'll follow first we'll create an AWS iot core thing and install and run the connection kit that iot core will automatically generate for us next we'll create the time stream database that we will use to store the Telemetry from our device and finally we'll use grafana to create a simple dashboard to display the data before we dive into the details let's review the simple high-level architecture of our demonstration this diagram shows the end-to-end view of our system from the device that is generating the data to the data visualization in our dashboard we'll use our laptop as an iot device that will generate CPU utilization telemetry this of course could be any device capable of generating data and either sending that data directly to AWS iot core or forwarding the data to another device such as a local Gateway or proxy that has a network connection AWS provides sdks in a variety of popular languages capable of running on microcontrollers such as esp32 or stm32 and also devices capable of running Windows or Linux in this demonstration we'll use the AWS iot python SDK on our laptop the CPU utilization Telemetry will be sent as mqtt messages over a secure mutually authenticated TLS 1.2 connection to AWS iot core the connection in the transmission of the data will be accomplished with two lines of python code one to establish the connection and the other to publish the data the device's identity is defined with an x509 certificate the device must present the certificate to establish the connection to AWS iot core the AWS iot SDK will take care of this for us the device's permitted actions including where the Telemetry data can be sent are defined with policies that are associated with or attached to the x509 certificate the connection is managed by the AWS iot core device Gateway the Gateway provides support for mqtt traffic based on both the mqtt 3.1.1 and 5.0 specifications and also supports other protocols such as HTTP websockets and lorawan but in this video we are focused on the mqtt protocol the AWS iot core rules engine can route our Telemetry data to other AWS services and also to external destinations the rules engine is very powerful and can inspect transform and augment messages using over 60 AWS provided built-in functions you'll find a link to the documentation for these functions in the notes Below in this demonstration we'll create a routing rule that subscribes to the mqtt topic the laptop is sending the data on and then refs selected items in the message payload to our time stream database we'll persist our Telemetry in a data store in this demonstration we'll use Amazon timestream as our data store but popular alternative destinations we could use include AWS iot analytics pipelines Amazon F3 Amazon dynamodb and Amazon Kinesis data streams we've selected Amazon timestream because it is a fast scalable and serverless time series database service that makes it easier to store and analyze trillions of events per day while automatically scaling up or down to adjust capacity and performance so we don't have to manage the underlying infrastructure now that we have the data in Amazon timestream we can build visual representations to observe the incoming data in close to real time or retrospectively analyze data over a given time window such as the last minute or last hour or whatever time period is needed in this demonstration we'll use Amazon managed grafana to build our visualization Amazon managed grafana is a fully managed service for grafana which is a popular open source dashboard platform you can find links to the grafana open source website and the Amazon manuscript Pharma service in the notes below we're using Amazon managed grafana because it automatically scales compute and database infrastructure as usage demands increase with automated version updates and security patching all this while enabling us to query and visualize our data Amazon managed grafana also provides a built-in connector for our Amazon timestream time series database so let's move on to the demonstration and see how quickly and easily we can build the end-to-end system described in this architecture first we will create an AWS iot core thing that represents our physical iot device in the real world which in our case is our laptop we'll also download and run a sample python application that AWS iot core will create for us here we are in the AWS iot core console we'll start the workflow to connect our device our first step is to ensure that the device can reach the ows iot core device Gateway with this simple pin command next we'll provide a name for our new thing and we can skip the optional additional configuration items for now thank you foreign the operating system and SDK language that we will use now we can download the device connection kit at AWS iot core has generated for us we simply unzip the connection kit into our preferred directory on our laptop and then execute the start script thank you the start script automates a number of steps for us including downloading the python sample application from GitHub ensuring the required python modules are installed and once the application is running we can see The Messengers arriving in the console so that's all there is to it in just a few minutes we've created an AWS iot core thing downloaded the connection kit which provides us with a sample client connected the sample client to iot core and began sending mqtt messages on our specified topic which in this case is SDK test python we can also observe the incoming data in the iot core console mqtt test client who simply subscribe to the SDK test python topic and we can see the data arriving in iot core we can inspect the iot core thing that has been created for us in the console we see that our next 509 certificate was automatically created for us and a permission policy created and attached as well we can see in the permission policy that the thing we created only has permission to connect publish And subscribe to the SDK test topics let's take a closer look at the sample client we are running our startup script downloaded the AWS iot python SDK which contains the sample client [Music] we can see that the start script executes the pub sub.pi sample and specifies the AWS iot core endpoint and the certificates used to establish the connection we take a closer look at the code we can see on line 80 we established the connection to AWS iot core foreign 114 we see the message being published the raw sample code doesn't use a Json message payload we can make a small modification to the client to publish a well formatted Json document containing our CPU utilization Telemetry you could build and include your own message payload at this point in the code but for now we'll just keep it simple and just include the CPU utilization you'll notice that I've also added an import statement for the psutil module so we can call the CPU percent method this import would normally be declared at the top of the file but I have just added it in line here to keep things simple for this demonstration I'll also add a print statement so we can see the well format of Json payload that we have sent thank you so now when we start the client again we can see the Json formatted payload showing the CPU utilization and we can see the Json messages arriving in the iot core test client console so we've created our iot thing and we're sending live CPU utilization Telemetry in a well-formatted Json message payload the next thing we need to do is to create a timestream database and table to store the incoming Telemetry we'll then create an iot rule to Route the Telemetry to the time stream table foreign first we'll create a new time stream database we'll use the naming Convention of demo thing and accept the default encryption options now that we have our new database we can create a table that will hold the incoming telemetry timestream provides a tiered storage model where you can specify how long you want data to reside in memory and when data will be deleted from disk timestream is capable of accepting light or delayed data with a timestamp of the incoming data is older than the retention period of the in-memory storage enabling magnetic storage rights allows time stream to update the magnetic storage when incoming data is delayed so now that we have our demo thing table we can create an iot core rule that will route the incoming data to the table the iot core rules engine is very powerful and enables us to select evaluate augment and transform incoming data and then route the data to Downstream services using one or more rule actions we'll use a simple select statement including the entire payload represented by the asterisk and specifying the mqtt topic that our incoming data will arrive on the where Clause can be used to provide the conditions in which the rule action will be invoked for example if we specify a clause of value is greater than 80. then the rule action will only be invoked when the CPU utilization exceeds 80 percent we won't need to use the where clause for this demonstration next we'll select the time stream action and configure the database table dimensions and timestamp values we can use substitution variables to take values directly from the incoming payload we will also create a new role that will enable the rule to write to the time stream table it is also best practice to configure an error action this action will be invoked if there is an error processing the rule from demonstration we'll simply publish the error details to a new mqtt topic will create a new role that will enable the rule to publish the messages to the error topic and finally we'll review the rule definition and create the new rule now that our new rule is active we can query the time stream database to verify that the data is being stored correctly now that we have our Telemetry stored in our time stream database we can create a dashboard to display the data as a simple time series line graph from the Amazon managed grafana console we can create a data source for our time stream table we add the timestream data source select the region and test the connection we can also set the default values for the database table and measure now that we have our time stream data source we can create a new dashboard add a new panel write the data query for the panel and configure the appearance of the panel to suit our needs and now we can see our live data generated by our iot thing being displayed in close to real time on our dashboard so to summarize what we have done our live data is being generated by our iot thing and transmitted securely to iot core where we used a rule to Route the data to a timestream database and we used Amazon manage grafana to display the live data we've seen how we can create an end-to-end iot data to dashboard solution simply using just three AWS managed services please see the links below for additional reference material and thank you for watching

Channel: AWS IoT

Views: 54,421

Keywords: aws, iot, awsiot, timestream, Grafana, dashboard, getting started, internet of things

Id: z8T4hAERuOg

Length: 17min 27sec (1047 seconds)

Published: Fri Mar 24 2023