ESP8288/ESP32 to AWS API Gateway Direct (Part 1 of 4)

hey there you may or may not know me if you've been following this channel but I'm Steve Boer say I decided to give you guys a little break today I'm not gonna have a webcam so you can see my face during the kovat outbreak I'm just gonna do this with audio and video I know you're disappointed but you'll survive anyways this is who I am you can read over this if you're excited about it but I'm really excited to bring you this lecture it's based on my new course on you to me so make sure to check that out if you want the lowest price coupon I think it'll be like 10 bucks for you just send me an email if that link I provide on this YouTube video is not available but this course covers connecting the esp8266 or the ESP 32 device to AWS and then doing all kinds of lambda API gateway DynamoDB s3 visualizations with the web host all kinds of exciting IOT stuff which is what I focus on specifically and for this video it's going to be a four part series I'm going to be doing the esp8266 or the ESP 32 I have sketches for both depending on which one you want to do connecting to API gateway via rest to your lambda function storing data and s3 and then finally we're gonna visualize that data in a static web post using either high charts or chart chess I'll decide a little bit later and you can use whatever charting program you like but I'll provide you all the coding material you're gonna need so you don't have to have any of your own code just to free to your AWS account or your own AWS paid account and if you have a paid account to supply cost you about five cents I mean these are all super cheap services for prototyping now a good question to be asking right now is why are you sending IOT data via a REST API to api gateway rather than sending MQTT to AWS IOT core well throughout the course and 95% of the time that's normally what I'm going to do is use MQTT and send it to AWS IOT gateway but I thought this would be more unique as far as a tutorial and walkthrough to show you indeed even with this less powerful esp8266 along with the more powerful ESB 32 we can send data directly to AP Gateway and avoid the AWS IOT broker if we want to for whatever reason and there's reasons you may want to for instance if you have an embedded device that's constrained to only HTTP rest or if you already have code designed and rest and you don't want to rewrite it there may be various reasons you may want to send it to api gateway directly rather than through the AWS IOT core broker but this will be something unique you can kind of do because there's not a lot of information out there on doing it this way and here's our agenda so for this video we're simply going to set up a private s3 bucket we'll make it public later when we need to and design a lambda function to send the test i OT payload to that bucket then you'll see what we're doing for video 2 video 3 and video 4 and I'll post a written walkthrough for the first three videos I don't want to do video 4 because that's gonna be too advanced at that point and drawn out for most people but if you do the walkthrough written and cross-reference it with this YouTube video you can go ahead and use video for it to kind of as a stretch lab for your IOT endeavors so with that said let me talk a little bit more specifically about what we're gonna be doing it's useful if we have sensors out in the field to bring IOT into our class to analyze the results whether we want to do machine learning AI or do basic visualizations business logic whatever we want to do so in this example it's going to be kind of a basic walkthrough of our device or esp8266 or ESP 32 is gonna send off fake data although you're welcome to put in any sensors you want peripheral libraries are very trivial into our dueño IDE and then we're gonna send that via a public facing API through API gateway we're gonna make an API key for our publicly facing URL so we're not just letting anybody connect and then that's going to connect to our lambda function which is gonna push that data on with a little bit of enrichment and filtering on to our s3 static data bucket and then in our s3 web host we're gonna ingest the data from that IOT data in that s3 data bucket and visualize our results so that's kind of a walk through we're gonna be doing so having said that let's go ahead and set up our s3 bucket set up our lambda connect the two and send the test payload so we make sure we're working and then we'll move on from this point okay so here we are in our AWS management console now I'm not going to be using the server list model the SLS from the command prompt you know a lot of people teach that way but I think for learning heuristics it's just better to go through the console if you understand the console you can always move to the serverless framework or chalice or whatever kind of frameworks you want to do so that's the learning path that I'm gonna teach and here I'm in Ohio you can be in any region you want I think I'm just gonna keep this in why don't we do Northern California that might make it more exciting I'm just trying to pick a region where I don't have a lot of clutter that's normally how I do it of course some regions have more stuff available in other regions usually North Virginia's the safest region us east ones because it's kind of the prototyping region but let's go ahead right now and whatever region you're in just keep these services consistent between each region so let's go ahead now and go to s3 and this is gonna be the easiest step I'm just gonna create an s3 bucket so go ahead and create a bucket now I kinda come up with a global unique name and the reason AWS requires a globally unique name is they're kind enough to give your buckets all their own static IP their own unique URL so to do that of course you need a globally unique name because there's only gonna be one static IP or global unique URL per address so we'll call this one test bucket and I'll say seven eight B and that should be pretty well a globally unique name I'm gonna assume I'm not gonna set up any options so no configuration tags or anything like that and I'll keep this bucket as private so go ahead and create this and indeed I was successful with the global unique name I'll go ahead and refresh this by day so I know the last state that I created this bucket always be my most recent bucket I'm gonna go and open this and I'll go ahead and create a new folder so we'll call this folder test folder this one does not have to be global unique and I'm not gonna have any kind of encryption in this folder alright great so we have this bucket and new test folder and there's going to be nothing in this test folder alright we're good to go with our s3 bucket and now we'll go ahead with the more complicated step and create our lambda and I'm gonna want to keep this in the same region so I'm just gonna duplicate this tab go over here and come back to AWS and now I'll choose lambda keep it in Northern California and we'll create a new function author-it from scratch and i'll call this my test function 7:8 just using the date I'm going to keep it in nodejs 12 and I'm gonna leave the permissions is just normal execution roll basic lambda purchas which allows it to execute itself I'm going to add some permissions at the end of the video you're gonna see why obviously to connect to s3 and to keep that there for now so go ahead and create that function it's going to spin for a minute okay good times it created that function successfully and of course there's justice hello from lamda message so now I'm gonna drop in my own lamda code in node so I'm gonna copy this and then I'm gonna explain briefly what it does and then we're gonna test it and then we can move on to the next video so I'll provide this code to you and of course being a Linux system you can't do a copy and paste so ctrl V and here's how it's gonna work I'm gonna bring in the AWS SDK and within that specifically AWS s3 object extension an object called s3 have a variable with our bucket name and a folder name and I'm gonna have the key name as date now now you can use the UUID library as well all we're trying to do with that is have a unique name for each data object in our bucket I like to use date dot now for epoch timestamp because I get the benefit of having non-collision just like the UUID library but I also get a nice sequential timestamp so I usually use date now instead of UUID and I don't have to roll up I don't have to do any NPM with rolling up in the external libraries now I have two comments commented out here I have a vent which is my incoming data object and I have more specifically when we go through API gateway I want to extract something nested deeper I use event query string parameters for this part of the lecture I'm just gonna use event so that's good enough for right now and everything else here is done filling the object and dispatching that via the s3 put object to our s3 bucket so the only thing I have to do now is fill this in and a real easy way to do this is come back here to s3 so first I can copy my bucket here right here I'm just gonna copy that come back to lambda stick that right where it says your bucket name and you'll have your own obviously unique bucket name because you can't have mine and then my folder name is just test folder and those are the only two parts you have to copy and everything else is going to be done for you so I'm going to go ahead and save that and the next thing I have to do is test this to see if my JSON data object is going in my s3 bucket but as I said earlier in the video I can't test it because currently my lambda does not have permission to send anything to s3 so here's what I'm gonna do I'm gonna scroll up here go to permissions and I'm gonna open my permission which so far is just to lam the basic execution roll allowing lambda to execute itself so I'm gonna add a managed policy so the way I do this is say attach policy and I'm just gonna type in this box X 3 and I'm lazy so instead of using the least permissive possible I'm gonna go ahead and give it all s 3 access since I'm allowing it to write the s3 I'm not too worried about it deleting remotely so if you want to make it more professionally you can narrow down the per nisshin's to just the s3 put but I don't want to waste the time doing that for this video for prototyping so I'm just gonna give it s3 full access and attach this policy all right cool that's already done so I can just go ahead and shut this off and I'm gonna go back here the configuration tab and now we're good to send whatever test payload we have from lambda to that s3 bucket we just created as designated right here so let's go ahead and configure our test events so I'm gonna go in and configure test event and I'll just put in a little bit of custom code here so you know this isn't fake and how about that alright so we have a JSON object here we just have to make sure it's proper JSON and I messed that up so I'll put that back and I'm just gonna call this t1 that's a catchy name go ahead and create that and then we're gonna test it and confirm that we didn't have any problem so a test and this is our results for successful we have a null result just because I haven't programmed any response but it looks good so the last thing we have to do in this video before moving on to API gateway has come back to our s3 bucket and refresh to make sure indeed we have a data object in here now we'll click this data object but you'll see if I click this right here it's gonna say access denied and the reason it's saying that is because this is a private bucket so I'm gonna make it public later but for now I'm going to keep it private so we can still overcome that I simply have to download my data object again remember this isn't a file because you can't concatenate or appendant it's a data object for blob storage so download it and now you can open this in any editor you like if you're really cool you're gonna be using vyas code I'm pretty lame so I'm gonna be using notepad plus plus I'm showing my age but I find the vs code to be kind of bloatware for most of what you would need to do all right you can see it's successfully sent my custom test event this is exactly what I set so you need to be at this point before you move on to video two to make sure everything's working to this point cuz if this isn't working nothing else going forward is going to work and that's true for each of these videos so let's move on to the next video where we set up API gateway and set up an API gateway security key in the next video

Channel: Steve on IoT

Views: 2,093

Keywords: ESP8266, AWS IoT, ESP32, Arduino, API Gateway, Lambda, S3, HighCharts.js, HTTP

Id: NALTj_ZepNE

Length: 12min 35sec (755 seconds)

Published: Wed Jul 15 2020