Using Python to Automate AWS Services | Lambda and EC2

what's up everybody it's Travis here from travis.media so a couple of days ago I released a full Python tutorial python for beginners and if you watch that you now know python but the question is what do you do with that python well that's where this video comes in in this video I'm going to take those new python skills that you've developed over to AWS and teach you how to automate some Services over there so if you're looking for more python practice or you're looking at how python integrates with AWS what people do over there then this video will be for you and here's what we're going to be building imagine that your boss or your team lead comes to you and says hey whatever your name is we're having trouble keeping up with who's creating all of these ec2 instances or servers on our system so here's what we need you to do we need you to set up some Automation in AWS so that when an ec2 instance or server is created it kicks off some kind of automation that takes the information from that server processes it and then goes and tags the ec2 instance with the name of the person who created it so if Bob creates an ec2 instance and didn't tell anybody about it this automation will kick off find that ec2 instance and tag that with owner Bob so that if we have 50 100 servers in our environment we have a tag for each one letting us know who created it who the owner of that server is that's what we need you to do and that's what we're going to do today so if you look at this simple diagram and there are things left out for simplicity's sake but if you look at this diagram here's essentially what's going to happen somebody's going to create an ec2 instance or server when that's created a rule and event bridge is gonna fire saying hey a new ec2 instance has been created and triggers a Lambda function in that Lambda function will capture data and process it and then send that owner information over to that ec2 instance and tag it with that information so ec2 instance is created it triggers that action triggers a rule and event bridge that fires this Lambda that goes back and tags the instance so that every time an instance is created instantly we have a tag with the user that did it that's what we're building today so the first thing you want to do is to create an AWS account if you don't have one it's very easy to do just go to AWS and create an account and once you're in there we have a couple of pieces that we want to put together so we need to create a Lambda function a Lambda function is just like a block of code that sits there until it's fired so something triggers it it fires and it does some kind of processing in AWS eventbridge is a place where you can set rules based on actions that happen in AWS that trigger Things based off of those actions so if an ec2 instance is created you can set a rule and event bridge that whenever that event happens it triggers a Lambda or it sends a text message or it does something in S3 it's hundreds of things you can do with it so those are the three main parts so let's start by creating this Lambda function so let's go in AWS up here in this search bar just type in Lambda and open that service let's go to create function and let's call it tag ec2 instance and for runtime we want to choose python because that's what we want to practice today so create function now down here we have our default python code we're going to leave that for now but what I like to do when I create a Lambda is to go ahead and create a test for it and it's going to be a blank test so if you go here and configure test event I'm going to name it my test and erase all of these keys and values and just save that we have a blank test and we can click test and it will show you the results here all it does is return status code 200 in a body hello from Lambda but the reason I'm doing that is because once you run a test it creates a law group so if I go to Monitor and click on view Cloud watch logs it creates this Law Group so now every time it runs we have a log of that run and by running it that first time we create the group providing those logs that's the only reason I do it so now we should have a group and the first log in there and we'll need that log in a few minutes to write our code all right so we have our Lambda function we'll leave that for now I'll just go back to code and leave it and we'll put all that together later so we've created our Lambda function now we need to create an event Bridge rule that's triggered when an ec2 instance is created now the eventbridge rule you'll see this in a minute we're going to use requires that we have a cloud trail trail in place because we're going to be using the cloudtrail API for this event so before we create this let's go ahead and create a trail so I'm going to go back to my Lambda screen and up here in the search box just type in cloudtrail and click on that track user activity in API Usage Now many of these events don't require you to have a trail but the one we're going to be using does that's why we're doing this and again I'll show you that in a minute all right so just go over here to create a trail we'll call it my trail and you have to store the logs in an S3 bucket so I'm going to create a new S3 bucket it gives me a name I'm just going to go with that um SSE KMS encryption I'm just going to disable that I don't want to deal with that right now in Cloud watch logs I'm going to enable and I'm going to leave that law group name and I have to create an IAM role for that I'll create a new one and let's call it whatever it says here cloud trail rule for cloud watch logs cloudtrail roll for cloud watch logs and this just gives it the correct permissions um it creates it for us so this is not a big deal and I'm just going to add these numbers on the back of this to keep them in sync and I think that's it so go to next I have a trail with a name already let's call it test Trail and go to next and then we're going to leave the event type management events API activity we're going to leave it read and write click next and everything looks good and create Trail okay so we have a new Trail called test Trail so back to our diagram now we're good to create this event Bridge so up here in the search box type in event Bridge Amazon event Bridge which is a serverless service for building event driven applications go to create rule or over here under buses you can click on rules and create rule we're going to call this ec2 launch description I'm going to leave blank event bus I'm going to leave default and enabled and for the rule type I'll choose the rule with an event pattern click next and then down here at the very bottom we're going to set up our event pattern so AWS Services as for the service we're going to choose ec2 because we want this rule to fire when an ec2 instance is created so ec2 and then it says event type the type we're going to choose here is an AWS API call Via cloudtrail this is why we had to create the trail if you want to have a rule for every time an instance starts and stops you don't need that trail you can just choose this ec2 instance State change notification and then you can choose whenever it changes to a certain state but we're not going to do that we need more information we need to use our information and so we have to go to cloudtrail for that so choose AWS API call Via cloudtrail and then it can be any operation but we want a specific operation called run instances now where did we get run instances from well let me show you real quick if you go um to ec2 instances I'm going to open this and I'm going to launch a new instance I'm going to call this my instance and just choose all the defaults I don't care about a key pair network doesn't matter Security Group I'll just select one and launch instance and if I now go to cloudtrail it'll show me everything that just happened and it usually takes a minute or two so I'll pause this and come back when it's done so if we click on event history we can see a few things so when I created that instance it triggered an event called run instances that's what we're choosing over here an event Bridge Run instances and you can see I stopped the instance here so it says stop instances so if I wanted this rule to be whenever an ec2 instance is stopped I could use that but we're going to use run instances so you'll see when I type that in it kind of sets up this event pattern over here sources ec2 here's the detail type as far as the detail Event Source is ec2 and the event name is whenever run instances happens so I'm going to keep that choose next Target what's going to be my target well we want the target to be the Lambda function every time this rule is matched it's going to trigger a Lambda function so select targets to invoke when an event matches your event pattern so let's type in Lambda Lambda function and then we want to choose the function we created which is tag ec2 instance so choose that click next we don't need any tags and everything looks good click create Rule now once that's created it should on our Lambda function create that trigger so let's go back to Lambda click on tag ec2 instance and we should now have a trigger so here's our event Bridge trigger so it automatically added that for us so if you look at our diagram when an ec2 instance is created we have the rules set up that should trigger this Lambda so the next step is we need to write the code in the Lambda to go back and tag that ec2 instance with whoever created it and to do that we need a package in Python called Bodo 3. so if you type in Moto 3 and go to the documentation it's pretty easy to use Bodo 3 is the AWS SDK for python you use it to create configure and manage AWS services so let's do a quick quick start actually we don't need to do this because it's going to tell us to install as if we're doing it on our own machine but when we're in a Lambda function we already have it we can just import it so I'm actually going to go to code examples and click on Amazon ec2 examples and just choose one of these and explain to you here what's going on so right here start and stop instances so we can get an example of how to use this or how to configure this I guess is the first thing we want to do so with Moto 3 you just import it import moto3 so let's do that import Json I'm just going to leave that import moto3 and we need to set a boto3 client so the ec2 client so boto3.client ec2 this allows us to interact with the ec2 service so let's set that and now that we have access to that ec2 service let's go find that documentation so go to available Services find ec2 ec2 click on that and we'll get information here so what do we want to do well we want to create tags we want to create tags on an ec2 instance so let's see if they have that these are the available methods so let's go to uh create tags here we go so ads are overwrites only the specified tags for the specified Amazon ec2 resource or resource so what we can do is do client which we've already defined client dot create tags and resources this is going to be our ec2 instance this is the instance we want to create the tag on and here's the tags we want to add to it but before we do that we need to get the information the ec2 instance information so in a Lambda function they have this data that comes in called event this is going to be data from the event that triggered the Lambda basically so let's get rid of this and what we're going to do is just do a print event just to see what kind of information we're getting so to update your function just click deploy and now let's create an ec2 instance make sure that it fires this Lambda and then check out what information we get from this event so instances launch instances and let's just call it a test instance we're going to keep the Amazon Linux instance type let's choose a T2 micro because it's free tier eligible key pair we don't need one we're not going to SSH into it network settings I don't really care about the settings here I don't want to create a new security group so I'm going to choose one but it doesn't really matter we just want to tag it we don't want to do anything else with it so after you do that just click launch instance and that should trigger the function so remember we have these logs these Lambda law groups that give us logs every time a Lambda function is invoked so we should have a new log here if everything worked properly yep we have a log right here in 147 so let's click that and see what kind of information we printed out so here's the event information this is the event that triggered the Lambda function we just printed it out let's take a look at it so let's select all of this and then let's go to Google and type in Json format and click this one paste it in and just format this mass of data quick process and it says it's invalid Json doesn't really matter just grab all of this we're just going to take a look at it we don't need it to be perfect and let's go back to air Lambda and then just paste it down here on the bottom and let's look at all this information so something didn't paste properly so let's undo that copy it again and paste it down here so what information do we need we need the instance ID that was created and we also need the username that created it so let's see if we can find that in this data so under detail let's see yeah under detail here's user identity and then we could do the Arn and slice off the user here but I'm just going to do username which is travis.media that's who created this so if we were to go to IEM and look at our list of users there should be a user called Travis dot media so let's get that information so how do we actually access that data well let's start out with a variable called user equals event and to access data we use these square brackets this is Json data we use square brackets so event what do we want to access well we want to access the details section so let's do square brackets detail and then from there in this details section the next level is going to be user identity so we need to access the user identity so detail user identity and once we're in this section so now we're in this section user identity we want to access the username so let's do username and that should give us in this example travis.media so that's going to get us our username which we're going to save in the variable user now we need to get the instance ID so if we scroll down further down to response elements and then instance set then there's an array of items which is just one here you can get the instance ID so this is what we want this instance ID that was created so let's start at the top so we have the Json here inside of this detail is going to be all of this information so we got to use detail so let's create a variable instance ID equals event and then detail so we're in the details section which is like almost the whole thing you can click this curly brace and it'll show you where the other one is way down here at the bottom yeah here it is so once you're in detail let's see what the next level is so detail and response elements and then instances set so response elements and then instance is set and then within instances set you have an items array so items and it's going to be the first item so we have to be sure to use that index which is 0 to access this first item so items and we want the first items let's use the zero and then instance ID we want the instance ID instance ID and that's going to give us our instance ID and to make sure that that worked we can test it and just print those out but I don't want to keep creating instances so we're going to run with that and just try to write all this and see if it works so now that we have the user in the instance ID we can go back to our Bodo 3 documentation and highlight this example that they give us and if you want to know information on the parameters like what's required and what's not what the defaults are it's all down here here's all the information you need so let's go back to the Lambda and paste this in and I'm not going to do a variable I'm just going to call client dot create tags and get rid of dry run we're not using that but where it says resources we need to enter the instance ID so we have that instance ID and for tags we have a key and a value so the key we're going to put owner and then the value is going to be the user so the user variable so this is going to tag the instance with a key of owner in the value of whatever user created it and it's going to tag this instance and that's really all we have to do here so after that I'm just going to do a return statement and then just because we don't need to return anything in fact I think it returns it returns none so there's really nothing there that we need so so let's just take all this information that we put down here and just delete it let's grab that delete it and make sure all of our indentations are right and save it let's deploy it and try it out so let's go back to the instances screen and create a new instance so launch instances let's do my new test choose the t2 Micro Key pair like I said earlier doesn't matter any of this just select an existing group so you don't create a new one and click launch instance and let's see what happens before we even check our Lambda let's just see if the instance gets tagged maybe we'll have a win on the first go all right so the instance is running now if we go over to tags let's see if we have that information so it looks like it didn't work let's refresh and select an instance and try it again nope it didn't work so what we can do is we can go back to our logs and refresh and see what happens so click your latest log so here it says name error name client is not defined on line 13. so client.create tags this client variable is not defined so let's go back to line 13. right here client.create tags we copied and pasted this I already know the issue we copied and pasted it without realizing that we didn't assign this client to a variable called client we assigned it to one called ec2 so it needs to be ec2 dot create tags and by the way when you're in Bodo 3 everything's a method so once you have this ec2 client you can call Dot whatever's on this list so there are so there are creates there's deletes there's describes like a get there's enable there's modify there's all kind of stuff you just call it by doing client dot or ec2 dot whatever method you want here so I think that'll fix it we just had the wrong thing here we had ec2 defined up here and we had client down here so making these two the same should solve it so let's deploy that and create another instance and don't worry about costs here they're all T2 micros and we're deleting them right away just make sure you don't leave it running and forget so let's do um my other instance and Amazon Linux T2 micro no key pair and select an existing Security Group launch instance and now we should see the tags come up all right so my instance is now running I'm going to click on it go to tags and it's still not there and I think I know why too this is how it works we try it out if it doesn't work we tweak it until it does work and I think I know what the issue is it's going to be a permissions issue so if we go back to our logs and go to the latest yeah client error an error occurred unauthorized when calling the create tags operation you're not authorized to perform this operation so with this Lambda function from this Lambda function I'm calling create tags on an ec2 instance while I don't have permissions this Lambda doesn't have permissions to do that on a ec2 instance we have to Grant it permissions and we can do that easily by going to configuration and permissions and here you see that there's an execution role this is a role that's created when a Lambda function is created so you create a Lambda function it creates a default execution role so what we need to do is we need to add permissions to this role to create tags on ec2 instances so if we click this role it's going to take us to that role page where we can alter its permissions so click this role and you'll see when a Lambda function is created this role gets the permissions to create a Law Group and to create a log stream and to put log events and we've looked at those logs during this video so let's add something to it so click edit policy so we'll go to add additional permissions choose a service we want this to be ec2 because we're adding permissions to create tags on the ec2 service so ec2 which actions will go to tagging and click on this drop down and click create tags we want to give it permission to create tags and then resources select resources to restrict access we're going to choose all resources and that's it just go to review policy and save changes and now we've added those permissions to this role so up here it says allow easy to create tags on all resources so now if we go back and create another instance I'm actually going to delete these three just click the three and terminate instance to get rid of those now I'm going to launch a new one called my final instance I'm that confident and all of the same settings no key pair and select an existing Security Group and launch instance now when this comes up we will have tags all right my instance is running click it go to tags and we should have the owner the owner is travis.media and so the benefit here is that for your company all servers that are created all ec2 instances that are created are now going to have the owner tag attached to great job you've used python to automate services in AWS if you found this helpful give it a thumbs up consider subscribing and I'll see you in the next video oh let me know Down Below in the comments what you'd like to see next see ya

Channel: Travis Media

Views: 20,841

Keywords: python and aws projects, aws automation, boto3 tutorial, python boto3, aws python tutorial, boto3 aws python, devops automation, automating aws, aws devops, aws devops tutorial, python lambda function, cloudtrail, eventbridge aws, python to automate aws, lambda function aws, ec2 aws, add tags to ec2 instance, travis media, travisdotmedia, travis media python

Id: 3DRiruDUhiA

Length: 24min 9sec (1449 seconds)

Published: Wed Jan 11 2023