Automate AWS EC2 with Python | DevOps Coding Challenge

what's up everybody it's Travis here from Travis dot media today we're going to be having fun with AWS in Python I thought it'd be fun to come up with a hypothetical challenge some kind of real life scenario and see if you can figure it out give you some practice with python in with AWS so if you're not in AWS yet check out my last video I walk you through how to get started how to set up an account how to set up a user get your CLI working all of that but if you're in AWS and you're interested in Python and you want to practice your automation devops skills this video is going to be for you and if you're not very good with python I think you can still figure this out I'll put some links Below on my python course recommendations but I think it should be okay so here's the scenario today you're at a company in your ec2 instances keep getting stopped no one knows who's doing it or who keeps forgetting to start these back up but your team lead needs someone to create automation to check all of the ec2 instances every hour and start the ones backup that are stopped so today's challenge for you is one every hour you need to check all of the ec2 instances in two if they're stopped or in a stopped State you need to start it back up if they're in a started State you just skip it that's your task it's pretty simple and I'll go over the architecture for you kind of a pseudo code and then have at it or just follow along with me and we'll do it together let's get started so the first thing we need to do is map out how this is going to go down like a architecture pseudocode and so just go to draw dot IO that's what it used to be called now it is app.diagrams.net it'll redirect you and from here we can map out what we want to do so we need to create a function to check the state of the ec2 instances whether they're stopped or in a start State and to do that we'll just use a Lambda function so go here type Lambda take the Lambda symbol and drag it in so we're going to create a Lambda function that's going to trigger this query and we're going to be using python so we're going to use boto3 which is the python SDK the AWS SDK so go ahead here type in Bodo and bring that in and we're going to have an arrow from Lambda to moto 3 and this is just rough this is not something I would give to my senior developer or senior architect but it's just rough for this video so draw this Arrow we're going to use boto 3 and boto 3 is going to Ping the ec2 instances so just drag that over and we'll draw an arrow here from Bodo to ec2 and remember we want the Lambda to fire every hour and to do that we're going to use event bridge and so let's grab that just put it wherever and let's drag an arrow from there to the Lambda function that's going to trigger a Lambda then let's put some text here that just says every one hour so that we know what we're doing and I'm just going to leave it at that this is just a rough drawing to give us some direction feel free to make it better now with this information do you think you can do it yourself if so feel free to go ahead create your Lambda function import Moto 3 talk to the ec2 instances if there's any stopped then start it and then create an event bridge to trigger that Lambda every hour give it a shot or follow along with me either way so the first thing we want to do is just create this Lambda function let's start with that so let's go to AWS click on Lambda let's create a new one and call it uh start ec2 instances and runtime is going to be python let's just choose the latest supported 3.10 and create function now before we write any code let's make sure we have permissions to talk to ec2 instances so let's go to configuration and permissions and you'll see we have an execution role called start ec2 instances roll and then some random letters so click on edit and down here where it says view the start ec2 instances roll just click on that and we're going to add the permission to talk to ec2 instances and then click on the policy and you see by default this Lambda function has the permissions to create a log group create a log stream and put log events so that the Lambda function can log events so let's add to that just go to edit policy and then add additional permissions and choose a service let's say ec2 and you want to be granular here but for sake of time I'm just going to choose all ec2 actions and scroll down and choose all resources so all actions all resources for sake of time and then review policy so ec2 full access to all resources save changes this is going to allow our Lambda function to talk to ec2 instances to do whatever it needs to do to query or to start or stop instances and again you can get granular you can give it read only and then just the ability to start and stop instances so feel free to do that so our permissions are good let's go back to our Lambda function and so in our code we have a Lambda Handler to do Implement let's put our code here so I'm going to go up to the top and do import O3 and I can get rid of this Json because I don't need this return statement for what I'm doing you can leave it doesn't matter but I'm going to keep it pretty minimal so how do we use Bodo 3 well we go to the documentation so boto 3 and click on the documentation and it's always good to just go to Quick Start and here you can install Bodo 3. we don't have to do that in Lambda functions we can just import it so configuration we don't have to worry about that and here's an example so import Moto 3 and then let's use Amazon S3 S3 equals bodo3.resource and then the S3 resource so I'm assuming we can do a boto3.resource ec2 so to use Bodo 3 most important and indicate which service or Services you're going to use now that you have an S3 resource you can make and send requests so I'm assuming we're using resources here so click on resources and you can read all about that there are identifiers and attributes there are actions there are references sub resources waiters all of that so I'm going to take this example that we had before like S3 I'm going to copy that and paste that first and just change it ec2 equals bodo3.resource ec2 and let's go ahead and comment this create ec2 client and now that I have access to the ec2 resource I need to Loop through the instances and get the state and if the state equals stopped we need to start it that's basically what we're doing so let's do for instance in and then again what do we do so let's go back to boto 3 and figure out how to Loop through things so back here on this resources page like I said you can get some good info but we want to know how to get a collection of instances how do we do that using this resource interface and if we look over here to the left we see resources we're on that tab but if you go down to you'll see one called collections so click on that and you get an example for Q in sqs.qs dot all so we're calling all on the resource you get another example for bucket in s3.buckets.all so I'm assuming we can do ec2 dot instances.all so let's go for instance in ec2 Dot instances.all and what do we want to do well we need to get the state of each instance so let's go back to our documentation and we're not going to find it on this page what I like to do is go actually into the API to that resource and see all of the options I have to call on that resource so I'm going to scroll down go to available services and here are all the available AWS services so let's find ec2 and here's ec2 and we have four options here we have client paginators waiters and resources let's click on ec2 and we start out with client the client is a low-level client representing AWS resources so it's a low-level client and it has tons and tons of methods for each AWS resource I used to use that all the time to get very fine grained details about these resources but today we're just keeping it simple we just want to know the state of the resource so I don't really have to get that detailed so let's keep scrolling down and we get to um paginators we don't need paginators or waiters we want resources resources are available in Moto 3 via the resource method and that's what we use from the beginning remember we did moto3.resource ec2 so we're going to roll with this and we want to get info on the instance and then from there we want to find out the state of the instance so let's go down to attributes and click on State and we have examples we have pending running shutting down terminated stopping and stopped and that's ec2 dot instance dot state so let's try this let's do print instance dot State let's see what that gives us and we haven't deployed this function yet so make sure to deploy it and then we have to create a test let's create a test for this and I'll call it my test and just delete all these things we don't need any of that and Save and now we can run this test and see what the state would be but you've probably figured this out we don't have any ec2 instances so let's actually go over here to services and create a couple ec2 instances we're not going to get anything back if there's nothing there right so go to instances and go to launch instances and I'm just going to say test instance and choose Amazon Linux we're going to go to instance type and choose T2 micro because it's free tier eligible if you're in the free tier you pay nothing if you're finished with the free tier that's all up you're going to pay like one penny per hour so if we if you do this and it takes you an hour and you spin up five instances you're gonna pay five cents so don't be worried about costs here just up here where it says number of instances choose five you can choose three or two it's up to you but I'm gonna choose five key pair I have a key pair I'm going to select that VPC I don't really care about I don't need to ssh in anything so I'm going to select a security group doesn't matters just do the default and launch instances so let's launch these so once these get in a ready State let's go ahead and stop one so I'm actually going to stop two of them I'm going to select these two instant State stop instance so we have three running two stopped now let's go back to our Lambda and print out the state of these instances so let's click test an error message task timed out after three seconds so let's go to configuration that's default let's go to configuration and general configuration and see how by default it gives you three seconds we just need to up that to like 10 seconds or something I'm gonna put 20. and go back to code and try it again test and there it is we have five instances two of them were stopped but we have these key values here so code 16 name running we just need the name value so let's go back and just add State and then put in Brackets state I think that's what it was no name sorry name and that'll give us the name of the state so deploy that test it again and running running stopped stopped running that's great so let's clean this up a little bit let's do state equals instance dot State bracket new to access that value not new I want to keep doing that name now we need to check if it stopped if it stopped we need to start it so if State equals and is that capital S it's lowercase all lowercase if State equals stopped then we need to start instance so how do we do that well let's go back to our documentation go back out of state to our instance resource and go to actions actually let me go back one more let's go to instance and actions so actions let's find start so this starts in ec2 instance ec2.instance dot start pretty simple and you have some parameters if you want to run it I don't need them but we're just going to put instance dot start if state is stopped and since dot start Let's test that out so deploy it and let's see if our instances start this time so test and let's check our instances refresh and we have pending so they are starting back up so our code Works refresh again and we are running so let's just update that to do a try accept so if State equals stopped try instance.start and if there is a problem we can just log it for now accept print something went wrong and deploy that test it again to make sure no issues come from that and that's it our code is done that's all we need to do now let's go back to our diagram the Lambda function is done we use boto3 we can talk to the ec2 instances now we need to trigger that Lambda every hour so we need an event Bridge so go back to your Lambda scroll to the top and you'll see add trigger click add trigger select a source the source is going to be eventbridge which is cloud watch events create a new rule I'll call it start ec2 every hour and we're going to do a schedule expression not an event pattern with a scheduled expression you can do cron jobs and it says self trigger your Target on an automated schedule using cron or rate Expressions so click on that to get an idea of how to create or how to configure this cron so the syntax is cron and then some things inside of this function so let's scroll down we get some examples run at 10 A.M every day run at 12 15 every day we get some examples here's one that says run every 15 minutes so let's just change this to every 60 minutes so I'm going to go here and type in cron and in here we're going to put so we got 0 15 star star star question mark star so we'll do 0 60 star star star star question mark star that's going to give us every hour and we made a big mistake we put that in rule description take that out put it down in schedule expression wrong place now click add and we have a trigger for every hour and it's enabled so every hour air Lambda is going to fire and make sure all of the ec2 instances are started hope you found this pretty easy and fun if you like this kind of thing let me know and I'll do more also consider giving this video a thumbs up subscribe and I'll see you in the next video

Channel: Travis Media

Views: 13,922

Keywords: boto3 tutorial, aws automation, python boto3, boto3 aws python, devops automation, boto3 aws, boto3 python, python devops, boto3 tutorial python, boto3 lambda tutorial, aws lambda python boto3, aws boto3 python tutorial, aws ec2 automation, aws ec2 lambda, aws lambda ec2 automation, aws eventbridge, aws eventbridge trigger lambda, aws python automation, automate aws with python, lambda automation, travis media

Id: h66RrCn5rvQ

Length: 16min 24sec (984 seconds)

Published: Tue May 02 2023