Intro to Packer with AWS

and here's the packer builder ec2 instance that's currently being set up and if we track the logs over here we can see it's just waiting for the instance to become ready so we can start running all of those provisioners [Music] are you still creating machine images by hand wouldn't it be great if you could automate the entire process in this video i'm going to show you how to use packer with aws to create amazon machine images packer can be used to create machine images for so many different platforms but i'm just gonna focus on using it with aws in this video so for this example i'm gonna be creating an ami that hosts a basic web app where every time you refresh the page you get a brand new cocktail and it's a super basic node application so all i really have to do is install node and make sure the application is running and i've talked about creating amis and using a mutable infrastructure in a different video that i'll link in the description i think you should check that out first if you haven't already seen it but basically i want to set up an ami that runs this application so then i can create brand new virtual machines from that ami and it's easier to horizontally scale and when i have a new update to my code i just create a new ami and then update my infrastructure and when i did this by hand i had to log into the aws web console create a new ec2 instance log into that instance using ssh install the necessary software set up the service create an ami from that instance and then tear down that instance and that's a little bit time consuming but there's also room for human error because if i'm doing that over and over again it's likely that at some point i'll get something wrong configure something incorrectly and everything will just break so what i'd like to do is automate that entire process so all i have to do is run a single script and the script just manages all of that for me and that's where packer comes in because we can specify the configuration we need in some code files hand those files off to packer and then packer handles everything else so packer will actually set up a new ec2 instance uh install any software get any code files on there and then create the ami from that instance for us and we can run this as many times we want to create as many amis as we want so every time we have a new update it's very easy we just run a script and our new ami gets created for us so let's create an ami for this cocktails app using packer and before we can start using packer there's a few things we have to do the first is actually installing packer and you can just go to packers website and look for the installation instructions for your machine for this and since we're using aws you also need to make sure that you've downloaded and configured the aws cli on your machine and once you have those two things done you're ready to go i have an empty demo project directory here that i'm going to do everything in for this video and the first thing i'm going to do is create a new directory and i'm just going to call this packer and when we're using packer like i said we're writing code so we're going to treat this as a code project we're going to put files in here i'm going to edit it with vs code i'm going to track versions of it using git so you treat this like a normal code project so i've opened this up in vs code and i'm going to create a new file called so this is for a cocktails app so i'm just going to call this cocktails.pkr for packer dot hcl for hashicorp configure language and hcl is the language that we use with some of the hashicorp tools like packer and terraform and it's a really simple language that's built up of mostly blocks and arguments so it's not that difficult to learn and when we're creating a pack file to work with aws there's three main blocks that we need to include there's the packer block the source block and the build block and the packer block is where we specify any settings that we want for packer and here is where we will say uh we are using aws and the source and the build blocks are the much more interesting blocks because this is where everything actually happens so like i said packer when we run this file is actually going to set up a new ec2 instance it's going to do a whole bunch of stuff on that ec2 instance and then create a brand new ami from the instance so the source block is where we tell packer which ami to use as the base so we might want to start from ubuntu or amazon linux 2 or something like that and where to save the ami so uh the starting point and the endpoint all go in the source block and then the build block is everything in between everything in between so uh what to install uh configure uh files to copy over to the instance just everything that would happen uh after you log on to the instance so let's actually fill out these blocks now uh so the first one i'm just going to copy and paste this is really never going to change for me as long as i'm using aws uh so i'm telling packer that we have a required plugin that is the amazon plugin because packer is the application that actually does everything but we need to require the plugin for the specific platform that we're using in this case aws and this will give it access to aws resources like amis and ec2 instances and we can google the amazon plugin for packer and we can see the different options that we get when we import this plug-in so here are the different builder options we have when we're using aws and there's four different options but uh amazon says if you're in doubt just use this first builder which is amazon ebs and this just creates the ami in the way of explain by first creating an ec2 instance configuring it and then creating the ami from that so in the source block we're going to write amazon ebs this is the way we're going to create the ami and then i just give it a local name so i don't know i'll just call this tails for now this is just so it can be referenced later within this packer file and again this is just kind of standard setup for any time you're using aws to create an ami and the hashicorp documentation is usually really good so if you want to see all the different options you have all the different things that you can do with packer i do recommend just looking at the documentation for right now within this source block i'm going to specify just a few arguments here uh this is going to be for a really simple setup you can specify more arguments if you want you can check the docs but i'm just going to go for a really simple implementation here so i'm going to need the ami name this is the name that is going to be given to the ami once it's created i need the source ami this is going to be the image that we build the ec2 instance from so maybe we use ubuntu or amazon linux as the base instance the instance type this is going to be like t2 micro or something the region that i want to create the ami in because all amis are region specific and the ssh username this is the user that packer will log in as to configure the ec2 instance before it creates the ami so i'll just start filling these out uh the ssh username is going to be ec2user because i am going to use an amazon linux2 base image the region i'm going to use us west 2 the instance type i'll use a t2 micro the ami name i'm just going to call this i guess tails app and then the source ami this is the id of the base ami that i'm going to use and i don't actually know this off the top of my head so i'm going to log into aws to show you how you can grab this so if you navigate to the ec2 dashboard there's a couple of different ways you can do this i'm just going to click launch new instance because the first step here is you can select one of the base amis and if i look at the top here i want amazon linux 2 maybe you want to use something like ubuntu or red hat but the ami id is right here for 64-bit x86 so i can just copy this id straight into packer and now it will create a new ec2 instance from that ami id using t2 micro in us west 2 and then it'll do a whole bunch of stuff and create a new ami called cocktails app so this is all i need right now in the source block to get this working and then i've got the build block which is where i tell it to do everything so i need it to copy the source code of my node app onto this instance i need to install node i need to set up a new service to run the node app and that's actually everything for now i might also want to set up like a cloud watch for logging or some other different things but this is a really simple one and the first thing i need to put in the build block is a sources argument which is an array of sources and i only have a single source here it's this amazon ebs and i can reference this just using dot syntax here so i can say source dot amazon ebs dot cocktails that's where that local name comes in and we could have multiple sources here i could be creating multiple different ec2 instances different amis i could even use different cloud providers here so i could have one that's creating an image for amazon another one is creating an image for google cloud all within the same file but in this simple one i'm only creating an ami for aws and then the other type of block that's going to go in here is a provisioner block and these are the things that actually do all of the heavy lifting so a provisioner can copy a file or run a shell script or run some sort of other script uh maybe with chef or puppet or ansible so these are really powerful and if we google uh packer provisioners provisioners and here you can actually see a list of all the provisioners and realistically i probably want to use a good tool like ansible but for this just to keep it simple i'm going to use a shell script and to get the code onto the virtual machine i'm going to use the file provisioner too so there's a bunch of details here on how to use them a bunch of good documentation if you want more details i'm just going to show you how to use it here so i'm going to type in shell so i'm going to use that shell script provisioner and i'm actually going to pass in a bash script that it can run on the ec2 instance sort of say script equals and then i'm going to pass in the path to the script so i'll just call this uh maybe app.sh this is just a bash script that i'm going to create here within this current directory so i'll just create a new file called app.sh in the same directory as my packer file and this is going to be a bash script and this is going to run on the ec2 instance to configure everything and there's a little bit of boilerplate code that you should put in the top of every ec2 instance script file uh the first is a sleep command to sleep for 30 seconds packer recommends using this because it might take a little bit of time for the ec2 instance to be fully set up and it doesn't want to run this code too early and the other one is just to run a quick update so since i'm using amazon linux 2 that would be sudo yum update dash y like i said this is running a node app on the ec2 instance so one of the first things i need to do is actually install node so i'm just going to run this code here to install node version 14. so if we ran this packer file right now it would do exactly that it would set up a brand new ec2 instance run this script install node and create an ami from that so this already will create a new ami that has node.js installed but that's kind of useless because i really do want that application to be running on it so the next thing i need to do is actually figure out how to get my source code onto the ec2 instance and there are so many different ways of getting code onto a virtual machine and it will always depend on your continuous deployment process you probably want to have some sort of trigger setup on your version control system so that once code ends up in the main branch it triggers a new build of packer and packer creates that new ami for you and since there are so many different ways of doing this i'm just going to use a basic file provisioner so what i'm going to do is i'm actually going to pull my code from github onto my local machine i'm going to zip it up and then upload it to the ec2 instance using this file provisioner and the reason i'm doing it this way is because it's a simple way that will work for pretty much everyone no matter where your code is hosted no matter what your pipeline is you will be able to download your code this way so i'm just going to grab this application from github and i'm going to clone this into a different directory i'm not actually cloning this into my packer directory because it's not part of my packer project my code project and my packer project will be separate they will be committed into git or whatever your version control system is separately they will be tracked separately so if i look at my directory structure i have my packer directory which has all my pack stuff in it and then i have my cocktails directory which is the cocktails app and i'm just going to zip this so it's easy to upload with packer so let's zip cocktails there we go all right so i now have cocktails zipped up and i'm going to use this file provisioner to copy over that zip folder to the virtual machine so the source is just the parent directory slash cocktails dot zip and the destination slash home slash ec2 user slash cocktails.zip i'm just gonna put it in the home directory uh and i need equal signs there there we go so now this will just take that zip folder and put it onto the ec2 instance for us and these provisioners will actually run in the order that we provide them so i should run this file provisioner first because packer will copy this zip folder onto the ec2 instance and then in my bash script here i can actually manage that so i can say i'm going to cd into the home directory and unzip the cocktails.zip folder then i'm going to cd into cocktails and mpm and install all of the dependencies uh only the production one this is pretty standard get the source code of a project onto the instance and then install all the dependencies and i should probably actually make sure that uh yum is installed know that unzip sorry is installed on this so i'm gonna make sure this is installed unzip the cocktails folder and install all the dependencies now i just need to make sure that my application is running as a service so as soon as i start the ec2 instance my cocktails app should be running and i have a video explaining how to deploy a node app onto an ec2 instance if you want to watch that for more details on what i'm doing but i'm just going to use systemd to make sure this application is running so i'm going to create a new file again in my packer directory and i'm just going to call this cocktails.service and i'm just going to create a new service file here and running this file with systemd will just make sure that my application is always running so i have this file here locally and again i need to get this onto my server so back in my packet directory i'm going to create a new file provisioner and i'm going to grab the cocktails dot service file and i need to get this service file to slash etsy cocktails dot service because this is just where systemd keeps its service files but this directory is actually owned by root so when packer logs in as easy to user it actually can't even access this directory so even though we need to get this file here packer can't do it for us so instead what we need to do is upload this to the temp directory or somewhere that packer can actually access and then when we're in our shell script here we can use sudo to actually move that around so if i grab this let's see there we go so here's where we're uploading it to and here's where we need it to be uploaded to so the pekka file will get it to this location and then with our shell script we can say sudo move it from this location to this location and this is the way packer suggests doing it every single time you need to get a file onto the virtual machine in a directory that is owned by the root user so once i have it here i can now run sudo system ctl enable cocktails.service and sudo system ctl start cocktails.service and this will make sure that the application my node application is always running every single time i create a brand new ec2 instance from this ami now if we review everything we're going to create a new ec2 instance from the amazon linux 2 ami we're going to copy over the source code copy over a service file and then run this batch script to install node install the dependencies and actually run the service file so if all of this goes well if i've done everything correctly here we should end up with an ami running this application so now that i'm ready to do this i'm going to head up back over to terminal and cd into my packer directory this is the directory that actually has the dotpacker.hcl file in it and the first thing i'm going to run is packerinit in the current directory and what this will do is it will find the pack of files and it will install any of the required plugins now since i already have the amazon plugin installed on my system because i do this frequently enough packer init did nothing but you might need to run that to actually install the dependencies then i can run packer build and i'm going to pass in the file here this will validate the file check that everything's correct and then it will start actually doing this whole process so it's going to create a brand new ec2 instance and run all that code and we can actually see this running so i put this in us west 2 and here's the packer builder ec2 instance that's currently being set up and if we track the logs over here we can see it's just waiting for the instance to become ready so it can start running all of those provisioners so that took a couple of minutes but now i can see that packer has successfully finished doing everything and it has created a new ami and if i look back over at my aws console here i can see that the ec2 instance it used to create everything is now being terminated so it's managing that for me and if i go over to amis here i can see that in my custom amis i have this new cocktails app ami that was built by packer so i could even actually uh use this ami now i can launch a new ec2 instance from this and i'll just do this very quickly i'm just going to use all the default settings and this is running a node app uh just on port 80 so i or 8080 rather so i'll just create a security group for that and i don't need a key pair here there we go so if everything worked if i actually got my source code on correctly run it as a service correctly uh this ec2 instance should just be running my app as soon as i visit the ip address at port 8080 and i just have to wait for this to actually be running first okay so i'm going to copy this public ip address now that it's running and i'm going to visit it at port 8080 and it's not working and the reason it's not working is i actually messed up this configuration so um i have created a directory called cocktails which is where my source code exists but in my service file for some reason i called it cocktails dash app so now this should work but since i've made a change to this i need to build a brand new ami and there's no updating with packer it's always creating a brand new resource that will create a brand new ami for me so if i go back into packer i can run the build command again because again i want this to just create a brand new ami and this time it failed because the ami name cocktails app is already being used obviously i just created an ami from that called cocktails app so to get this to work i'd actually have to go and either delete the ami from aws or come up with a brand new name here and what we should really do is have a unique name created every single time i run this pack a build command so that i can keep generating new amis every time there's a new update to my code or every time i need to do anything new and it will just work i won't run into any naming collisions so if we go back to packer.io and go to tutorials getting started with aws build an image again these tutorials are great i do recommend going through and reading through these for more details but in let's see variables the variable section here we go they have a nice little example here of a local variable block that creates a timestamp variable that we can use to generate a unique name so in this ami name section i'm going to call this cocktails app and i'm just going to go dash and throw in this timestamp here so i'm always going to start all my amis for this application are going to be prefixed with cocktails app but then it's always going to have a timestamp on it so if i generate a bunch of these they can all exist together and i'll be able to see which one's the newest one just based on the timestamp so i'm going to run that build command again and i can't just write timestamp i have to write locals local variable.timestamp nope it's not locals it's local singular i should have just read the docs there we go okay so now it's going to generate a brand new ami and you can see the ami name here is cocktails dash app dash and then here's that timestamp so i can keep running this packet build command as many times as i want and it will always generate a brand new ami with a unique name and i'll have to wait like another five minutes for this to run while that's happening though i'm going to go back to this packer tutorial here and pick up another block of code here this source ami filter i'm going to copy this into the project because right now i'm hard coding this ami id so all the other arguments here are fine but this source ami is hard-coded in a bad way because this id is specific to the us west 2 region so if i want to build this exact image for us west 1 or a different region i would have to go and find the ami id for that region and copy and paste it in here the ami ids are also attached to the specific version of the ami so if my base ami has an update for security reasons or whatever i'm always going to have the older version of the ami rather than the newer one and that's generally not preferred so instead of hard coding the ami like this we want to use a source ami filter where we can specify a certain number of arguments here and we can have packer actually figure out what the idea is so i'm going to comment this out so in this example it's using ubuntu and it's just going to grab the most recent ubuntu image i still want to use amazon linux 2. and i'm going to show you one way that you can figure out all of these arguments in aws is kind of a bit of a backwards way but if i go back and uh find that ami id i'm kind of doing this completely backwards so i know that i want this amazon linux 2 instance i'm just going to grab the ami id and i'm going to go back and i'm going to go to this ami section here and i am going to do this drop down to public images so this is every single ami that is public on aws and here you can find things like the id the source the owner visibility device types things like that so i'm going to paste in that id and i'm going to get all of the details about the amazon linux 2 ami so here is the name given to it i'm going to copy that into here the root device type is ebs the virtualization type is hvm we don't really have to worry about that right now there are more details on packers website if you're interested in those details um the owner is amazon let's see owner yes just amazon lowercase and i want the most recent so using this i will get the most recent version of this ami uh the problem here is that i'm getting amazon linux 2. um all of these details here are versioning details so i don't actually want to get a specific version i just want to get the amazon linux 2 ami that is for x86 64. so i will delete all of this code here and just replace it with a wild card a star so no matter what version of the amazon linux 2 ami exists just get me the most recent one and this is what my source ami filter is going to look like instead of my source ami id and this will end up being the exact same thing as this in this case because this is the id of the most recent amazon linux 2 instance in us west 2 but now this will work even when there's an update to the ami and even if i change the region so if we look at this file again we have the packer block at the top where we can specify uh which platform we're going to use we can create local variables here in this case i'm just creating a timestamp the source block which is going to look more like this because we need to specify the ami name that needs to be unique so we're going to use that time stamp we always want to use a source ami filter not actually hard code the ami id and then we've got the type and region and ssh username and then this build block is just going to contain everything that needs to happen once the ec2 instance has been set up so i'm going to copy over some files run a shell script you might use different provisioners but it's always going to kind of look like this so this is a simple implementation of packer and packer has now built my new ami with that new ami name so if i go back to my amis and select owned by me i should be able to see there's my first cocktails app and then there's the second one with that new timestamp on it so i'm going to try this again i'm going to launch this ami and hopefully everything's going to work this time configuration storage tags security group yep it's going to run on port 8080 don't need a keeper so again this is just setting up that ec2 instance in the most basic way possible oh here's my previous one i should actually shut down so i just have to wait for this new instance to become ready and then hopefully this should be working on port 8080 and there we go so this one is now working every time i refresh the page i can see a brand new cocktail so i now have an ami and if i create a new ec2 instance from that ami it will have this application running now one issue here is that although i'm automating the process of creating the ami which is great i'm still doing everything else manually i'm still having to come in here and click buttons to create ec2 instances from the ami i'm now done with this so i'm gonna have to terminate this manually so if you want to automate all of this infrastructure creation and destruction you can use a tool called terraform which works really well with pekka and i'll be covering that in my next video [Music] you

Channel: Sam Meech-Ward

Views: 434

Keywords: hashicorp packer tutorial, packer aws builder, hashicorp packer aws, packer tutorial aws, packer aws ami, packer aws, hashicorp packer ami, hashicorp packer, hashicorp packer aws tutorial, aws packer tutorial, aws packer terraform, aws packer ami, packer node, node ami, devops tutorial, packer tutorial for beginners, devops tutorial for beginners, packer terraform aws, aws packer, infrastructure as code

Id: dde-bWUeRMo

Length: 26min 1sec (1561 seconds)

Published: Mon Oct 25 2021