Build Automated Machine Images with HashiCorp Packer | #TechTalks

[Music] okay all right so packer all packer is is a way for automating the way that you build an image so um it's pretty simple but it's really powerful if you use it with the certain workflows um and i'll demo that today but packer is actually kind of a competitor to configuration management tools like chef puppet and ansible hashicorp has their own vision on how infrastructure should be and packer kind of complements that and that's the beautiful immutable infrastructure you probably heard that term is kind of a buzzword um but essentially this top diagram is the traditional way that most companies deploy changes to their environment they have their server image they provision their image into an instance and then any change they make they apply it on top of each change with puppet or chef or ansible and some of the downsides for this is you have to have really good code coverage with your configuration management tool so like for your windows server you're not going to configure each and every setting and puppet and enforce all those settings you're just going to have the ones that are important to your company the other way of provisioning infrastructure is the immutable way which packer really complements um so here you have your server image and then you provision your instance and any change you need to make you go back and you change the image and redeploy that server into the environment so you don't have to have you know 100 code coverage under change because any sort of change you make is like rebuilding the server again um just kind of like how containers work it's like the exact same way um so that's kind of hashicorp's vision of how server infrastructure should be it should fall more in the lines of like your containers with your docker file and packer really allows you to do that and one of the cons to having this sort of method with your server instances is your application architecture has to support it so there's not really a clean way of doing this going back on your old software and then kind of revamping it into this architecture you kind of have to stick with the traditional method but a lot of people are starting to build um if they're building new software they they're starting to follow this art architecture and actually a survey came out i wanted to pull up a stack overflow for 2020 and uh chef and puppet are now some of the most dreaded tools and devops of this year so a lot of people are getting off of chef and puppet and they're going towards more of the immutable tools like chef and a chef like ansible and terraform so that's pretty pretty interesting you don't see uh packer at the top of the most loved list yeah that would be pretty cool so yeah a lot of people don't align uh hacker with like a devops toys it's kind of it's kind of been hidden for a while um but it's starting to get more presence now it's actually starting to grow quite a bit but yeah i assume eventually it will it'll make it on the list but yeah terraforms on there i guess the most loved one of the most loved so i went to throw packer in terraform because they're both they both work really well together and i'll show them in the demo i guess one thing for the labs perspective is that uh startup performance compared to configuration management would be much better with pacquiao right yes yep 100 because you're doing all the configuring in the image as opposed to after it's deployed so yep um so hacker is really it's just a binary um and you feed it a json file and that json file has all the configuration for building the image and it's really simple you have your variables your builders and your provisioners and your variables are the um the configuration settings that you want to like feed into your configuration so here um aws access and secret key i'm feeding that into my packing configuration um so i can use that here and the builder section is essentially it's just your configuration for the environment that you want to build an image in so here we have a aws environment and i have all the settings in order to build an ami in aws and down here we have our provisioners your provisioners are the configuration that you're applying to that image so essentially what packer is going to do is it will stand up in ec2 in aws and it will ssh into it and it will run these commands to provision it and then it will de-provision that ec2 instance and then make it into an ami so really simple and it just does all the heavy lifting to you so you don't have to script it out yourself for each environment so we'll go ahead and show so packer validate is the command to it's just like linting um it does static analysis on your configuration you just do tracker validate exponential and then if there's an issue it will actually error out so that works really well for pipelines um so bunch of here and i'll get it so yeah it's really um good to add that as a step into pipelines um and to do the build it's just packer build and then json file and now it'll reach out into aws deploy an ec2 instance and then it will run the these commands to install redis basically and then turn it into an ami one thing to note in this builder section this is really interesting so i can have multiple builders in here i could have multiple aws environments and i can push out a image into all those environments in parallel so here's a really good example um on their on their website they're they're pushing out a image to aws and digitalocean and then it's the same provisioner that i'm using to install redis on and it'll just do that in parallel when you run your packer build so you can manage one image in multiple environments with the same code which is really really powerful when you start getting into multi-cloud and do you know if the uh one of the builders is like a docker image yeah let me i was going to show this so builders their documentation is really good so these are all the builders that they have there's a lot this has grown a lot in the last year oracle hyper-v vmware i've used the vmware one um yep docker so yeah they keep adding ones and then you can create your own uh there's people that have been doing that so yeah they have a lot of different environments that you can build stuff in for sure does it always depend on ssh or would it work for some of those images no so your there's different provisioners that you can use right now i'm just using the shell provisioner to run shell commands through ssh uh you can also use chef so all you can take all your old configuration management scripts and then use those to build your images as well uh to reuse all that stuff so puppet ansible salts um inspect so you can transfer files to it yeah there's there's a lot of different things that you can do to build out your image however you need um so yeah this is wrapping up here it's stopping the instance creating the ami and then that's it we have our ami and aws now if we wanted to take this a step further we can create a ci cd pipeline with terraform so we could have packer build the image and then have terraform deploy the server instance and then as far as like uh routine management or if you wanted to add a additional package to the server image you'd go and make your change and then your ci cd pipeline would go ahead and build it and then provision it out automatically and we'll go ahead and show that real quick i have a repository in azure devops here and i have my terraform code for a server i have a web server that's sitting in aws that i deployed with my terraform code using an ami that was built by hacker so let's say we wanted to go ahead and add a new package in here we just make a new branch configuration also you can use um a script so i could have a script file within this images folder and just call that with the packet configuration that's kind of the way that i've always done it so then um i don't have to muddy up this configuration uh i can make it clean and just manage my script for provisioning um so change this to snap d change to this branch and we'll do a pull request this is a remote repository that's in azure devops password oh my branch is in azure devops now this is just like your standard workflow with doing a pull request when you change and just flip through it okay so now this pull request will trigger a pipeline build to start building out this image and then it will redeploy the server with the new ami image so terraform will look and see that that server exists and see that the ami that server used is different than the new one and it will destroy it and then reprovision it so how does it uh see that it's different like what's the connection oh yeah i'll show you that in a sec so here's my pipeline i broke it up into two different stages the build and then the deploy so the way that packer passes that new ami over to terraform so terraform is configured right now with a variable for ami id and that's passed in to the configuration for the ec2 instance and here on the packer configuration at the bottom here i've added this post processor so post processors with packer is anything that you want to do like after the image is built so here all i'm doing is i'm using the it's called the manifest post processor and that creates a manifest json file with any sort of attributes of the image that it created so it actually creates the ami id and here in my let me show the code here so here i'm grabbing that manifest json file before i run terraform and i'm saving that into a variable and then just passing it straight into the terraform configuration through here so i'm just updating the ami id with the new one so that's how you would pass that through um also uh post processors are pretty interesting you can do a lot of different things with them here's all the posts no here's all the post processors that you can do docker save push you can tag things after the image is built um this amazon import one is really interesting so here's an example they have where they built an image on vmware and then they use a post processor to actually push that up into an s3 bucket so you can do a lot of additional um crazy things with post processors in packard so that's kind of how that works right now it's still building so the packard itself doesn't have a state like terraform you can't do packer output ami id or no no yeah you have to use files you have to use a little duct tape and glue to do it but yeah it works pretty nice that you can output that stuff into that manifest file and then then yeah you just parse that through and feed it into your terraform configuration so um so yeah once this finishes it will just rebuild that this ec2 instance with the snapdot or snapd package so um yeah it's it's pretty slick because all i had to do is just modify the image file and then it just handles it so you can get really fancy and do like zero downtime deployment rollouts with um idc2 clusters and stuff so um yeah really powerful compared to like the traditional way of just trying to make changes on the fly and roll them out that gets really complex with puppet and chef um i've had to do that before it's a lot easier to just change the image and then just roll it out so that's that's why that's becoming a little more popular now and just for completeness there's nothing that says you have to use terraform you could have used cloud formation or something else right oh yeah to push out the yeah to push up the ami yep yep on doing is just passing it through into terraform through variable so um yeah if you if you can do the same with cloud formation you would just replace this stuff with your cloud formation commands and yep same thing so um are there any questions at all well thanks for the uh nice uh multi-cloud tech talk there using azure and aws excellent yep um but so i'll let uh stefano and luke ask first if they have any questions okay [Music] nothing really important for my side i'm i'm wondering how we can integrate it and use it i think right now in our infrastructure we use a um image builder from aws uh to build like the uh dcs um i mean the dmi for the ics nodes and probably we'll do that yeah for you we'll do the same for eks but maybe backer could be a good up there too yeah it really shines when it's multiple environments so if you have to maintain one image all over the place at my previous company we had a standardized image that was like approved by security you had to go through all these security gates to verify this image and so in order to maintain it in all the environments that we had aws and azure and in our vmware environment we just used packer to push it out and it was really really nice so we didn't have to manually maintain it in each each environment so yeah that's to me that's where it's really powerful yeah i thought it was interesting that you can use packer to make containers because usually you'll think if you can make a container you might prefer to do that versus a virtual machine image but packer can sort of handle both cases for when you can't actually use containers and i think our eks is a good example that we we can't just run eks using a container image we have to have the ami you can also use uh teratest to do testing with hacker and like deploying a container it's pretty pretty interesting so yeah you can test your container builds oh stefano well what about performance so is it faster than the default template managers such as cloud formation or a cloud deployment manager for gcp did you find a comparison between them um i don't i don't expect it to be much slower because it's just manipulating the apis for each environment i mean for i know for azure um if you do it through the image builder it's that you're using the same apis so you're not going to see any difference there another thing to note is so if wherever i'm doing my packer build like here i'm sshing into the vm so there's network constraints that you have to be considerate about um and if you're doing any sort of like file transfers from like your packer repository to that server you gotta account for that too so um i guess yeah that depends really what you're doing okay thanks and and what about the uh usage with the programming languages uh does packer provide you provides you um a kind of sdk to be used uh with the python node or with other programming languages or it only provides you the packaging and uh so you can make your own custom extension uh it's all written in go though there's no other languages so they have your they have a way to make your own little plugins for like if someone's developing some application and they want to have their users to be able to use packer with it they can make their own builders for it um as far as like uh other languages like you know terraform has the python sdk they don't really have anything like that that actually manipulates the packer building they're converting right now packer from json so this configuration is all json they're configuring it to hcl which is the language that terraform configurations are written in um so once they get that done i assume they'll start using like the python sdk that they're using for terraform with packer at some point okay thanks i think just to wrap up luke any um features you would like to see back or implement or is there any anything that you think is missing from packer um so when i first started using it yeah there was a lot of like pain points um especially on the vmware side uh you could only build vmware images on like a standard one server uh it was kind of a pain in the butt um but over the last year like this product has just like exploded um so as far like right now i don't really have any issues or pain points using packer it's gotten so much better just in the last like six months and i i assume it's gonna get even better and better because they have more people on the team i think six months ago there was only two people working on packer it was like that small and since then they just expanded that team out at hashicorp so we'll probably see a lot of cool things come out of this is it uh like terraform that most of these builders are hosted on github so you could make pull requests if there was something missing yep yeah it's all open source so nice all right any last questions guys or comments okay luke thanks for the great presentation and going the extra mile with that multi-cloud demo yeah okay so take care guys luca hopefully it was useful maybe i don't know if it can outperform image builder if you guys are aws native for everything but i think it's worth considering yeah yeah thanks thanks a lot you

Channel: Cloud Academy

Views: 969

Keywords: #cloudacademy, packer, hashicorp packer, hashicorp's packer, build automated machine images, automated machine, Terraform, DevOps, VM images, development, it professionals, packer tutorial

Id: Wjyev6XMXYg

Length: 22min 44sec (1364 seconds)

Published: Fri Apr 09 2021