5 Lessons Learned From Writing Over 300,000 Lines of Infrastructure Code

everybody thank you for coming this talk I'm gonna try to share some of the lessons that we learned from writing 300 thousand lines of infrastructure code at gruntwork first lesson is I actually originally called this talk this talk ten lessons learned it's a really bad idea before you write the talk is that you write it and it's like if three hours long okay so it's gonna be five lessons hopefully will be enough for now and the goal is as you start writing infrastructure code as you start using some of these tools hopefully you'll be able to avoid some of the mistakes some of the silly little things that we did but before I get into that I want to set the context a little bit and I want to give you a little bit of a confession so here's the truth I've been doing this stuff for a while working on develops and infrastructure and the reality is DevOps is still very much in the stone Age's and I don't mean I don't say that to be me and it just we're new to this right we've only been doing a bob so we've only been figuring out sre practices how to do software delivery for a little bit of time and this is a little scary though because we're being asked to build infrastructure that looks like this right so bill supposed to build these incredible modern powerful systems but it feels like we're trying to do that with tools that look like that right yeah you're like duct tape and a little bit of bubble gum and like good luck of course if you read the headlines it sounds like everything is so cutting edge right you open any web browser you look at any blog post and you hear nothing but kubernetes and docker and serverless and micro services and buzzword this and buzzword that and it sounds really cool but it doesn't feel like that to me I don't about the rest of you but when I'm actually doing this work when I'm actually working on infrastructure it doesn't feel cutting-edge it feels a little more like that right I'm like trying to heat up like a slice of pizza on an iron or like a bloat what is Halleck i don't know what the hell is going on here but this is DevOps right this is my new hashtag this is DevOps okay it doesn't feel cutting-edge it feels like this like I mean I don't know I guess this works but like why would why are we doing it this way this is crazy all right and like nothing fits together right like no I sure good come on man like you this like this is probably an accurate depiction of like the typical CI seedy pipeline at most companies right like ah like this is what it feels like and we don't talk about that at us okay we don't admit to ourselves often enough we'd like to talk about all the cool stuff we don't talk enough about the fact that the stuff is hard building production great infrastructure it's hard it's really hard it actually be really stressful I think generally speaking it's probably not crazy to say that people in the DevOps industry tend to be a little more stressed out than other parts of software engineering it's also really time consuming and this aspect is always hidden because you only see the end result right you read a blog posts and at the end of the blog post it's like and tada kubernetes and you're like wow that sounds great you have no idea what time or work went into it and so I want to share with you some of the rough numbers we've found from working with a lot of companies doing this stuff a lot of times and these numbers are probably best-case scenarios to be honest with you so hopefully you can read that this is what its gonna take to write infrastructure code that you can actually use in production and I'll come back to what I really mean by production grade infrastructure a little later in the talk but here's the basic numbers if you're using a managed service something that you know AWS or Azure or GCP is managing for you you can probably get up and running in a production capacity in a couple weeks with that one to two weeks because the managed service provider is doing all the hard work but if you're gonna spend up your own app and it's some sort of distributed system but it's stateless it's not writing to desk or anything that you care about on disk like an ojs apps and sort of front-end applications that's gonna be double two to four weeks you want to build a distributed system that's stateful something that writes the desk and you don't want to lose that data something like setting up your own elasticsearch cluster your own Kafka cluster MongoDB now we're talking two to four months we're gonna jump an order of magnitude here and that's for one distributed data system right if you have multiple it's two to four months for each one and then the interactions between them actually make things even more complicated and finally if you want to put together your entire architecture from scratch we're talking a six to 24 month project six months is probably for a tiny little startup 24 months is more realistic for most larger companies for some it'll take far far longer so when you read those blog posts that are celebrating these incredible milestones like put a little checkbox in your head that like these are the time frames we're talking about these are the best-case scenarios for things that are understood so the good news is a lot of this stuff is getting better so I don't want to be a downer there are problems but they are getting better and one trend that I really like I think something that's very positive is we're able to manage more and more of what we do as code and we're seeing this kind of across the stack right we're going from manually provisioning hardware to infrastructures code we're going from manually configuring our servers to using configuration management tools and using code going from manual application config to configuration files everything is moving to code and there's a lot of benefits to code I mean this is why I'm excited about it hopefully if you're at this conference you don't need to be convinced too much of the benefits of code but it's everything from being able to automate things instead of having your poor sre do everything by hand 300 times you get to store the state of all of your infrastructure and version control you get to review every change you get to test every change the code is documentation so it's not just locked in somebody's head and you get reuse so you can use code written by other people as well so a grunt work what we've done is we've kind of run with this idea of infrastructures code and we built a library of reusable code most of it is written in terraform go bash I'll come back to the tools we use a little bit later on and we have these off-the-shelf solutions that give you production grade infrastructure for a whole bunch of different technologies and we have a whole lot of customers using this stuff in production and so what that really enables is you go from this these time frames of weeks and months to do these things from scratch to down to about a day mostly cuz the grunt work team has invested that time on your behalf that's the benefit of code you get to reuse us kind of toiling away for months and years on this stuff and we've been working on it for a while right so we've been working on this library for more than three years it's well over three hundred thousand lines of code it's pretty sizable amount of and structure code and so based on all the ways we got this wrong hopefully I can help teach you all some pretty useful lessons there we go so I'm ganna Brookman go by the nickname Jim I'm one of the cofounders of grunt work author of a couple books terraform up and running and hello start up these books talked quite a bit about these develops concepts and so this is what I'm these are the five lessons that I'm gonna share with you today first I'm gonna go through a checklist of what it really means for infrastructure to be ready for production and then I'll talk about the tools that we use and some of the lessons we learned around that I'll talk about how to build modules in a way that they're actually useful I'll talk about how to test your infrastructure code it's a very rare topic and then I'll talk about how to release this stuff into the wild so we'll start with a checklist and when I tell people about these numbers right when I show somebody that's relatively new that it's gonna take this long I get a lot of skeptical faces right usually they look something like that it's like come on really six to 24 months I read a blog post I read this thing I'll get this done in a couple days no you won't it's gonna take a long time you're just not gonna and so why so I get a lot of incredulity and people are asking why why does it take this long and so I think there's two really main reasons they're very similar but they have slightly different flavors to them so the first one is something called yak shaving who here is familiar with the term yak shaving wow it's less than half okay so for the rest of you you're welcome this is gonna be a term you're gonna use probably every day of your life from now on so pay attention so here's what yak shaving is yak shaving is this like seemingly endless list of tasks you have to do before you can work on the thing you actually want it to do that day so the best definition I've seen of this is from Seth Godin blog and he gives an example like this so you come home one day and you want to wax your car so you go to your backyard you grab the hose and put crap my hose is broken no problem I'll go to Home Depot I'll get a new hose you get in your car you're just about to go and then you remember to get to Home Depot I need to go through a bunch of toll booths I need an EZ Pass no problem I'll go to the neighborhood neighbor's house grab his EZ Pass then you remember he's not gonna lend me his e-zpass because I borrowed his pillows I got to return the pillows first so you go rooting through your house you find the pillows and you realize that all the Yak hair has fallen out of the pillows while you were borrowing them and so the next thing you know there you are sitting in a zoo shaving a yak just so you can wax your car so that's yak shaving and if you do DevOps you know what I'm talking about right you want to go deploy a service over here and then there's some TLS thing you have to do and while you're working on that you uncover this other weird bug and then while working on that you broke something in production and like you're miles away from what you originally wanted to do and this happens all the time DevOps just seems to consist of a thousand tiny little interconnected systems and they all break and everything you do you end up yak shaving and the weirdest part is at the end of the project you're like the hell did I just spend the last three months doing it I feel like I feel like this should have been five minutes but I three it's it's maddening so yak shaving is one big reason second reason is there is genuinely a long checklist of things you have to do to build infrastructure for production so I'm gonna share our checklist with you these are the things we look for when we're building infrastructure that we want to use in production and what I mean by that is this is the kind of infrastructure you would bet your company on right if you're gonna deploy a database you don't want it to lose your data because if you do that could be a company ending event so this is what I'm talking about when I say production great so here's the checklist actually this is the beginning of the checklist and the first part of it most people are familiar with it these are the parts that just about everybody can guess right you have to have a way to install your software you have to configure it you know port numbers users folders etc you have to provision some hardware to run your software and you have to then deploy it potentially update it over time so everybody knows this this part is easy but that's part one of four so this is what you get this is what most people forget so here's part two security is usually an afterthought things like how do I encrypt things in transit write that whole TLS thing how to encrypt things on disk authorization authentication secrets management how about monitoring right if you want to deploy something to production you probably don't want to know what's going on with that thing have some sort of metrics and alerts you want logging you want to rotate the logs on disk so you don't run out of disk space you want to aggregate them somewhere so you can search them backup and restore again if you're writing if you're running a distributed data system or any data system you probably want to know that if it crashes you're not gonna lose all your data and go out of business here's a few more items that tend to get forgotten how about the networking setup right all your subnets all your route tables static dynamic IPs how about SSH access VPN access you have to think through these things these are not optional you need these for almost every piece of infrastructure then you got to think about high availability what happens if a server crashes what about a whole data center scalability what happens when traffic goes up and down how do you react to that and performance right there's a whole field of things that you have to do you know if you're deploying a JVM app you have to think about GC settings for just about anything you have to think about memory CPU you might have to do load testing that could be months of work just by itself and then these ones almost nobody gets the page four right that's part four or people usually have dropped out long before this right cost optimization how do you make this thing affordable documentation if you ask people where the docs for your infrastructure code they kind of like what and tests like it's rare to see tests for infrastructure code but this is the checklist okay these four pages are what you want to use what you're building infrastructure this is what you need to do for just about every piece now not every single piece of infrastructure needs absolutely every single item here but you want to at least make the conscious decision that we're gonna build these aspects and not build some others you don't want that to happen by accident you don't want to find out six months later oh crap I forgot about backup and restore and you usually learn that the hard way so key takeaway next time you go to build a piece of infrastructure use this checklist I'll be sharing this slide deck actually it's already on SlideShare but I'll tweet out the link and I'll post it in the hajikko on how icons hashtag we also have a on the gruntwork website there's a checklist for your entire architecture so not just one piece of infrastructure but everything that you're gonna do and so you can find that on our website grunt work io / DevOps checklist it's also in the footer so it might just be easier just to go to groundwork that IO and click it okay that's the checklist second thing that's worth thinking about are the tools so you know what you need to do what are the tools you're going to use to actually implement that checklist so when we're thinking about the tools that we want to use these are usually the principles that we're following we want things that let us of course define our infrastructures code we want them to be open source and have a large community we don't have to build everything ourselves hopefully their cloud agnostic and work across many different providers we want tools that a very strong support for code reuse and Composition and that'll come up in the next section we don't want to run extra infrastructure right I want to deploy my infrastructure not deploy infrastructure to deploy my infrastructure I want to keep it simple and we like immutable infrastructure practices so when possible we like to use tools that make that easier so the tool set that we're using now and bear in mind this is not some like religious debate you this is just what we're using today because for practical reasons it makes sense is as follows so our basic layer the underlying infrastructure we use terraform so that's going to lay out all of the basic networking configuration load balancers databases and of course all of the servers that we're going to run our apps on so those servers we need to tell them what software to run so on top of that we're gonna use Packer to build our virtual machine images some of those VMs will have things like kafka installed Redis some of those VMs are gonna have kubernetes agents or ECS agents and so they're gonna form a docker cluster and in that docker cluster we're gonna be able to run docker containers and generally speaking the way we're at work as of today is we use the docker containers for stateless applications and stateful applications we run outside of docker usually has their own VMs that's starting to change I think docker kubernetes are becoming a little more mature in that space but I would still be terrified to run like my sequel on top of kubernetes that wouldn't do it final piece of technology that we're using here is under the hood all of this stuff this is our duct tape this is our you know paper clip and our twist tie as a combination of scripts like bash and Python we use them for example in the duct to define the docker containers we use them as glue code we also use go any time we need a standalone binary that's gonna work on any operating system we'll write that application and go so those are the main tools basically docker Packer terraform and some mix of general-purpose programming languages to kind of hold everything together so here's an important note these are the tools we've picked they fit our use cases you're welcome to use them you might find other tools or better fit for your use cases but the key insight here is not you should use those tools that's not really what I'm trying to teach you here the key insight here is that while tools are useful they aren't enough so we've you can do whatever tools you want in the world it could be the most incredible piece of technology out there but if you don't change the behavior of your team it's not actually gonna help you very much and here's why so if you think about it before you start using infrastructures code the way that your team is used to working is if you need to make a change you do it directly you SSH to the server and you were on some command you connect to the database and you run some query and what you're saying now is I want to change this so now there's this level of indirection this layer of indirection you need to go change some code there might be code reviews there might be some tests then you got to run a tool and that thing is gonna apply the actual change and that's great there's a lot of good reasons to do that but these pieces in the middle they take time to learn and understand and use properly and if you don't invest that time you're gonna have the following scenario something's gonna break and your ops person is gonna have two choices they can either do what they already know how to do you know make a change directly five minutes or they can spend two weeks learning that stuff it's a rational person every time they're gonna choose to make the change manually but if they do that now your code whatever tool you chose that code is not going to accurately reflect reality and so the next time somebody tries to use the they're gonna get an error and then they're not gonna trust that code right I tried to run terraform I got a weirdo error never mind they're gonna go and make a change manually so then the next person is gonna be even more screwed and the next person and within a few weeks that incredible tool that you spent all that time working on is basically useless because people are changing things by hand so the problem with that of course is that doesn't scale making changes manually if you have a few people a few pieces of infrastructure fine but as your infrastructure grows as your team grows this does not grow with it very nicely whereas this does and so the key takeaway here is you need to change behavior if you're gonna be adopting terraform pack or dock or any of these tools you have to give people the time to learn these things the resources to learn these things you know do training go to conferences like this go read blog posts and you have to make sure that they're bought into it right the last thing you want to do is have an ops person who just like I hate this terraform thing I'm never gonna use it like that's not gonna work terraform is not gonna help you so you got to change behavior that's really the takeaway here okay third item is how do you build modules something that we found something that we learned to the hard way is even if you pick a good set of tools even if you change behavior and you just sit somebody down who hasn't done this before and tell them okay go build your infrastructure they tend to shove all of it into a single file or a single set of files that's all gonna get deployed together so the dev environment test environment stage environment everything is just shoved into one super module essentially and what you'll find out very quickly and very painfully is this has a huge number of downsides for one thing is just gonna be slower we've had customers we're running terraform plan would take five minutes just to get the plan output because there's like three thousand resources and you have to like make API calls and fetch data about them but even if that wasn't as big of an issue the plan output is impossible to understand right if you're on Terra firme plan and you get 900 lines of result no one's gonna read that you're just gonna say yeah I hope it's good find apply yes and you're not gonna catch that little line in the middle that's red that's by the way I'm deleting your database you're not gonna see that if you have a gigantic module it's also harder to test so we'll talk about testing a little later but the bigger the module is the harder it is to test it it's harder to reuse the code because if everything is in one giant super module it's hard to reuse parts of it another one that a lot of people miss is if everything is in one super module then to run anything you need permissions and access for everything it's what every user has to be an admin because all of your infrastructure is in there and that's terrible from a security perspective and finally your concurrency is limited basically to one so only one person at a time can change anything in any environment ever so that's obviously a problem and of course the biggest issue with all of this is if you make a mistake anywhere in this code you could break everything you might be editing your staging environment you make a silly typo and you're not paying attention and because it's all shoved into one set of code you blow away production this happens this isn't hypothetical this happens a regular basis so here's what I'm gonna claim as large modules in infrastructure code world are harmful they're an anti-pattern they're very bad idea so what you really want is you want isolation you want the different parts of your environment to be separated from each other in the at the code level and obviously in the infrastructure level I mean that's why you have separate environments in the first place right you want stage to be isolated from prod you actually also want isolation between different types of infrastructure so for example your V PC is something you'll probably deploy once and you're probably not going to touch it again for a year or a long time whereas your front-end app you might be deploying that ten times a day if they're in the same set of code then ten times a day you're putting your entire V PC at risk of just some silly typo some silly wrong command for no reason whatsoever there's absolutely no reason to do that so you kind of want to group your infrastructure by risk level by how often it gets deployed so the way you do that is you take your architecture and it can be arbitrarily complex you know you can have your V pcs you can have your servers your databases load balancers etc and you're not going to define that out of one gigantic super module you're going to define it out of a bunch of tiny little individual standalone modules this pattern is the only one we've seen that works well especially at scale especially as your infrastructure gets more complicated it's building it out of these little tiny building blocks so what that's gonna look like in terms of code is you'll start with your environments as the top level items so dev stage prod and they're all separate from each other within each environment you'll have the different types of infrastructure so maybe the V PC is separate from your database is separate from your front-end application and of course you don't have to copy and paste between all of these the actual code itself can be implemented in these small reusable modules and even the modules themselves can be built out of still smaller pieces right kind of like modules all the way down and there's a lot of advantages to doing this basically it's the opposite of all the problems I talked about things will run faster you can limit permissions based on just what you need for that one particular module if you break something you're the damage you do is limited etc so here's a quick example that I'll browse through in the terraform registry we have a module to run vault which is a reasonably complicated distributed system so this is some open source code and if you take a look at that module there are three key folders this is how we organize the code for even one piece of infrastructure and so those three folders are modules examples and tests so I like to show you in IntelliJ itself so here we have vault this is the modules folder that font might be a little small but essentially in here we have a few things and I'll switch back to the slide deck since the font it's not super readable we have for example vault cluster is a separate sub module so you notice even though it's just vault it still consists of a whole bunch of different little sub modules vault cluster is terraform code to deploy an auto scaling group a launch configuration security group whatever you need to run a cluster of vault servers separate from that we have other modules such as vault ELB this is a terraform code to deploy a load balancer security group rules are separate install vault isn't terraform that's a batch script you can see it takes in what version of vault to install using vault Enterprise the path to where the binary is we have run vault in here that's a script that will start vault on boot and here it will configure TLS port numbers etc so why do we have all of these tiny little sub modules in that code base the reason for that is it gives you a lot of power you're going to be able to combine and compose those pieces in many different ways if everything was in one gigantic module it either works for your use case or it doesn't and this isn't some hypothetical thing where you just want to support every possible use case out there it turns out that you're gonna want to run your own infrastructure in different ways in different environments for example in production you might want to run vault across three separate nodes vault usually you run it with console you might want to run console on three separate nodes but in a pre prod environment you might want to co-locate all of those on maybe one server or just one auto scaling group and so you need to write your code in a way that you can do that and the best way to do that is to break it up into these tiny little modules and so just to recap these are actually patterns you might want to commit them to memory or you know use the slide deck later when you're building your own modules as they come up again and again it again almost every piece of infrastructure has a similar structure you have the modules folder there's some sort of install script it's good you need to install the module as you can remember from the checklist there's a run script this is what's gonna configure it during boot there's going to be some sort of terraform code to provision the infrastructure for that module and there's also going to be a bunch of these reusable components that you might want to share so security group rules are a separate module because I might want to use it with my ball cluster but if I'm running vault co-located with a console I might want to attach the same security group rules to my console cluster instead so these little pieces are very very important and each of them exposes a bunch of input variables and output variables so that you can wire them all together in a bunch of different ways and so you can have one configuration of vault that's set up like this or using the same set of code you can configure it differently with a different load balance or a different way to install it configure it you have that flexibility and power and you're going to use it yourself let alone what happens as the infrastructure changes over time there's an examples folder and that's going to show you examples of different ways to combine and compose those little sub modules now we build that in part as documentation it's executable documentation but it turns out it's more than that so some of these examples are terraform codes some could be a Packer template setter but where the examples become really powerful is with testing and so in the next section we'll talk about testing but the key thing to remember here is when we're doing testing what we're testing is usually those examples we're usually deploying those actual examples as the way to validate that these modules work so the examples are really really really useful so key take away build your infrastructure out of these small composable pieces if you see a gigantic module you should react to it a lot like you would if you were using Java or Python or any general-purpose language and you saw a single function that was 10,000 lines long right we all know that's an anti-pattern you don't write 10,000 line long functions it's the same and infrastructure code you should not have 10,000 line long infrastructure modules they should be built out of these little pieces okay how do we test this stuff something that we found again the hard way and something that you're gonna find as you starting with this journey is that infrastructure code rots very very quickly pretty much everything we're using is changing all the time right Tara from 12 is about to come out that's a massive change to the whole language Dockers changing packer kubernetes AWS GCP these are all moving targets none of them stand still and so your code even if it works today might not tomorrow and this happens really really quickly much faster I think than a lot of other programming languages and so really the best way to put it is not that it rots very quickly but I think this is the general law infrastructure code that does not have automated tests is broken and I don't mean like it's gonna be broken in the future I mean it's probably broken right now something we found out the hard way is even if you test this thing manually you know you're building a module to run vault you deploy it it runs you can store secrets you can read secrets as best as you can tell manual it's working when you take the time to write the automated test every single time we have caught non-trivial bugs in our code every single time almost without exception so if you don't have tests on your infrastructure code its broken just be aware of that now we know how to test code in general-purpose languages ash go etc we can write unit tests that mock the outside world and we can run those tests on localhost for infrastructure code this is a little harder there's no local host for terraform right if I have to terraform code to deploy in AWS VPC I can't deploy that onto my own laptop there's also no real unit testing I can't for the most part mock out the outside world because the only thing terraform does is talk to the outside world so really all of your terraform testing is actually going to be integration tests and the test strategy looks like this you're going to deploy the infrastructure for real remember those examples that we saw earlier in the modules you're going to deploy those examples for real so if you're building infrastructure for AWS you're gonna deploy into a real AWS account you're then gonna validate that it works now how that works obviously depends on your use case for vault our automated tests actually initialize the cluster they unseal the cluster they store some data they read some data and then at the end you're gonna underplay that infrastructure so it's not very sexy but this is how you test infrastructure code and so to make that a little easier we've open sourced a tool called terror test I did a talk here last year it was not open source then so I'm happy to announce that now it is available as an open source library and it allows you to write your tests in go and it gives you a lot of utilities that make this style of testing easier and the philosophy here is basically if you're trying to figure out how do I test this module using terror test all you really need to think about is how would I have tested it manually because that's all we're doing we're just automating whatever it is we would have done by hand so here's a typical test structure that you would use with terror test so you can see at the top what we're testing here is one of those examples in the vault repo and so we tell terraform ok this is where the example code lives at the start of the tests we're gonna run terraform in it and apply to deploy that example into a real AWS account we're then going to validate that this thing is working now what does that do that of course is very specific to whatever you're testing but terror test is a whole suite of utilities to make that piece easier as well so there's libraries for looking up IP addresses of ec2 instances there's libraries for making HTTP calls checking the response code and body and doing it in a retry loop because you'll find that there's a lot of asynchronous and eventually consistent systems here there's a lot of code that uses this SSH SSH helper to basically SSH to a server and execute a command and give you back the standard out and standard error and so there's all these utilities built-in for validating that your code actually works and then at the end of the test that's what that fur word means it means at the very end we're gonna run terraform destroy to clean up after ourselves and so these are all terror test helpers and it's not gonna make the tests easy to write but i hack them a lot easier than doing it from scratch and Tara toughs these days supports AWS supports GCP Oracle cloud there was a talk today about using it with Azure so you should be able to use it for a whole lot of use cases one important note is that when you're running these tests we're gonna be spinning up a lot of infrastructure and tearing it down all the time so two important things to note about that the first one you want to run your tests in a completely isolated sandbox account don't run them in prod I hope that's obvious but I wouldn't even use my staging or dev account I would actually create a completely separate place to run these tests where you don't have to worry what happens if I accidentally delete the wrong thing or create the wrong thing second tip is occasionally the tests don't clean up after themselves very well the test might crash you might lose internet connectivity so you don't want to leave a whole bunch of resources around and so we built a tool called cloud nuke as the name implies it's quite destructive don't run it in prod but in your isolated sandbox account you can use this to clean up resources older than a day or older than a week and basically not lose lots of money okay one final thing to chat about with testing is this test pyramid usually you're gonna have a lot of unit tests smaller number of integration tests smaller number of end-to-end tests and the reason for that is as you go up the pyramid it takes longer to write those tests they're more brittle and they run slower and so you want to catch as many errors at the bottom of the pyramid as you can so within an infrastructure code similar idea the bottom of the pyramid and that's going to be those individual sub modules you want to have small little sub modules that you can test and be confident that they work as you put them together then integration test is combining a few sub modules and to end test is your entire stack you're not gonna have too many of those and the reason for that is they're really slow so the unit tests for those individual sub modules they're not fast but it's as fast as you're gonna get and then it gets slower and slower as you go up the stack checkout tear test docks we have a lot of best practices about how to speed up testing so make sure to read that so add tests I'm almost out of time so final section is releases this one's really quick I'm gonna put together the whole talk for all of you so here's how you're gonna build your infrastructure from now on one use the checklist make sure you're building the right thing to go write code don't do this stuff by hand go write code three write automated tests with terror test four do a code review you know make sure your codes actually working have your team members take a look five release a new version of your code what that really means is that it gets tagged and now you can take that code this versioned artifact and you can deploy it to QA and if it works well then you can take the exact same code because the artifacts immutable deployed the staging and if it works well finally to prod and so here's the key takeaway for this talk we're gonna go from that to that you know we go from pizza on an iron to tested code that's been through a checklist a code review it's versioned and we're rolling it out from environment to environment it's not easy it will take a long time but it is helpful that's it thank you everybody you

Channel: HashiCorp

Views: 68,048

Rating: 4.9556332 out of 5

Keywords: HashiCorp, HashiCorp Terraform, Terraform

Id: RTEgE2lcyk4

Length: 35min 32sec (2132 seconds)

Published: Tue Nov 27 2018