Create, Change, and Orchestrate AWS Infrastructure with Terraform

hello everyone thank you for your patience we're going to get started so thank you all for joining us my name is Jana Berta and the director of events at hashey Corp we're really thrilled to have this webinar today today's webinar will include a presentation from Brandon Chavez he's a Solutions Architect at Amazon Web Services and then a presentation from Mitchell Hashimoto our co-founder and CTO of hashey Corp if you have any questions during the webinar you know please enter them on the side panel and folks from our team will try to answer as many as we can during the webinar and then there's some questions that we'll assign to Mitchell and Brandon while they're presenting and then some of those questions will keep to the very end and will hold a QA so without further ado first up is Brandon great thanks Jana everyone a brand of Travis I'm a Solutions Architect with Amazon Web Services and I work in the Amazon partner network which means that I work with our technology and consulting partners that bring products and solutions to our AWS customers so you might be wondering why AWS is here but really what I'm here to talk about is how AWS and Ashley Corp work together to bring interesting solutions to our customers that solve for their cloud computing needs so the first thing that I want to talk about was really how fast things change AWS and I've been here for about four years and maybe the most impressive thing has been just the constant rate of innovation so this graph is interesting to look at this is the amount of features that we release year-over-year and this only goes to 2015 so it's even crazier this year basically I work here and I can't keep up we've over 70 services interact with we're constantly adding new features to all of these services at the time this graph was produced we released over 2300 new features and/or services since the inception of AWS so basically what this means is it's it's crazy to try to keep up with this rate of change and this just kind of stresses the need for effective tooling to interact with AWS another important thing to think about with Amazon Web Services is that we have over a million customers in 190 countries so we provide many options for global deployment and we do that while also providing for extremely high reliability so we currently provide 13 regions and 35 availability zones and to kind of elaborate on that availability zone concept an individual region will always contain at least two availability zones and those azs are on completely separate power grids and floodplains and always comprised multiple data centers and this is to provide redundancy within that availability zone we don't build regions that only contain a single data center because it just wouldn't offer the reliability that our customers require to run enterprise businesses for example so this highly available global network allows you to deploy your applications near your customers much faster and in a simpler way then what can be achieved through a traditional model we also give you a options in how you deploy so AWS has supported hybrid cloud deployments ever since our inception so we found that with legacy systems there's always going to be a period where you're where you're in both environments you're running on premise and you're running in the cloud and so we've put a lot of time and effort into making sure that your on-prem resources can really operate seamlessly with the cloud so we've been working with enterprises since our inception by in 2006 to use AWS in all manners of hybrid architectures and we built a set of services and capabilities that really provide the broadest and deepest hybrid functionality of any cloud provider today so we give you functionality to extend your network into AWS ways of controlling both on-premise and AWS resources at the same time a terraform is a fantastic tool for our customers because whether they're all in on the cloud or in transition terraform provides a way of creating and managing the resources use in AWS as well as on-premise so we'll dig in to terraform a little bit more especially once Mitchel comes on but real quick we don't wanted to talk a little bit about the Hashi Corp and AWS relationship and what we're doing to bring solutions to our customers so I wanted to frame this by talking real quickly about the Amazon partner network which is basically an organization inside AWS that's specifically for working with our technology and consulting partners so we work together with these partners to build products that solve needs for our customers and we also use the APN as a mechanism to help our customers identify which partners can help them in their specific cloud journey all customers have individual needs so one of the things we've done is operated we've built and operated this program called the competency program and we use this to maintain what's basically like an all-star list of technology companies and consulting partners that are best-in-class so we've got a variety of categories like life sciences or big data or DevOps in particular and I'd recommend you take a look at these these competencies if you're trying to identify a partner that could solve for a specific need on AWS so how she is a DevOps competency partner and that means that they offer some of the premier DevOps tools that work with AWS products so more specifically our DevOps competency partners provide solutions to or have deep experience working with businesses that help them do things like implement continuous integration and continuous delivery practices or they help them automate infrastructure provisioning and management with configuration management tools in AWS also part of these competencies are we ensuring these competency partners have public references which helps us ensure that our partners have proven success in solving problems for AWS customers and so hashey Corp is a great example of one of our DevOps competency partners another interesting of the partnership is there's a huge active community of developers that use AWS and terraform together and I think if you look at the project on github it really reflects the amount of enthusiasm that terraform users have for AWS and vice versa so AWS and hash your core customers and just the community contribute back to the codebase and the result of this is that we see AWS features supported extremely rapidly so despite the massive rate of change you see in AWS platform as we talked about a little bit ago the terraform community does a great job of keeping up with it and this really ensures a terraform can just help you keep up with the rate of change at AWS it's fun to see all this development done in the open the maintainer ZAR excellent being responsive and it's a wonderful community so we're excited to be a part of that it's important to call out though that it's not just the community that provides the AWS integrations so for example AWS and hash record work together to bring a number of integrations in these in these products one of which that's interesting to call out is hash eco revolt as a aw a specific secret backend which brings customers some helpful features one of which is called secure introduction in which you can automatically retrieve a vault token by trusting AWS as a trusted third party and you can get essentially this uses some cryptographically signed dynamic metadata information that uniquely represents each ec2 instance and trust that as a proof of identity which is an awesome way of interacting with a vault and you can also use it to generate AWS access credentials dynamically based on I am policies so basically you can generate policies on the fly and credentials on the fly and then automatically revoke them when the vault lease expires so we're working together to kind of solve some of these challenging problems across the hacci core suite of products involved is simply the most recent and most interesting collaboration I think but what we want to convey here is that if there's something that you think that AWS and hash Ecore could help you resolve by working together you know by all means let us know at AWS are extremely receptive to future requests and if there's something that result and there's something that you need in order to achieve a specific use case definitely give us that feedback and we're always excited to hear what our customers want and you'll also occasionally find AWS and has record collaborating on blog posts and other technical material just to help out our community so a good example of this is earlier this year we posted a blog it's about getting started with terraform on AWS specifically so in this particular blog we talked about some best practices with handling secrets for example so like how you can use the standard AWS credentials and this includes I am roles which is really cool so if you run terraform from an ec2 instance with an IM role apply terraform will use it automatically which is great we love IM roles so we also in this blog post talked about some aspects of structuring your terraform templates to logically correlate to the structure of your AWS infrastructure we talked about how to use variables and modules but Mitchell will probably talk about all these specifics in a little bit here and then finally this is maybe the latest aspect of our collaboration we've built two new quick starts and so quick start is a program that we run here at AWS to help our customers quickly deploy production ready reference architectures so we worked with hashey Corp to build both console and vault quicksort quick starts we think this is great because these are two widely used hashing core products on AWS so we really wanted to help our customers get up and running quickly on AWS while providing a little bit of AWS common sense as I like to call it built into the templates so these are available as cloud formation templates they're open source so you can need to deploy them just as they are or you can modify them and make them your own and use them as a base for your own deployments so the console QuickStart can be deployed into your existing V PC or you can let the QuickStart set up a VP C for you and that V PC will be configured kind of with some best practices so in this case it deploys the V PC across multiple availability zones with both private and public subnets and in the public set subnets we've configured some NAT gateways and then all the console nodes get deployed into the private subnets we let you configure three five or seven console servers and then as many console clients as you want and they're configured in an auto scaling group and then the ec2 I'm sorry the console servers at ec2 auto-recovery turned on which is a nice feature in case of failure of your instance so a lot of the heavy lifting of giving us up and running is kind of taken care of for you and essentially it's one-click and filling out a couple of parameters and then you have a fully running console cluster and then to extend that we've also built a vault quick start and the vault quick start is built to take advantage of the console QuickStart and it will use console as the backend we chose that because of the integration between Bolton console it provides high availability for vault which is great so if you deployed the vault QuickStart you can actually choose to deploy the console QuickStart at the same time and in the vault QuickStart specifically we deploy and configure two vault instances and they both have ec2 auto recovery enabled and then we've done some of the work to set up monitoring cloud watch logs pushing vault audit logs to cloud watch logs for example and then we've written a nice deployment guide to walk you through the whole process so from spinning up the infrastructure to actually logging in and getting started with vault you should have a a complete step to getting this running in production in your account so go check these out you can find a summary blog post on the Amazon partner Network blog and then links to all the the assets from there so finally the last thing I want to do is just talk about a customer that's using AWS and terraform Enterprise to provision and manage their AWS infrastructure so Red Bull media it's a case study that we're working on right now and excuse me so Red Bull media is the arm of Red Bull that provides all the live streaming TV and audio services and they're running a fairly large deployment on AWS between 300 and 500 instances at any given time and so they're using terraform enterprise to automate and manage all this infrastructure at once but what moving to terraform enterprise enabled for them was essentially codified infrastructure that they can store in version control and then it really helped to establish common and repeatable workflows that allow them to iterate on this infrastructure across their development teams which is fantastic so essentially what's happening is Red Bull media house is able to leverage a hashey core PQ system to take advantage of many of the features in AWS and improve the rigidity and their ability to create and manage their infrastructure all from a single source that's it for me I'm actually going to pass it off to Mitchell now to dive into some of the specifics of terraform thanks everyone hello everybody um second one second all right okay so I'm going to be talking about a terraform obviously introducing terraform thanks Brandon for the present high level information about Asha Asha Corp works with AWS so I'm going to focus today on introducing you to terraform if you've already used terraform in some capacity some of this information is probably redundant but I really want to sort of start with just an introduction to why we built tear form what problems of solving and and how you get started with it so let's get going so start I want to talk about the problem like what inspired the creation of terraform and and it could be represented in a couple ways this is one way I like to look at it which is the problem of rising data center complexity so this diagram is one we use in a number of our talks to really represent our view of what I like to call the modern data center and the modern data center is really what's funny is it's called the data center or we call it a data center but really the word data center is becoming more and more abstract it's becoming less of a physical thing you could point at or look at and just really representing to me the the collection of all the things you need to run in order to run an application and before it used to just be no servers in the data center but nowadays it's it's a lot more nebulous so there are still data centers you know Amazon in Amazon there might be regions and availability zones within those they're still servers running you're running VMs on those servers on those VMs you might be running containers you might have servers that you're just running containers on directly there's a mishmash of like a ton of different things going on there that you all have to manage so it's pretty complex and then at the same time all those things communicate with other resources so represented by these cloud icons there's a ton of just service providers out there that run what used to be sort of core software of your data center they run that now as a service so things like DNS CDNs databases these are things that you can now just you know buy basically and get full management and and obviously Amazon provides all those things and and so when you ask somebody now to spin up you know the sort of the minimal infrastructure needed to run your application it's not just as easy as starting one server anymore it usually involves starting a number of things and connecting them together and it only gets worse as you go from minimal infrastructure necessary to real production scale infrastructure with multiple applications and multiple business units and things like that in your company and so we wanted to build a tool to basically automate all of this all this for you and one other way to look at it is this diagram here and this diagram is just a diagram of a fairly standard vanilla web application so you you basically have a CDN in front of a load balancer which is front of in front of application tiers our web server tiers and a database and sort of static stuff in s3 there's nothing in this diagram that's particularly I would say you know too advanced it's all not super beginner level but it's not it's not something that only an expert should be able to do this is a pretty standard web architecture and so when you look at this diagram you can see here too that it's if someone asks you to spin up a new web application that that you know is more than just for development is really that that's method serve real traffic and you're faced with spinning up all these resources it could be daunting and and we wanted to automate that so a sort of in summary here the problem really is that the data center is a complex multi provider problem the minimum infrastructure required for deployment is high and manual creation just becomes too time intensive very very quickly which leads us to terraform which is a tool we created to solve this so the goals of terraform from the beginning were to unify the view of resources using code so being able to actually code your infrastructure no matter whether you're using a lbs or route 53 or instances or maybe even another provider with AWS and it was to support the modern data center which is what I sort of just defined but this involves you know all layers of infrastructure the service platform as a service and software is the service it's pretty rare today to find any reasonably sized deployment that doesn't use all three of these things so you know as an example is in Amazon is ec2 platform as a service could be a number of things in Amazon it could be something like Beanstalk it could be something like opsworks or it could be e CS even in some cases and then software's the service or things like RDS and route 53 and so on and and you're usually running all three so you need to be able to communicate with all three it's not enough to have a tool for example that just spins up virtual machines that's not going to be enough for this use case we also wanted a way to safely and predictably change infrastructure so at the time before terraform existed when I was looking at this problem there was a lot of tools out there that that made it pretty nice to create infrastructure but there weren't many it was very very rare to find a tool that stuck around as you change your infrastructure going forward so and even with the tools that did that I saw a lot of people the way they were using the tools was they completely codified how they create their infrastructure but once it's created they go back to normal manual modification and change and and that isn't very scalable and so we wanted a tool that you would feel confident could safely change your infrastructure and then we want to try to workflows technology agnostic this goes in line with you know multiple providers we live in a world where a lot of your things might be on something like Amazon but there's always one or two things that isn't so an example for me that I won't name any specific vendors I don't even know if I'm allowed to but like I use Amazon for most things but I don't use route 53 for DNS au something else for de Nets and I still want to be able to use a tool to spin up my Amazon resources and then uh you know configure the DNS with those resources even though the DNS isn't on Amazon and I wanted one tool to do that so so terrible could do that and then the final thing is a little bit more broad forward-thinking visioning about the project and what we recognize is that everything today has an API and everything pretty much everything has lifecycle associated with it you create something you update something you destroy something any you know anything with an API usually has those life cycles so we viewed terraform as a more abstract tool to pretty much manage anything with a lifecycle API and and that was the goal here terraform versus other tools sort of in the abstract we we wanted to do better and and to do that we aim to provide a high level description of infrastructure so you'll see that with the the language that I'll show you shortly we wanted to allow for sort of composition and combination of multiple providers but also multiple levels so one of the first demos we ever tried with terraform zero point one like way back just to see if it was solving the problems we had was the demo which created created sort of bare-metal actually created the OpenStack type installation requested VMs from that sort of thing installed a scheduler on top of that and then deployed applications on top of that so it's like a four level system and the only reason we didn't use AWS for that was to just try to go as low as we can just to see if it was possible and and it was and so that's what a composition was really important what's here for the other thing is parallel execution so terraform paralyzes as well as it can and ends up doing things very very efficiently and then the last thing was the most important thing which you know I remember saying when we first created terraform which is if you can't do this we might as well not create it at all which is separating planning from execution so being a to get to that goal I stated in the previous slide of being able to trust terraform to change your infrastructure over time it is to be able to ask terraform what it's going to do before it does it and so terraform supports this notion of planning which shows you what it's going to do before it does it so you could be confident that's going to do the right thing and we'll talk more about that in a second and then finally just before we dive into like how you stair form I just want to talk about the save care form so first of all Brandon mentioned a lot of this too is it's open source I'm surprised sort of how many how many talks I give on our tools we're at the end of the hour you know people come up to me and think I just gave a talk on commercial software for the whole hour but I wanted to say that you know tear form and everything I talked about here is open source completely free the first release was a little over two years ago is in July 2014 so it's it's not a super old project of course but it's also it is also very mature there's over 700 contributors currently so it's very active a very large community over 6200 github stars that's a vanity metric but even though it's a vanity metric it's usually a pretty good measure of of community acceptance of something so take that as you will and we act we average a release about every two weeks so we're very very act in shipping and we do that very often and just to show the power of the community especially when we started terraform and Brandon again talked about this but one of the biggest concerns we got from people was you know how can a tool that aims to you know control all these different providers and not just AWS but even just AWS how can a third party ever keep up with them and and and it really comes down to the power of community is the fact that at Hacha corp you know we have you know a number of people a dozen or so working on terraform but they don't even need to keep up because we have hundreds of community members so it's sort of like having hundreds of people working on making sure terraform supports the latest features so we actually since terraform as a little over two years now we we went back and looked and saw sort of the average time between an announcement of an AWS feature and the pull request opening to support it is about 30 minutes so you know that could be when people are over sleep it doesn't need to be you know an official thing like we we get a pull request really really quickly to support new features and and also as part of that when there are announcements that are particularly impactful for example when Amazon announced application load balancers people one of those really really quick so will prioritize releases for special announcements just to make sure that new features are out of terraform so while we do average release every two weeks for example when alb was announced we did a release the next day just to make sure that the community had access to manage alb resources okay so let's dive into the basics of terraform and to get started i'm just going to show you sort of what terraform looks like and when you when you write the code to manage resources so here's a valid terraform syntax here this is managing an AWS instance and and as you can see it's it's human readable it's all code and you just describe what you want we'll jump back to that code in a second so the goal of M structures code here is is to provide a codified workflow to create a manager infrastructure you get benefits like integrating with with application code workflows so since it's code you just treat it sort of like an application you put it in source control management you're able to do CI on it you're able to do code review with it it's not something that a person just manually does or you follow a bunch of checklists it's actually like you see the code and how it evolves and the most important thing I think with is this last point those distribution of knowledge so I remember in the first operations position I sort of took in a company the process was kind of you know I was hired there was various wiki pages around of how the infrastructure worked but they were usually outdated so really how I ended up learning how everything worked was the more senior operations people over you know months and months would just slowly tell me you know how things work when something failed they'd be like oh yeah you haven't seen this before this does this this is how I manage it is how it works and it's sort of this oral tradition of knowledge getting passed down just orally just this bocalee and it's really inefficient and it's also prone really easily to mistakes you know if somebody if somebody says oh this is how it works but somebody changed it since that person said that it it's no longer true and so you're just relying on human memory which i think is well shown to be pretty fallible at this point and so being able to codify your infrastructure is really important because with something like terraform what happens is you hire somebody any and that new person says okay how do we how does the in structure work how does the networking work how does how do things communicate and the more senior people can say just look at the code we don't have to tell you anything you know that person the more senior person could could quit something could happen and every all the knowledge is still there you don't need to rely basically on more senior people to get your more junior people up and running and and that's super important mostly from being able to grow because the number of applications and servers that we're deploying is just growing at an insane pace and relying on just a manual sort of education isn't enough we really need to allow people to self educate on this stuff and tear form an instructor's code does a good job of that and so back to the configuration syntax though the syntax that I just showed you and I'll show you on the next slide is called HCl it stands for the Corp configuration language it's actually a language we use across all our tooling or most of it and uh it's been a few years now since h sales existed and our community you know 99% of the community thinks it's actually a very pleasant way to work with our tools however since it is in our own configuration format we do include complete json compatibility with it in every case so if you want to write json you could do that although the community accepted best practice almost all the terraform you're going to see is going to be written in HCl the JSON nowadays is primarily is very important and and it's a first-class feature it's not a second-class sort of thing but it's primarily used for machine generated configurations so it lets it lets machines like CIS and and other things actually generate terraform configuration which is pretty useful so let's go back to this example this just shows you how to text so it's easy to see diffs so if someone committed a change to your infrastructure using something like it you could just really easily see the change you don't need to you don't need to to dig in too deep to understand what happens and another important point here is that temporal configuration only represents the end state of what you want to achieve if you don't tell terraform how to get there you don't give terraform a set of steps you just tell terror from what you want your infrastructure to look like and terraform terrible job is to make it so and we'll talk more about how the nuance there later so the syntax really easily the first part of it that's bold here resource is just a key word that says we're defining a resource a resource and terraform is anything with lifecycle attached to it so create destroy update it's something that's managed so in this case the source is an AWS instance the second part is the type of thing you want to manage so you could also put an e lb here wrap fifty-three record etc the third thing is a unique name and this name is just for terraform this AWS doesn't see this name it doesn't nap to any specific parameters in AWS this is just for you to reference and understand within terraform itself so we name it in this case web this has to be unique so if you created another database instance if you wanted to manage to you couldn't name that one web as well you would have to rename that you know web 2 or anything else pretty much and then finally these things on the inside are the attributes so attributes are the configuration for the type so this will differ for each type so if I was configuring an ELB here it wouldn't have an ami option for example it's different so in this case we set some types that tell terraform how you all databases that's created and again to reiterate this only includes the end state what you want it to be not what it is today and then the last features is what we call interpolations there ways to reference attributes of other resources within terraform so for example if we're managing an ELB one of the attributes there is to specify the instances that the alb is load-balancing to and that's a list of instance IDs and one way we could do that is if we wanted to load balance to that instance that terraform just created its just reference the ID of the instance it just created we don't know it you know until it's created so by putting this here terraform at runtime we'll fill in that value and make the LD manage that instance so the first command that you're really introduced to after writing terraform configuration is plan so plan is that feature i talked about earlier which just shows you what's going to happen and this is super important and it's it's better than just a dry run so there are some tools out there that provide what they might call no op modes or dry runs and terraform plan is that but but more because what you can do is hear from klein has also saved the plan and if you save the plan you could run terraform with that plan and tell everyone basically you only do this so usually a dry run is it what a dry run is basically saying in other software not not tear from competitors or anything just like other classes of software what a dry run usually says is if you ran this software right at this moment for this state of the world this is what would happen but you know the state of the world changes and if you run it a second from now or a minute from now or a year from now it's probably going to do something different so with terraform by default a plan is a dry run but you can also save a plan and that gives you the additional guarantee of telling terraform to do only something and don't do anything else even if the state of the world changed and this this becomes very very important as you as you manage sort of production critical infrastructure tear form and and really the the problem is the solving is no prior to terraform people had to just guess sort of what what was going to happen so early on when we were going when we were when we released her form we'd go to some big potential users that wanted to learn about it and they would have a team of you know 20 or 30 people managing their AWS infrastructure but there was one or two gatekeepers so the whole thing that that had I like to call them sort of the Oracles of the infrastructure because they had the sort of knowledge where if you showed them what you wanted to do they would have to sort of divine what the ultimate rollout effect of this change would be like with it we're changing the ami of this instance trigger a new you know a change that's necessarily lb 2.2 this instance does that trigger DNS changes they would have to think of this full rollout what terraform plan actually does is alleviates this so that company actually that was a real example in that company actually adopted terraform and was able to not no longer have gatekeepers everybody is able to deploy to to production as long as the plans look good so I mean there's still people approving the plans but a larger number of people could do it so this gives more junior people more power but safely and so a plan when you output it is formatted similarly to sort of like add if there's colored symbols next to it to indicate what it means so a green plus means that in a resource will be created a red minus sign means the resource is going to be destroyed a yellow or orange tilde means that a resource is going to be updated in place so it's not going to be destroyed to be updated we could update it in place that still might mean there's downtime depending on the type of resource you're updating but at least we don't need to destroy it and then a - / + again in yellow orange indicates that a change requires a resource to be destroyed and then recreated so example here might be if you're updating an ami on an AWS instance you can't do that in place to update an ami on an instance you have to actually create a new instance so that would be a - / + operation so on that configuration we just had here's sort of what it might look like if your anti-reform plan we in this case you know we don't have any instance yet so what it's showing you here is that it's going to create with the green + it's going to create that instance you can see all the attributes that it's going to have some of them are known so the AMI and the instance type we specified so it just tells you that's what it's going to be but there also a number of other things that terraform knows that it will know but that it can't know until it's actually created so things like block devices or IP addresses and so on those things terraform knows will become available after the instance is created and the reason why terraform knows this is so that other things like ELB or dns can reference those things so your dns resource can reference a for example the Public DNS of an of a non created instance because terraform can guarantee that that will be available and order things properly so that your DNS records only created one set is available so once the plan looks good the next the next command you you hop into is terraform apply and what Terra Firma Pillai does is actually execute those changes to reach your desired state so everything you see in the plan tarot apply actually performs so the plan does it's important to just reiterate that the plan does not impact your infrastructure a plan never ever performs changes to your infrastructure it's purely read-only my query your infrastructure to to see what its current state is but it never ever performs a right or you know a change operation Terra Firma fly on the other hand execute all of those changes and so whatever imply does is it determines the ordering so I don't think I have the slides to show but in that previous example with the ELB that reference the instance that actually tells terraform the ordering that's necessary so I'm actually just going to go back to it so in this case for example you'll notice that we didn't explicitly specify any ordering of the instance must be created before the alb but terraform does consider that interpolation there being an explicit direction to do that so because that interpolation there is there terraform knows that the instance must be created for the alb and does the ordering properly in this sort of simple example it's easy you know in a manual fashion to know that you must create the instance where the alb and what gets really interesting with terraform is when you start managing more and more complex infrastructures and doing more and more complex changes it's understanding sort of the the ordering necessary to make things safe it becomes more and more difficult and it's really nice actually to run a terraform plan and just see what's going to happen know that tear forms going to handle the ordering for you properly but again you can verify that with the plan and and also it paralyzes things so with this example if we were creating two ada base instances but we were only putting web in the ELB it would create both the the instances in parallel and also create the alb before maybe the second instance was ready because it doesn't need it so terraform will paralyze as much as possible and this is super important for cloud resources because cloud resources are fast but you don't get them instantly you know they take seconds dozens of seconds minutes some things like RTS you know could take multiple minutes because they're pretty complicated and so it's really important that it's we'll be able to do multiple things in parallel in order to just get as much done before the cloud control plane can respond and then the last thing that's also just critically important for the tool is to be able to handle and recover from transient errors so clouds are complex things and if you're building you know infrastructure that has 500 or you know hundreds dozens even of resources it's not unusual to just get a transient error just a temporary error happens or maybe it might be on Amazon side but a lot of times they might be on your side if you're creating infrastructure that takes ten minutes to come up it's possible in those 10 minutes that your internet cuts out or something something just happens and and and because of this you need to be able handle it safely and terraform does so what actually one of our sales engineers when demoing terraform one other I don't know if they still do it but one of their favorite things they used to do was start a reasonably long in a multi minute terraformer fly and then in the middle of terraform fly actually just turn off their Wi-Fi and the really fun thing about this this terraform will air out and say that you know lost it can no longer communicate with Amazon and exits but then when you turn on your Wi-Fi again and rerun terraform it completes the operation it doesn't it doesn't create duplicate resources it it knows what it did already and it sort of completes it and being able to handle from these transient errors and that's a good example being able to handle these transient errors so I recommend when you're just getting start with terraformer to just try things out like that in order to build confidence and understand how terraform behaves but in the face of error it tries to complete as much pop as possible exits and then when you run it again it just completes what it needs to to reach your end state so here's an example of what terraform apply looks like when you run it so it just says what it's creating so this is again our example we just had it shows you the instance it's creating shows you the the attributes that I got it has some nice output about you know letting you know if things are still happening again with cloud resources is very common for things to take awhile and so instead of staring at a completely non interactive screen tear formless you know every 10 seconds you know what's still going on and then at the end of tells you the summary of what it did how many things are created how many things have changed how many things it should and yeah so Tim reply gets you from your current state to your target state and if you remember I've said a few times that your configuration only represents your end state of what you want your structure look like it says nothing about what your current state is and it's terraformers job in order to figure out that diff to figure out what what's going on currently and what changes do I need to get there and so when it inspects this if possible it'll just update existing resources when it can when a cancer if they don't exist it will create resources so here's an example where if we take an example we just did and in this blue here we added some tags so we just made a change we already applied it once now we're just making a change to the tags and tags don't require creating any resource you could add tags after resource after an instance is created so if you run plan you could see that the plan shows you the squiggly line which means that it's going to be updated in place and then the plan only shows that the only things that's changing is are these tags so you can see that the number of tags is changing from one to three and that the two tags that are being added or foo and zip and you can see in the summary nothing's being created nothing is being destroyed but one thing is being changed and then when you apply it you can see that instead of creating it's now a modification operation and does it in place so in this case I didn't destroy your old instance it didn't in Kearney downtime or anything it just did this all in place and you can see that by the way it ran and and this is a this is again in this small example really easy to say you know a person could do that it's really not complicated but when you're changing the end state of a complex infrastructure it's really not uncommon for a plan to have dozens of changes dozens of additions dozens of destructions and in those scenarios it's really hard to reason about the ordering things need to happen or or sort of the effect of it and one of my favorite examples of this one and this wasn't as a nativist example but someone was using a cloud provider where they requested an instance and one of the things that their company would do was would use terraform programmatically to resize the instances the the memory allocation on the instances and this required destroying and recreating those instances and they ran their business on this and they did other things but what was really really cool what makes the story really cool is at some point that cloud provider added the ability for memory change to be an in-place operation and no longer required downtime you could actually dynamically change the RAM allocation on a machine and because this company you know focused on other things they don't keep up they didn't keep up with sort of the news and this sort of thing but one day they updated terraform and suddenly there applies we're doing in-place updates instead of full resource destruction creation and they just sort of got that for free and that's one of the really important properties of terraform that I just can't state enough which is that with terraform you get the combined infrastructure knowledge of the entire community to make terraform work so you don't need to keep up with the latest changes of how things work sometimes you just update terraform and it just does things better and that's a pretty cool thing to get and then the last command I'm just sort of going to cover here is tear from destroy it's really simple it just destroys everything that it created so it only touches the infrastructure that it managed itself it doesn't touch anything else you have increased rate in your infrastructure but it's a really nice way to clean things up it makes it really easy to use terraform for things like staging environments or development environments I just tear them down when you're done and terraform destroy has all the same properties as apply with handling partial errors and so on so you know if you're running destroying something like a CI to clean up it's really nice because if it fails you just rerun it and you just rerun it until it succeeds and it's not common to have failures like that but it's nice to know that you don't have to think about it you just run it until it exit code zero and you're sort of good to go okay so that was sort of like a whirlwind introduction to terraform and and tear forms you know like many tools is something that is really easy to get into I think you know enough at this point to easily sort of get started with tear form but it has a lot of depth to and due to sort of the time of the time constraints of the webinar I can't go too deep but something I wanted to talk about is okay you have this basic knowledge how do you start applying it this to a realistic example and there's going to be a very high level sort of workflow example of what what you should do when you're getting started to terraform the examples I show you here aren't going to be runnable exactly as is or anything but they should give you you know enough breadcrumbs to get to where you need to go and so going back to this example here that I showed earlier in the webinar this is a really standard web application deployment like I said when you start using terraform you'll start to see things see these things sort of like this and and with this example what's actually happening here is that the purple circles are resources and then the arrows between them are the interpolations and when you start thinking about your structure very quickly after you start using terraform this is how you're going to start seeing things and and but it's a really useful way to see things so going back to here the purple circles are the resources and and the arrows are the interpolations so let's get started turning this into a terraformed infrastructure and so what's really important about adopting terraform is that you don't have to go all in you don't tear form isn't useful only if it's completely managing your infrastructure it could manage is the partial part of your infrastructure and still be very useful and so that's the best way to adopt it is I by something especially when you're getting started just identify something that's easy to automate a low risk and just start there so in this case I put a purple box around what is probably a safe thing to automate in every web application which is sort of the web server tier the web server tiers are usually a stateless they could come up and down and load balancers handle you know routing for them only when they're healthy and draining when they're not and so they're usually a pretty safe way to get started so if we're getting started here what the box is around is pretty much just purple circles there's no arrows fully contained in the box set so we just have to create a number of resources so that might look like this so we create the resource it's an ABS instance type it's part of the web tier so we'll just name it web remember that's just you know an internal name for us and then we specify a number of attributes the one thing here that we didn't talk about that I will now is this count thing and so every resource and terraform could specify accounts and this is a meta parameter so it's a parameter to terraform and not to AWS itself and what terraform does with this is just duplicates this thing exactly that number of times so in this case we're saying count equals four because in that diagram I just showed you there was four instances so it might just look as simply as this in the diagram actually like exactly it's using an auto scale group but you know to get started with terraform you might just start with this and work your way up to that so we're just going to show this eight businesses without an auto scale group as an example so once you have that in place the green box shows us what's done we've now sort of terraformed those four things and now we can think what's next like what's the next easiest thing and you can kind of go any direction but in this case this load balancer looks like a good good place to go and so if we want to do the load balancer we're now looking at this purple circle some Angelo bouncer but also the arrows that are pointing to those web servers we need to be able to reference those instances somehow so here's how that might look we add an e lb resource and then for the instances we just reference all so this is again it's kind of new syntax I've shown you interpolations of referencing one but when you have multiple you could use the star syntax that's in here to say I want all of them so in this case if we change the count of Atos instance for example from 4 to 5 or 4 to 3 or 4 to 40 after doing all that terraform would update your ELB with all of those instances id's and so that automatically creates those 4 arrows that we had and so now at this point you would have sort of a green box around all these things and you'd be automating all of this with terraform and at this point you sort of just rinse and repeat you you choose what you want to automate and what you don't want to automate and and really even for people that use their from really really heavily it's not uncommon for people to just feel comfortable with a few things just still manage manually for whatever reason maybe it's just difficult to automate maybe they have a really good system in place that doesn't require terraform you just you don't need to apply it to 100% of your infrastructure and and that's one of the reasons it's it could be so great so just rinse rinse repeat and so I want to give you some next steps since I know this is a very surface level sort of talk so I'll give you some next steps to just sort of guide you towards if if this was interesting to you what you could look at next in order to learn more about terraform and implemented successfully so there's a few features that you should definitely look at you know all the basics and so these features wouldn't be scary to look at but variables and outputs are a way to parameterize your terraformed configuration which becomes important pretty quickly so those would be important interpolation functions so you saw interpolations like referencing the ID and set terraform ships with a really rich library of probably a 50 or so interpolation functions that you could actually call to mutate some of that stuff so this comes from really basic things like turning things all lowercase to more advanced things but like actually manipulating things like cider Blockson and network addresses and parsing IP addresses which becomes really important with infrastructure so I recommend taking a look at the interpolation functions it's just like a standard library that comes with terraform modules are a way to encapsulate your configuration so that you don't need to repeat yourself so for example so for example you could wrap the the your company approved way to manage VPC layouts you could wrap it in a module until everybody if you ever need a B PC use this module and users of the module wouldn't have to really know the details of what kind of site or block to use what's what size space and how many subnets how do we configure Nats how do we compute your outs you don't need to know anything that you could just say I want a V PC and then the module would create it for you and give you sort of the relevant information like security groups and the ID of the V PC and things like that so modules they pop up relatively quickly once you start using terraform now and then the last thing is remote state so remote state is a way to basically collaborate on terraform more efficiently and I'll just sort of leave it at that and you could take a closer look but these are the things that you probably want to look at next and then the resources that are available to you is the project website so the project website has Doc's covering everything so the project website is going to be indispensable as use terraform as reference material there's a third-party community github organization called terraform community modules which actually we're sort of gearing up to become more involved in but this is just a community thing that's fun up that show that it's a really good way to look at example terraform modules and and they're pretty high quality so take a look at those Google is your friend so especially in the past six months terraform has just been on a really rocket ship level of adoption and so there's a lot of blog posts out there that cover how to use terraform so don't be afraid to Google things you'll probably find an answer as part of that I should mention the terraform also has sort of official community avenue so there's a gator channel for real-time communication which has a couple hundred people in there on average and I hang out in there and see it's pretty active and then there's a May the mailing list is a good way for if you don't want an answer now send it to the mailing list and you'll probably actually get more people viewing your thing on the mailing list than you will in the real time chat so if you go wait a little bit the mailing list usually gives you a very high-quality responses to a posh group of the company also does various events sort of like this we do user groups we also do trainings as well if your company wanted that and then the last thing that I wanna mention it is is books there's actually just in the past month we've gone from zero terraform books to to care form books so terraform book calm written by someone in the community and is actually available now so you can take a look at that and then actually just today so I didn't even update the slides I didn't even know until today but just today O'Reilly announced that they they have an early pre-release version of a terraform book as well so those are both pretty up-to-date with the latest version of terraform and you can check those out and then the last thing I'll show is just sort of like just a screenshot of the website of the websites documentation you on the main repo sort of addressing multiple modules so I will mention that this is actually a use case that we have planned sort of solutions for fruit of the next four Terra 0.9 which is going to be early next year so right now you could you could totally do it and we have customers that are managing this it's just a little more manual than we would like and then in the in the future we're actually going to give you a bunch of workflow tools to manage this so hope that helps you

Channel: HashiCorp

Views: 66,402

Keywords:

Id: TFLQcgZr0no

Length: 64min 25sec (3865 seconds)

Published: Thu Nov 17 2016