Introduction to HashiCorp Terraform with Armon Dadgar

form today so terraform if you tool for doing provisioning so when we talk about provisioning there's really two different problems we're talking about right so as we provision there is our day 1 challenge where we haven't actually started running anything yet and so how do we go from running nothing to running something and then there's actually are more challenging day 2 plus which is great day 1 we have nothing and we prove it in our initial set of infrastructure and then day 2 we have an existing set of infrastructure but we're trying to evolve it we're changing it over time we're adding new services removing services and generally evolving the way our infrastructure looks so how does terraform really solve this problem we fundamentally believe in taking an infrastructure as code approach and so what I mean by that is we allow you to declaratively define in sort of a set of terraform config files what you want your infrastructure to look like so in something that's very simple and human readable you describe your overall topology and so what I mean by that is when we talk about infrastructure as many moving pieces I might have a VP scene or mic or network on top of that I might provision my security group rules the way my network security is set up and then within that environment I define my set of virtual machines I have some VMs that I want to provision and then overlaid on top of that I have a load balancer and so you can imagine how this graph extends over time I add different pieces different amounts of a complexity to it and I incrementally evolve my infrastructure day over day week over week and so while this is sort of graphical in nature the way we capture it in a terraform configuration is very lightweight in describing sort of high-level resources we describe you know the VPC resource and the fields that are required to create this this element of our infrastructure and then as we define these other things we can reference the other components of our infrastructure and so what terraform then does is gives us a workflow for creating and managing bus and ultimately this is sort of three different main commands that we run the first command is terraform refresh and what terraform refresh does is basically reconcile what terraform thinks the world looks like so Tara forms view of our for a structure with the real world and so how this works is terraform goes out and talks to the real world ill query our different infrastructure providers whether that's VMware or Amazon or Asher and ask them what's actually running and so in this way terraform gets an up-to-date view of what our infrastructure actually is then we run a plan and a plan is where terraform figures out what it needs to do and so you can think about a plan as reconciling sort of the real world what things are actually running with what we want to be running our sort of desired configuration so when we talk about Tara forms configuration what we're putting into this graph this is our desired state of the world this might not actually be the world as it is so as an example on day one when we execute this nothing is yet running so when terraform runs the plan it realizes that there is nothing yet running in the real world so to get to this desired configuration terraform needs to boot all four of these resources and so this becomes the plan terraform will give us an output that says to bring the world to look like what you want it to be I'm gonna go boot these four things so then the last part is actually doing this this is the apply phase of Tara forms execution so we apply Tara forms execution it starts with our plan of what we need to do and it goes out and executes sit against the real world and so in doing this Tara form will go out and figure out what is the right order in which these need to be done so there's a natural dependency here the network must be defined first before we can define the security groups around it once those are defined we can boot all of our VMs whether we're booting one or booting you know hundreds of these in parallel and terraform will figure out where it has that opportunity to create things in parallel where where it needs to be sequential and once these vm's are available then terraform can lastly create our load balancer and it's a all done so this gives us a way to do our day one infrastructure so day one we describe what we want our universe to look like we're antara from refresh it so there was nothing existing plan said all of these things are going to be created and then applied goes out and builds it what now happens over day two is we start evolving our infrastructure so we're where this was our infrastructure day one day two we might come in and say I really want a DNS right that points to my load balancer I want to have you know a content distribution network like fastly sitting in front of that and I want my VMs to be connected to a monitoring system so I might monitor these different VMs and so now we change our terraform configuration we enhance it by adding you know the definition of these other elements to our TF config and we rerun the exact same workflow so when we run refresh terraform will realize these first four resources exist but these new ones do not and so when we run our plan it'll tell us that nothing needs to be done with these four they already exist and they're in the desired State but that we must add three new elements to get to where we need to be and so now when we run terraform apply it goes ahead and top story alert and adds these three new things and brings the world into looking like our desired State the advantage of this workflow is a few fold one our day one experience is identical to our day two and this really is important because day two is sort of forever this is something we're gonna keep doing and then the other advantage of this is something we actually skipped which is day n when we decide we might need to decommission this we don't need the service anymore we don't want this infrastructure maybe this was just a staging environment or a test environment we then have the option to come in and destroy and when we do it destroy it's basically an unwinding of everything that's been done right in some sense it's it's a specialized version of apply which really looks at creates a special destroy plan and then goes out and talks to the real world to destroy all the elements that exist and so now we can efficiently at the end of our lifecycle go from nothing to something evolving a day over day and in the final day when we need to decommission it Tara knows how to clean up all the resources associated with that application so so far everything we've talked about is really about the workflow so it's not specific to any technology so how does Tara form actually make any of this stuff work this is where firms architecture becomes important so when we talk about the way terraform actually works there's sort of two major components one is Tara forms core so the monolithic core is responsible for a few things it takes the terraform configuration which is being provided by the user and then it takes Tara forms view of the of the world or a terraform state this is managed by Tara form itself and so these get fed into the core and the core is responsible for figuring out what is that graph of our different resources how do these different pieces relate to each other what needs to be created what needs to be updated what needs to be destroyed and so it does all the essential lifecycle management on the backside Tara form supports many different providers and providers are how terraform connects out to the rest of the world so these can be things like cloud providers things like AWS or Adger or GCP they could be on-premise infrastructure so things like OpenStack or VMware but it's not restricted to just infrastructure as a service terraform can also manage higher level things platforms as a service such as Heroku or kubernetes or things like a lambdas so these are higher level than what we would traditionally consider infrastructure as a service more in the realm of a platform as a service the other types of things terraform can manage are pure software as a service very high level things maybe monitoring like data dog or CDNs like fastly or even things like managing github teams so these are what we'd consider pure software as a service these are externally managed services but the view that we take as terraform authors is that all of these pieces are connected when I talk about a modern infrastructure I might be composing all of these different resources so I might have some set of my infrastructure that is infrastructure as a service and then I might compose that and say you know what I'm gonna provision a kubernetes cluster and on top of my kubernetes cluster I'm going to define a namespace and I'm going to define services and so these now exist on top of infrastructure as a service and I might compose even these with higher level things like DNS and CD ends and so all of these are important pieces of our infrastructure we can't deliver our application without the ayahs without the platform-as-a-service and without the higher-level software-as-a-service they're all part of our logical end-to-end delivery and what we want to get to with terraform is a single unified workflow we don't want it to be you use one tool to manage this and now you have to figure out you know how you get past this disjoint experience to manage the next section and then another disjoint experience and you have to piece together all these things instead anything that really has an API and has a lifecycle associated with it how can we enable that to be a provider such the terraform can help us solve this problem and to end and this is really where providers come in today we have over a hundred providers and those individually can manage over a thousand different resources where a resource can be you know an AWS ec2 VM or an azure blob store right these different types of resources or anything that a cloud or even these higher level things expose and that list is constantly growing terraform is a absolutely massive open source project with thousands of contributors and so every a there's new providers and new resources so how does this really start getting used at different levels within teams so this becomes an interesting workflow question around how you actually manage terraform so initially this goes back to that core workflow we talked about so we start with a single individual practitioner who's using terraform locally and their workflow is to start by writing some terraform configuration then they run terraform plan plan tells them what is this configuration going to do what does terraform think needs to be done to apply these changes and if this looks good the practitioner will apply the changes and then this cycle continues they continue to evolve the infrastructure this might look very similar to software development where we're writing some code we're applying unit tests we're committing those changes and we're continuing our cycle so what happens as we go from a single individual to now we add multiple team members we have other people we want to collaborate on managing the same infrastructure well there's a few challenges much like with software writing which is how do we make sure we have a consistent view of what the configuration actually is and how do we make sure we don't step on each other's toes and run multiple changes in parallel so if we both try and add five new VMs how do we avoid booting ten new VMs instead so this problem in some sense ends up being very similar to the problem of using git locally versus using git as a team so as we go to a team we use a system like github to provide that central collaboration and so our equivalent of that is what we call terraform enterprise so the terraform enterprise workflow augments this a little bit so what we would do instead of now running it locally and running these coordination risks has an individual we're still writing our terraform and planning locally but now we're pushing that into a version control system so this could be github could be bitbucket could really be anything but the idea is how do we move to having a central repository where we're managing this and coordinating much like we do for source control and then we drive off of this our terraform enterprise application and the goal here is a few fold one is how do we make sure the state management is done locally so like I said terraform keeps track of all the resources it's provision so that it knows at destroy time what it needs to tear down or as we incrementally make changes what already exists so this state file is an important thing to have centrally managed so it doesn't diverge or end up with sort of different lineages where you know we fork off and then independently run terraform and now the state files conflict with one another so by managing that centrally we avoid those sort of conflicts the other important advantage here is that we make sure we're only doing one run at a time right so we ensure there's the sequential application so if Alice and Bob are both working on the same infrastructure and they both try and make a change they don't step on each other's toes lastly one of the challenges is oftentimes in this environment developers are using local variables to populate things like AWS I am keys or other sensitive credentials and so we don't want these sensitive credentials to be strong about on all of our developers machines instead we want those variables to be kept centrally and in so that we don't have to worry about these things getting exposed or leaking out in plain text we can put them in a central place and version control and access control who can actually modify and view these things so this sort of changes our workflow and now enables a handful of people to collaborate safely together well what starts happening as they start wanting to add even more people right so for example we might be going from sort of our early operators who are very familiar with the cloud environment to exposing it more to our development groups this is where Terra forms notion of a module comes at handy so a terraform module is sort of like a black box of infrastructure or what we can define is a set of input variables that let us toggle the behavior of this module and the module might emit a set of output variables so an example of this might be I might define a module for doing Java for deploying my java application in AWS and my inputs might be what's the path to my jar and how many instances of my java application do I want and now inside this module we can operate as a black box and deploy either VMs using ec2 we can deploy lambda functions we could deploy containers with ECS all of these become a detail hidden within this module that a consumer doesn't need to know about and their output can just be what's the address of the load balancer that hits my application so in this way we encapsulate that complexity and make it more accessible to a broader audience so how do we actually manage and share these modules in general we publish these things to a central registry so there's a central registry that exists today for terraform if you go to terraform registry terraformed io that's our global public registry and there you'll find hundreds of modules that cover common forms of infrastructure common architectural patterns and many of these are actually published by the cloud providers themselves so you can log in and see Google's official recommendation for how to provision their network or as Rose official recommendation for how to publish a virtual network inside of Asia and so these modules make it much more consumable for us to operate in these environments cuz we don't have to know every detail of it we can just go and say I want a network in Amazon or Google or Azure and fill in a few of the inputs that matter to us and let the module take care of the rest of the complexity right and so the same idea can be extended to different applications different types of infrastructure one way we can apply this same pattern internally to our organization is by running a private registry so instead of using the public registry that's available sort of globally we can run a private registry within our organization and this is a capability integrated into terraform enterprise and so what this really lets us start to do is make this split between the set of people internally who are publishers and are familiar with our internal infrastructure and how it should be managed in cloud environments and they can push in to basically a catalog or a module registry all of the different key patterns that are needed so we might publish here's how we do a java app here's how we do a c-sharp app here's how we do a database and expose this out to our internal developer community and then are much larger base of consumers can simply go into the catalog and pull out the things they need and so these can these consumers can know much less about how cloud works but still get these approved modules that are sort of the best practice internally right and for there's sort of a few ways of consuming this for people that are more familiar with terraform they can consume it just like a normal terraform module and do an infrastructure as code approach for folks that are less familiar they might want to take more of a WYSIWYG approach and they can do that with terraform Enterprise by selecting their modules filling in the different variables and basically letting the system automatically generate the equivalent terraform for them so they can either use that equivalent terraform as a starting point and customize it further from there or if they have really no interest in learning and using terraform directly they can just use the auto-generated terraform and self-service their infrastructure and so our goal really becomes how do we enable this large audience of consumers to get to self-service on infrastructure in a way that is still going to be compliant with how the business wants to do it and not create this immense workload on publishers to constantly generate custom infrastructure and custom definitions for the vast body of consumers within an organization so at a snapshot this is terraform this is how it applies in infrastructure as code approach to provisioning really looking at both the day one challenge and more importantly a more challenging lis the day 2 plus challenge and then the natural journey as we go from a single individual using terraform to a small team to kind of an entire organization who's trying to leverage terraform for infrastructure management if you're interested in learning more about Tara form or Tara for enterprise I recommend checking out our online resources thank you

Channel: HashiCorp

Views: 261,847

Rating: 4.9172716 out of 5

Keywords: HashiCorp, Terraform, HashiCorp Terraform, Armon Dadgar

Id: h970ZBgKINg

Length: 18min 38sec (1118 seconds)

Published: Tue May 29 2018