Complete Terraform Course - From BEGINNER to PRO! (Learn Infrastructure as Code)

hey dean sid here from devops directive and i want to welcome you to my complete terraform course this course is for anyone looking to level up their devops skills and add terraform as a tool in their tool belt this course assumes some familiarity with basic programming constructs as well as with amazon web services the cloud provider used for the examples but everything else will be covered in the next two and a half hours i'm going to go through a progression from the very basics all the way to a modular automated infrastructures code configuration deploying to both a staging and a production environment using terraform i alternate between theory and hands-on examples to give a solid foundation while also showing how the principles can be applied in the real world if at any point during the course you have questions feel free to ask them in the comment section below or i've added a dedicated channel on the devops directive discord where myself or other members of the community will help without further ado let's get into it so what is terraform in hashicorp's words and that's the company behind terraform it is a tool for building changing and versioning infrastructure safely and efficiently it falls into this category of infrastructure as code tools which allow you to define your entire cloud infrastructure as a set of config files which then the tool terraform can go off and interact with the cloud provider api and provision and manage on our behalf this course is structured into a number of different modules and so right now i'm just going to go through kind of the high level overview of what each of those modules contains the first section is just kind of about the evolution of cloud infrastructure in general and where infrastructure as code fits into that the second and third modules are about terraform giving an overview of the tool itself how to get set up the third one gives us a very basic idea of how to use terraform with aws in modules four and five we start to get a little bit more advanced taking advantage of variables and outputs within hashicorp configuration language the language that terraform uses as well as explore some of the other language features that enable us to make this a very powerful tool for managing cloud infrastructure in modules 6 and 7 we take a look at how you should be organizing your terraform projects including building reusable modules to make your code much more extensible and applicable to various environments and then in module 7 we take a look at how we can manage multiple environments like a staging environment a development environment and a production environment using this tool finally in module 8 and 9 we start to look at how you can test your infrastructure's code configurations and what various developer workflows might look like including automating deployment with something like github actions now throughout the course i'll be using this reference architecture diagrammed here on the right to show what you can do with something like terraform in this case it's a basic web application taking advantage of a number of amazon web services products including multiple instances running on ec2 using an elastic load balancer for incoming traffic an rds database instance storage within s3 and then amazon route 53 for dns i'm using the and i'm using the default vpc for simplicity now keeping all of this within aws was just a choice made for simplicity of the examples it is not a limitation of terraform in fact terraform can interact with pretty much anything in the cloud with an api i've also built out this reference architecture and all the examples that you'll be seeing throughout the course into this companion github repo so it's under my github username sid palace devops directive terraform course you can see the the folder structure here is organized along with the modules that i just built and so with each module we'll have some examples i'll show what those examples look like and go through them in the the video but if you want to follow along and or use these in your own learnings you're welcome to take this repo fork it modify it in any way and use it as you learn and deploy your infrastructure with terraform with that overview out of the way let's go ahead and move into part one of the course if we think about what tech companies who are building applications that were deployed to the web needed to do in the early 90s and 2000s here's what it looked like you would come up with your idea you would then need to write the software for your application and then you need to go off and buy a whole bunch of servers and set up a data center somewhere handle all of the power management and networking and operational overhead that comes with running your own data center and this was a very challenging process it meant that it was much harder to get started if i wanted to host something there weren't companies out there like aws like google cloud to do so at the click of a button instead we often needed to buy capital expenses like big servers and deal with all that ourselves this shifted quite a bit in the 2010s so now again you have your idea you program it up on a much more modern personal computer here and then rather than provision your own servers you deploy to the cloud it's become pretty much the de facto standard you can obviously still buy your own servers and and host them yourselves but most companies when getting started would rather have a on-demand resource that they can pay a a cloud provider to spin up and spin down servers versus having to deal with managing all that themselves so it's a very different world than it was uh back in the 90s and 2000s the the major differences is that infrastructure is now provisioned via api so application program interfaces this is the interface that someone like aws provides us where i can go in issue a call to their system and say hey i need five more servers and within a couple minutes those servers will be online the speed at which those servers can be brought up and down is a game changer in terms of how we think about those things if i'm buying a server that thing in order for me to get my money's worth out of it it needs to be running operational for many years before i'm going to be able to pay off that that cost now if i know that i have a big demand coming i can scale up i can triple the size of my infrastructure in a few minutes and then let's say it was black friday and that sale is over and the traffic has has gone down i can destroy those just as quickly we used to think about infrastructure as long lived and mutable by that by mutable i mean it can change so i would spin this up i would install my operating system i would install all the dependencies that server would be managed over time i'd have system administrators who were keeping making sure that they were patched and up to date some of that starts to be offloaded that responsibility is offloaded onto the cloud and now we think about cloud infrastructure as short-lived and immutable so rather than having a server that's up for years on end and is is being modified and different dependencies are installed and changed and patched now we if we need to change the dependency oftentimes we'll just provision a brand new server with those dependencies already installed and tear down the old one and so each individual unit is the short-lived immutable thing that we're never going to change and that's that's kind of a paradigm shift in how we think about infrastructure for web applications now there's three main approaches for provisioning cloud resources the first of which is kind of what most people when they're first getting started with cloud you'll go and you'll log into your cloud provider and they give you this nice graphical user interface the cloud console and just by clicking around you can you can interact with all of their services this way the second method is via an api or a command line interface so all of the major cloud providers give you access via a command line interface for aws it's called aws and you can type out aws ec2 new instance etc and it will it will do the same thing as clicking in the interface but it's a little bit more easy to interact with programmatically so that is important and then the third approach is infrastructure as code and so that's the one that we're really focused on within this course and it enables you to take your entire infrastructure and define it within your code base this is really good for a few reasons one you know exactly what is provisioned at any given time it's very easy for someone to go into the gui and provision something you set all these options as you're going but then it's very hard to reason about what exactly the state of uh what your current infrastructure is and let's say you're provisioning multiple environments a staging environment and a production environment you don't have any guarantees that those are the same and so by defining things as code as configurations within your code base you can have much higher confidence about what you actually have deployed and use the power of programming languages to have multiple copies of the same thing and be confident that they're deployed identically and so what actually is infrastructure as code i'm taking a definition that i really liked out of a book called terraform up and running by javini brinkman it's a great book that you should take a look at but there's a number of different categories of infrastructures code tools the first of which is just an ad hoc script so maybe you're writing a shell script that makes some calls to amazon and says provision me five ec2 instances um and that's kind of the baseline of like is that really infrastructure code i'd say it's kind of borderline but it does allow you to have something in your code base that is telling you what infrastructure to provision the second category of infrastructure's code tools is configuration management tools and so this is things like ansible um or puppet or chef where you are they are really positioned to manage uh the software that is running and the configuration of infrastructure and so these are more well suited for on-prem setups where you're provisioning some hardware and then you need to manage how the you need to manage what software is installed and how those are configured the third category is server templating tools and so this is this this category is for building out a template for what you're going to provision onto a server so if you've heard of an ami an amazon machine image or the basically any virtual machine image is provisioned from some template and you can build in all your dependencies into that template and so that's the third category of how you can actually build that template so you can spawn multiple copies of the same server over and over the fourth category is orchestration tools the most popular these days in terms of orchestration tools is kubernetes which is for orchestrating containers there's a number of other orchestration tools out there as well but these are for how you can define your application deployment less less in terms of defining the the servers behind it but how you can take your code and deploy that in a certain way onto whatever system you have provisioned in the background and then the fifth one is the provisioning tools and as the name might suggest that's focused on provisioning those cloud resources to begin with and an important thing to call out here is the concept of uh declarative versus imperative and so what i mean by that is declarative tools you define the end state of what you want i want five servers i want one load balancer i want an s3 bucket etc and then the tool manages what api calls need to be made and how to actually make that happen imperative on the other hand is you tell the system what you want to happen and the sequence in which you want them to happen and so some of these tools a lot of the configuration management tools for fall more on the imperative side uh they do offer some utilities to make them more declarative and make those scripts item potent so you can run them multiple times but a lot of the provisioning tools which of which terraform falls into they are primarily on the declarative side so you specify the end state that you want your infrastructure to take and then you let the tool handle the details of how to actually get there and so with that breakdown of these different types of infrastructure as code tools like to actually take a look at kind of the landscape of what software exists that falls into these categories and how they how they actually work the first way to look at the landscape is in terms of cloud specific versus cloud agnostic so on the left hand side cloud specific is things like cloud formation things like azure resource manager or google cloud deployment manager these are all tools provided by a major cloud provider and they are focused on provisioning infrastructure within that cloud so cloudformation is an aws specific tool if you need to provision something within aws it's great if you need to provisions anything outside of aws you're kind of out of luck same thing for these other tools they're really focused on provisioning infrastructure within a specific cloud on the cloud agnostic side these are tools which can be used across any cloud provider so things like terraform things like pollumi there's a number of other tools as well they can be they can interact with almost anything with an api online and so if you have your application deployed across multiple clouds or you want to be able to use auxiliary services maybe you're using cloudflare or you're using atlas for mongodb or some other third-party hosting service that's outside of your primary cloud having a cloud agnostic approach is a really powerful thing and so that's just something to think about as you're looking at infrastructure as code for your project are you planning to have everything within one cloud or may it span across multiple clouds and so terraform having that cloud agnostic approach is really powerful if you might have resources outside of your primary cloud so let's move on to part two of the course and this is kind of an overview of terraform itself and how to get set up and authorized with aws the process of authorizing terraform to work with each individual cloud provider or any tool is quite similar and so we'll go through that process and what it looks like for aws and there's documentation online you can follow for for any of the other systems that you might need to interact with okay i talked about this a little bit in the intro and again here's that definition from hashicorp terraform is a tool for building changing and versioning infrastructure safely and easily and so why would we want a tool that can do these things the reason is that it enables us to take all of the learnings and the best practices that have been developed for software development and apply them to infrastructure so if we we want to be able to use version control to understand the changes from one one day to the next we want to be able to have code real code reviews on software so we can sure quality is high and bugs don't slip through infrastructure's code allows us to take those practices and things like them and apply them to infrastructure development the other thing that's great about terraform is that it is cloud agnostic and is compatible with many clouds and services and can interact with pretty much anything with an api and so in this diagram here at the bottom we're taking our configuration files that live in our version control system we pass them through terraform and we get out a a set of servers and networking config provisioned in the cloud provider that we can then go off and use in the previous module i talked a little bit about the differences between terraform and ansible and the different categories of infrastructure's code tools and there's many common patterns for using terraform with some of those other types of tools so terraform is a provisioning focus tool we might use it with a configuration management tool so let's say we have our setup and terraform is going to provision a number of virtual machines for us we could then take a tool like ansible and install all the necessary dependencies inside of those virtual machines so let's say we're using some standard base image like the that is provided by amazon itself we then might need to take that base image which is running ubuntu or some other operating system and install all the necessary dependencies for our application we could manage that piece of it with uh with ansible so that's one pattern that some companies use we also can use terraform with uh templating tools so environment templating tools this is the logo for a tool called packer which is also from hashicorp and in this case uh terraform provisions the servers and packer is used to build the image from which those virtual machines are created so rather than provisioning and then installing and configuring like we were with ansible now we can pre-package all of that into a machine image that we can provision copies of with with terraform so templating tools and provisioning tools pair really nicely together if if that's what we want to go with and we can actually even build our application code into that server template so that once it's provisioned not only does it have all the dependencies it also has our application bundled right in now a third pattern that i really like is combining a provisioning tool like terraform with an orchestration tool like kubernetes so in this case we're using terraform to provision our kubernetes clusters maybe it's a managed cluster like an elastic kubernetes service eks cluster with an aws maybe it's a self-managed cluster where we're provisioning a bunch of virtual machines and installing kubernetes onto it but we're using terraform to define the cloud resources and then we're using kubernetes to define how our application is deployed and managed on those cloud resources so i think that's another common pattern that is very powerful to use when thinking about infrastructure as code now before we actually go and set terraform up i want to talk a little bit about how the project is architected so at the very center of terraform we have what's called terraform core and this is kind of the engine that takes our configuration files so that terraform config there on the bottom in conjunction with our current state of the world so the terraform state file which terraform manages and it basically contains references to all the arc all the infrastructure that we've already provisioned so it can take those two inputs and then it needs to figure out how to interact with the cloud provider apis to make that state match the config that we want it to and so what happens here is we actually have broken out what they call providers and so these providers are kind of like plugins to the core that tell terraform how to map a specific configuration let's say for aws onto the current state of aws's api or if we're provisioning something in cloudflare we need to take that configuration and map it onto the specific set of api calls to achieve the desired state and so there's many different providers that are available you'll find that pretty much any major uh internet service that you would want to use will likely have a provider there's i think there's over uh at least over a hundred uh maybe even more and so you'll be able to find good coverage of the types of resources that you want to provision and use those you install them alongside terraform core so that you're able to authenticate and then make all these necessary requests to the proper api and so that just gives a high level overview of kind of how the terraform system and ecosystem is set up let's go ahead and get this installed on your system okay so for this first demo portion i'm just going to go through a few quick steps of getting started with terraform basically what i just described in that last module i'm going to show you how to install terraform authenticate with the aws provider i kind of do a hello world config and then i'll actually go and provision a virtual machine on aws using that configuration so to install terraform i'm using a system with mac os and so i'll do just brew install terraform and that'll go off and download it from uh the package manager uh we also there's a number of other ways you can install if you just google for install terraform it'll take you to the hashicorp documentation uh they've got a few different options here one you can just download the binary the pre-compiled binary from github you can use homebrew if you're on windows you can use chocolaty so i'd just suggest going to this page and installing with the the method of your choosing here we see it's already installed we've got the latest version 1.1.5 and that should be good the next thing is authenticating to aws and so for aws i have created a uh a user so if i go to i am here i've created a specific user for this course and given it the exact set of i am roles that are needed to perform the different actions that i'll be doing throughout the course if i go here to the user groups i've got this terraform course group within this we can see that this terraform course group has rds access because we're going to provision a database later on ec2 access will be provisioning virtual machines and ec2 the ability to manage am roles because we'll also be using terraform to set up some ion policies uh we'll be creating s3 buckets and dynamodb tables as well as some route 53 rules so this is the set of permissions that are required for the configurations that are within the github repo so if you want to duplicate this within your own account that could be useful and then i have installed the aws command line so i have the command line installed here if you need to install that i would just google for aws cli and then over here on the right it has instructions on how to install for various operating systems once you get that installed you'll want to run the aws configure command and that will allow us to pass in uh three things one an access key id and so this is going to correspond to the service account that we are using and that terraform is going to use to actually provision this stuff so if we go under users and we choose this terraform user under secure security credentials and then i'll create a new access key and so this access key id we'll use we also want the secret key and don't worry i'm going to delete this key as soon as i finish recording and then a default region to use and us east1 is fine so i'm just not going to put anything i will leave the output format as json and so now that has populated a file in my home directory aws this credentials file and so inside that credentials file i now have the secret keys and terraform if we don't if we don't do anything else we'll use that we'll use those credentials to authenticate on our behalf at this point we have terraform installed and we are authenticated to aws and here is essentially the basic the most basic terraform configuration that we can have in this top block we specify which providers we're going to use and so i'm using the aws provider specified at version 3.0 within here i'm defining a default region for that provider and then this is corresponding to an instance within ec2 i'm naming it example we're providing a an operating system so this particular string corresponds to ubuntu 20.04 in the u.s east region so that's telling it when we provision this vm we want to be running that ubuntu operating system and then we're going to use the t2 micro instance type there's a whole bunch of other options that we can use and i can show you how to find those within the terraform documentation so here under the aws provider we can see this resource aws instance and the types of fields that are allowed so we can specify an aw an ami like we did you can tell it which availability zone you can tell it how many cpu cores there's all of these options most of them are optional and so i'm just giving it the very minimal amount of config so that we can see how this process works now if i navigate into that directory so we're within the o2 overview directory i can do terraform in it and that will initialize the back end because i didn't specify anything else it's going to store this this back end and state locally we'll hear more about that in future modules but this is the first step to using terraform you initialize within the directory where your code is stored now let's run the terraform plan command that's going to go off and query the aws api and compare hey what is currently deployed and how does that compare to the the resources that we've specified in our main.tf there and it's showing us that that example instance does not exist and so if we run apply it will be created with all of these attributes these are going to be known after apply because we didn't specify them so it's going to figure that out once we actually do the apply and so this is looking good and i'll go ahead and run terraform apply and that's going to go and take the actions that we saw on the plan and go off and provision that it prompts me if i if i do indeed want to proceed i do and so this can take a little while while that instance provision's in the background i can go into the aws interface and actually see it happening so if i go back to here and now go to ec2 we should see we're in the correct region northern virginia and we see this instance pending so this is being provisioned as we speak we can pull this over here we see it's creating apply complete resources added if i go back to the aws page we can see that it's now running and so that terraform apply was successful i was able to create the instance and that's essentially the most basic configuration that you can use we'll be building upon this in future modules but i just wanted to show the getting started process installing authenticating and provisioning something with terraform now i'll do the final step here to clean up the resources so we don't end up leaving them running and having to pay for them and that's the terraform destroy command it's going to ask me whether or not i would like to actually do that i do and it's going to go off and destroy that instance in the background we see it shutting down here on the right hand side and that should be all we need to do at this point let's jump back in and learn a little bit more about terraform and some of the other features as we start to build out our infrastructure using it now that you've seen that hello world example walked through let's take a look at some some more basic terraform usage as we saw the sequence for interacting with terraform configurations is generally these four primary commands there's the init command which initializes your project the plan command which takes your configuration checks it against the currently deployed state of the world and your state file and figures out the set of the sequence of things that need to happen to provision that infrastructure the terraform apply command takes that set of commands and applies them so that you end up with the infrastructure you want and then the final one if you're cleaning up resources after doing an example like this or if you are just if you are taking down infrastructure that was being used previously but is no longer needed is the terraform destroy command this is kind of the general sequence of things that you'll see happen over and over and over again as you're working with terraform as a reminder the architecture looks like this with the core providing that engine for parsing your configuration in the state files and the providers mapping what's needed from the core to your cloud providers themselves and there's many many different providers if you go to the registry.terraform.io you can find a list of existing providers you can also because it's open source you can actually program your own providers for any other resources that you would need to provision here i've actually zoomed in on just the aws specific provider so if we were to click on that orange button we would see this specific provider from aws you can see it has that official tag so it's the official aws provider if it doesn't have the official tag it may just be some third-party individual or organization that's managing that so the official tags give us a higher level of confidence and trust in this provider being up-to-date and high quality and then there's many modules associated with that provider down below finally the way that we use providers in our code we saw this in the hello world example is you specify within that first terraform block the required providers and you can pin the version so you have a specific version of that provider and then each provider may have some set of configuration that is required for example specifying the region for the aws provider i'm going to go through a little more in detail each of those commands that we've already seen but actually explain what all they're doing so the terraform init command and on the right hand side i've run the the tree command which is going to show us the set of files in our directory and that will help us understand the sequence of events that happen when we in it so we've got our working directory here there's nothing in it only our main.terraform file containing that hello world example is in it so we see that here on the right when we run the init command that actually goes off and downloads the associated providers that we defined in that terraform block so it's going to get the code for the aws provider from the terraform registry it actually downloads that and puts it into our working directory so if we run the tree command again we see oh now we have a dot terraform hidden directory with a provider subdirectory a registry.terraform.o directory and what that's showing is that's the official registry but you can have additional custom terraform registries or third party registries where those providers are actually stored and then we go all the way down we see that final directory we're seeing the version the architecture and then the actual code for that provider lives in that final sub sub directory we also now have a lock file and that lock file contains information about the specific dependencies and providers that are installed within this workspace the next thing that it does is if you have used any modules and we'll talk a little bit more about what modules are in the future but they're essentially a way to bundle up terraform code so that it's reusable if you're using any of those it would go and download that as well and pull those into our working directory and so the where those get slotted into our into our file system is also within that terraform subdirectory we have the providers and then as a sibling we have this module subdirectory and so that would go off and download and store all of the terraform configurations associated with those and now another important concept about terraform that we need to know is the state file and the state file is terraform's representation of the world so it is a json file uses the json file format and it contains information about every single resource and or data object that we have deployed using terraform so resources resources can be anything in the cloud provider itself as you can see here on the right this is the portion of a state file referencing a specific amazon instance that we have provisioned it's got lots of metadata about that instance such as the ip address or the or the arn the the unique id for that uh resource within aws there's many more attributes that i've pulled out just to save space but you could imagine this having all the information about all the resources we've deployed there's also the concept of a data object so within terraform there are blocks which correspond to resources there's also blocks which correspond to data and those maybe we're pulling some information from a third-party api or we could have some fixed data within our code and those could be used to reference things that that were not provisioned and are not managed by terraform but we want to pull those in to influence how the infrastructure is actually provisioned now finally the state file will contain sensitive info so if we're creating a password on the fly for a database that we're provisioning that password will be stored within the state file so you need to protect the state file accordingly make sure that it's encrypted and the permissions are set so that only the correct set of individuals would have access to it so that's just something important to note about the state file you can also store the state file locally or remotely when you first install terraform i believe the default is for it to be stored locally so it'll be within the working directory of our project we also and this is how terraform is generally used in within a company where you have multiple people working on it you can store that state file remotely usually in an object store like a s3 bucket or google cloud storage and we'll we'll talk about what that means and how to set things up accordingly so the first option is a local backend so this is the simplest option to get started we've got you the individual we've got our laptop or desktop computer and we've got our terraform state file and this is great because it's super easy to get started it just sort of works out of the box it'll store the state file right alongside our code it's not so good for a number of reasons so one it will have our sensitive values in plain text within that json file like i was just describing so that's not great to have things like that on your local system it just provides a potential attack target it also is uncollaborative so there's no way for me to take a state file living on my laptop and have someone else work on it in an easy fashion so that makes it very challenging to work with other engineers on your infrastructure configuration and three it's very manual so every time i'm interacting with these configuration and applying it i generally need to run a terraform plan or a terraform apply command within my command within my terminal and that is not how we want to be doing this ideally ideally we want to automate a lot of this and so if we think about what a remote backend enables we can separate the individual developer on the left hand side from the state file which is now stored in a remote server somewhere one option is terraform has a managed offering called terraform cloud that will host our state files for us and manage things like permissions etc we can also self manage a remote backend to store those state files using something like amazon s3 google cloud storage there's a number of different remote backends that we can configure this is great because it allows us to encrypt all the sensitive data that would be within that file and gets it off of our our local system so that's a big win it is also good because now we can have multiple people that are all interacting with this same remote back end so it makes collaboration with other engineers much easier it uh allows us to automate things so now because we are no longer dependent on running those terraform commands locally on our system we can run things like github actions or other automation pipelines that can interact with this remote state as well the one downside is that it does add increased complexity compared to the the local back end so if you're just an individual getting started with terraform it's easier to get started with the local back end but all these benefits generally outweigh the con of the the slight increase in complexity there let's move on to the next the next command within the the general sequence and that's the terraform plan command and so what it is doing it is taking our terraform config which we're mapping which we're defining on our system that is the desired state so what we want our our infrastructure to look like and then it compares it with the terraform state which is the actual state of the world and that's a slight misnomer because you could have gone in and modified something within the gui or sort of out of band of our terraform workflow as long as you haven't done that the terraform state should represent the actual state of the world if you have done that you can actually get yourself into trouble so you should you should always try to keep you should try to avoid making modifications to your infrastructure outside of the terraform workflow that being said we compare the desired state with the actual state so let's say in our terraform configuration on the left hand side we've got a network configuration we've got four servers that we want to provision we've got a database but previously we've already deployed this and we've got our network configuration three servers and a database so when we run that plan command it's going to go through our config and say okay the network config looks identical great the database looks identical great however we've got a mismatch in the number of servers that we want provision we're scaling up we told it we want four but there's only three currently deployed and so the plan command is going to bring that in and say i need to add one virtual machine here that plan can then be fed into the aws provider and it can figure out the sequence of api calls that are necessary to actually provision that vm now if we go and run the terraform apply command it's going to go off and do that so we plan we hit the apply command and it will create that new resource within aws and then great we have our configuration our desired state matching our deployed actual state that's what we wanted now finally the destroy command so let's say this was just an example and i'm cleaning up and i want to to get rid of all these resources now what that does is i issue the command and it will go off and destroy everything within that configuration as associated to this particular project and so this command you only want to run when you want to clean up at the end of a project you do never you never want to run this for a live project that you actually are still executing on so just a warning that's what happens i talked earlier about local back ends versus remote back ends and so right now i want to give an overview of the two primary options for dealing with remote back ends the first of which is terraform cloud so this is a managed uh offering from tara from hashicorp itself and so to use the terform back-end within that initial terraform block you would specify a back-end field of type remote and then you within the web application for for terraform cloud you would have created a organization and a workspace name and so we can specify those two attributes here and this is kind of what it looks like on the web ui you can go in you have your account you've created your organization and your workspace and when you deploy to that you can see it here and interact with it and so this is a this is a very easy way to get started and it can be great it is free up to five users within an organization however if you need more than five users so as your company grows it does start to cost twenty dollars per user per month so as the number of developers increases it can be expensive to pay for this and so while terraform itself is free to use this is how hashicorp actually makes money from this product is through this managed offering the other option for remote back-end is a self-managed backend and so aws has one option there's also a gcp and an azure option here but for the aws option you're specifying an s3 bucket as well as a dynamodb table and so the s3 bucket is where the state file will actually live and we can tell it whether we want it to be encrypted or not generally you want that to be true but for the dynamodb table the important thing there is that because we could have multiple people working on the same project at once you want to prevent two people from trying to apply different changes at the same time so we use that the atomic guarantees that the dynamodb offers to if i issue a command i can lock the terraform configuration such that if someone else one of my colleagues issues another terraform apply command they will get reject that apply command would be rejected until mine has finished and so this just prevents us from getting into a weird state where two things are being applied at once and that could cause issues however you might be thinking sid how are you going to provision these s3 bucket and dynamodb and dynamodb tables given that we want to provision everything with infrastructure's code but we don't have these resources provisioned yet so it's a bit of a chicken and egg problem and so there is what uh i'll refer to as a bootstrapping process that will enable us to provision those resources and then import them into our configuration so that even the remote backend resources can be managed by terraform as well and so the way that we do that is first we specify our terraform configuration with no remote backend so it will default to a local backend we then define the resources that we need that s3 bucket and that dynamodb table it's important here that the hash key is lock id so that that's a key attribute that needs to match this exactly in order for this to work we then would go through our normal apply process so we would run terraform apply it would tell us here's what's going to be created a dynamodb table and an s3 bucket we say yes it goes off and applies though so now within our terraform state file we have those two resources within our aws account we also have those resources provisioned we can then change our back end so before we didn't specify backend now we're specifying we want to use the remote backend the s3 configured back end with the proper uh configuration applied this is all the the configuration that we had before that's unchanged and now if we go and rerun our terraform init command it will recognize oh before you were using a local backend now you have a remote backend do you want to import that state into the new backend this is where we'll say yes it'll go off upload that state file into the s3 bucket and now we have our state in that remote back end including the bucket the s3 bucket and dynamodb table that we're using as the the resources that are backing that so it's kind of a tricky thing with that bootstrapping process but that's how we can go from having nothing to having our the infrastructure needed for the remote backend into our terraform config you can see our state files within that bucket itself they will be encrypted and the dynamodb table as i said is just used for locking and unlocking uh the state so that we can apply safely without of without having conflicts between two separate applies happening at the same time now that we have a better understanding of how terraform state and terraform back-ends work as well as a few different options for hosting remote backends i'm going to go ahead and walk through that initial architecture that i described early on in the course and show how we could build this out with the terraform configuration that's going to involve a number of different aws resources now because this is not an aws course i'm not going to go into too much detail about how i'm configuring each of the things but i'll walk through each of the elements within the configuration and then we can apply it and see it come to life also within this basics subdirectory within the course repo i have two i have both of the back end configurations that i showed in the the walkthrough uh for the aws back end uh we've got the definition including the s3 bucket as well as the dynamodb table so i've actually already provisioned those and that's the remote back-end that i'll be using so here we can see within the s3 portion i've got my terraform state bucket and then within dynamodb i've got my state locking table and so that's the remote back-end that i'll be using feel free to use terraform cloud or continue to use a local backend for now if that suits you best within this web app directory i've defined all of the resources associated with that architecture that web application architecture and so let's just walk through that quickly then we'll go through the plan and apply steps and see it provision within aws so you'll notice we now have this additional backend block within the terraform configuration this is where i'm specifying that remote back end before i didn't have that block and it just assumed i wanted to use a local backend i'm using the s3 type and specifying and pointing it to that s3 bucket with that dynamodb table for locking and this is where within the bucket it will store that state so if it's in the devops directive tf state bucket it'll be at this prefix so i could actually go navigate to that in the bucket and see that file also once again i'll be using the aws provider configured to operate within the us east1 region now the first two things that i've added here are the two ec2 instances so this compute portion the two ec2 instances i'm using the same operating system as i used in my hello world example it's going to be a t2 micro instance once again the two new fields here are the security groups and we have to set up some security groups to enable inbound traffic by default they wouldn't be accessible for inbound traffic and so because we want this to be a web application we need traffic to be able to reach it and then this is kind of a little hack to set up the most simple web server i could it's a bash script that populates the index.html file with hello world one and then uses python to start an http server on port 8080. i did the exact same thing for a second instance so we can have multiple replicas of our of our web application running this one has hello world 2 and then we start up the web server just as before so once we've provisioned these we're actually going to put a a load balancer in front of them so when we refresh the page we'll be able to see it hitting one instance and then hitting the other and that's kind of how to demonstrate that i'm creating an s3 bucket i'm not actually using it for anything but just wanted to show if you did have large files that you wanted to store you would likely populate those within s3 and maybe store reference to them within your application database the configuration for an s3 bucket is quite simple i'm just naming it web app data and i am going to turn on server side encryption i think by default it is off so it's useful to go ahead and specify what kind of encryption you want for that bucket on the server side when provisioning things we often need to specify which virtual private cloud vpc and which subnet within that vpc we want our resources to go into uh because i didn't want to configure a new vpc for this example i'm actually using the data block here rather than resource block so data's data blocks will reference an existing resource within aws so i'm just going to reference the default vpc which is created in my account by default and the default subnet within that vpc and so i'll reference those later in in other resources like i said i needed to define some security groups in order to allow inbound traffic so that's what this is and then you attach a group rule a security group rule to that security group it's kind of like i am policies where you have the users and then you can attach roles to those users here we have security groups we attach rules to them i'm setting it up so we can have inbound traffic on port 8080 to the instance using tcp protocol and allowing all all ip addresses for that next up we're going to set up the load balancer and the configuration for that load balancer to have inbound traffic uh coming from the the web uh we have to set up a variety of things we're going to be listing on port 80 we're not going to deal with setting up certificate and https we're just going to go with standard http to keep things simple and this is just kind of the way that we configure that load balancer listener such that if we hit a url that we don't recognize that we haven't configured it'll return this 404 page but if we do get a request that we recognize we forward that to our our instances we can specify where we want to send that traffic by defining a target group and that target group is going to contain our ec2 instances with some various health check information we then need to attach our two ec2 instances into that target group so that the load balancer will know where to send the traffic and on what port we need to set up a listener rule so that we can in this case we're just going to take all paths and forward those along if you had more specific rules you wanted to set up you could do so here we need slightly different security groups for the load balancer in terms of the traffic that it's accepting i'm allowing it to have inbound traffic on port 80 and then setting up an egress rule for outbound traffic as well finally with all of that config with all of that configuration in place we can define the load the load balancer itself tell it which subnet to provision into and which security group to use the next piece that we'll want to touch is the route 53 for dns so setting it up so we can type an actual domain into our browser and access our site so in this case i'm using a domain that i own devopsdeploy.com and we provision it as a zone within route 53 and then each zone can have different records associated with it and so within the zone we'll have an a record it takes traffic to devopsdeploy.com and we'll point it at our load balancer so traffic coming into devopsdeploy.com we'll hit our load balancer and that traffic will get forwarded to one of our two ec2 instances now finally the last piece of infrastructure is the database similar to the s3 bucket i'm not actually going to query i'm not actually going to set up an application that connects to it i just wanted to show how we would provision it and so we've got that database there and this is the set of config that we need to provision the rds instance let's go into our web app directory run our terraform init that's going to initialize that remote back end so set up everything we need to to talk to that s3 bucket and dynamodb table and then i'll do a terraform plan and this is because i haven't provisioned this yet this is going to show me a massive plan where it's going to actually go create all of these resources so we're adding all 17 of those resources uh nothing exists so we're not changing anything and and we're not destroying anything this should match our configuration pretty much everything that i just walked through and let me just go ahead and do terraform apply uh it prompts me if i want to apply it there is a command line flag i can use to avoid that prompt like if we're doing this within an automated system and we'll use that later when we set up a github action to automate some of this but now it's going off and we can see it creating all of these resources some are quite quick some are slower i think the rds instance is probably the slowest but we can go click around in the interface and start to see things come online if we go to ec2 instances let me go to the right region okay we've got one instance online we've got our load balancers coming up it's provisioning we've got route 53 let's check on that uh we've got a hosted zone which is what we wanted for devopsdeploy.com uh cool uh now this domain is actually uh hosted in google domains and so i need to update the name servers to use these yeah so i'm going to go under dns and use custom servers i had it using cloudflare before but now i'm going to update to use these aws name servers there add another there one more save and hopefully that propagates quickly but it could take some time in the meantime in the meantime let's go check on that load balancer again because even before our dns name is resolving we should be able to [Music] we should be able to hit the load balancer directly okay so our load balancer is now active and has this dns name so if i hit this domain we see that it has hit our instance number one if i refresh a few times we start to see sometimes it hits number one sometimes it hits number two so that is a good sign that our load balancer is connected to both and we're able to access things accordingly um let me just go to devopsdeploy.com and see if that domain has propagated and it looks like it has and so when we refresh sometimes we get one sometimes we get two but they are active and how is our provisioning coming along so it looks like everything has provisioned except that db instance and the rds instances are one of the slower resources within aws so that can take some time but if we look here we'll see it in the provisioning state databases terraform and so here is that instance uh it says okay the security group is active the instance itself is available so does that mean it's done yeah so right after we got there it finished we successfully provisioned all of these elements and configured them to work together now if we're actually building this out there's a number of steps that we would have taken to make our terraform configuration much cleaner and as we learn more about the different features of hcl and terraform throughout this course we'll be taking this base and extending on it in this example we put everything in this main.tf file it turned it into quite a large file that's kind of difficult to reason about we would likely have broken that into maybe the different portions of our architecture so we could have one for compute one for networking one for persistent storage perhaps we can start to pull out some things that we've hard coded here into variables so rather than have a specific machine type here we could actually have that be a variable and we'll learn how to use variables to do that type of thing obviously also if we had an actual application we would load that within our instance instead of our little bash script and we would communicate with the database and the s3 bucket but because the focus of this course is on the terraform aspects i'll mostly be upgrading how we're using this configuration and making it more extensible and cleaner as we move forward now once again i'm going to do a terraform destroy to clean those resources up so i don't end up with a big bill it says this will destroy all 17 resources yes and we'll let that deal with that in the background while that example showing the different components of the web application how we define those resources and apply those resources we already have a pretty powerful set of tools under our belt but we've only just scratched the surface of what the hashicorp configuration language hcl can do and in particular there were a lot of things that were hard coded there that we can break out as variables and make our code much more flexible and much more modular so in part four i want to look at variables and outputs as two features of that hcl language there's a few different types of variables that we can use within terraform the first of which is an input variable which is referenced using var.name of variable so over here on the right we can have a variable we'll name that variable instance type and then that can be a number of different types here it's a string and we're setting it to t2 micro so here we might use this if we were provisioning ec2 instances and we wanted to make that configurable such that we ran uh we ran our terraform apply command we could choose based on a separate variables file whether we wanted a small instance a large instance etc so you can think of input variables like input parameters or arguments for a function we also have local variables so these when you reference them use local.name but when we declare them we use locals plural so that's important distinction and these are like the temporary variables within the scope of a function if you're thinking about it in terms of normal programming languages and these just allow me to take a value which is repeated a few times throughout my configuration and pull it into a variable that can be reused so i might say this service is named x or i am the owner of these these resources then finally there's output variables using that same function analogy these are like the return value of the function and these are what allow us to take multiple terraform configurations and bundle them together so i might take the output of my apply and return a instance ip address so that a different portion of my terraform config can consume that and use it to set up configuration elsewhere so if i need to map or set up a firewall rule so that my so that my web application can talk to my database i can use an output variable like this to get that to get those data and apply the configuration appropriately now in order to set input variables there's a number of way we can actually apply values to those and these are ranked in order of precedence so the first on the list is the lowest precedence and the the final on the list is the highest and so if you've declared a variable in multiple ways this is the ordering in which one will be applied over the other you can if you don't specify a variable anywhere and there's no default value when you run the plan command the terraform cli the command line interface will prompt you and ask you to put a value in you generally don't want to be doing it that way because then it makes it very easy to make a typo or have a mistake such that the variables change throughout different runs so probably want to avoid that first one altogether the second one is if you have a default value in your block when you declare the variable itself that's kind of the fallback of what it will use if you don't specify a value you can also use environment variables to specify them so if you you need to have that environment variable start with the prefix of tf underscore var underscore and then the name of the variable this is sometimes useful in ci environments or other environments where you would want to change the value based on different attributes you can also define these things within files on your file system so you can have a terraform.tfvrs file which has all the values that you want and so this can be useful if you want to have maybe a staging one or a production one you have different tfrs files for each of those etc you also can have any files that's that's a wild card there that asterisk dot auto.tfvrs file so this can be auto applied and these will actually be used over top of whatever you have in your tfrs file and then finally when you issue your terraform plan command or your terraform apply command you can have a a dash bar or dash var file option that will pass those in at the time of that we'll pass those in at runtime so that is the highest precedent so if i specify a dash var on my terraform apply command that one's going to be set to that value even if i've defined it in one of these other ways a number of different value types that these variables can hold there's some primitive types that you're probably familiar with you can have a string you can have a number or you can have a boolean so true false there are also a number of more complex types that you can have that can be constructed of those primitive types you can have lists sets maps etc you're probably familiar with these from other programming languages so i won't go into too much detail about what they are but just know that you can define variables that take on these more complex types and there's also some validation that you can have within terraform type checking happens automatically so if you specify a variable as a boolean type and then you try to pass it a number that will terraform will throw an error when you try to use that command and so that's very useful to help us avoid issues with with passing the wrong values into these variables you can also write your own validation rules that can be applied and so that can be a powerful technique to avoid having issues or to run some automated testing against your code base to do some static checks to make sure that your configurations are using the types of things that you that you care about i mentioned before how sensitive data gets handled a little bit within the state file but it is important to think about oftentimes you will use variables to pass those sensitive data in so you might be passing in a database password for example you want to set the attribute sensitive equals true when you're defining those variables if you are passing it in you don't want to have it in a file so those are those are a great candidate for using that tf underscore var environment variable approach or using the dashbar command and retrieve the value for that from let's say aws secrets manager or hashicorp vault when you're issuing that command you also can use an external secret store and so you can reference one of those secret stores within your terraform configuration and then use the output variables to pull those into other portions of your config when you have sensitive data stored within uh an object within terraform you'll see when it outputs the plan to the command line it will mask those so here let's say it had a password it would put parentheses parentheses sensitive et cetera so that you're not leaking those credentials wherever you're running these terraform commands now that we know about terraform variables and outputs i want to show a couple of examples and then apply to our web application configuration to make it a little bit more generalizable first we've got this examples directory here in a main.tf file as always we need to define our backend and our providers i'm showing how you could use a local variable here so local variables as i described are things that are scoped to within this project and we can't actually pass them in at run times way to define a variable that will get reused throughout the configuration so that if it does change we don't have to update that in all the different places within this main configuration i've got an ec2 instance defined as well as a database instance you'll notice though that now instead of passing in a hard-coded string for the instance type and the ami i'm passing those in as variables and those variables are actually defined within this variables.tf file they don't have to be in a separate file they could be in the main file but i've pulled them out here so that we have sort of a nice clean place where it def it defines all the different variables that we can input when we use this configuration so we're passing in instance name ami instance type dbuser and dbpass so these are all inputs to this terraform configuration that i can configure and change at runtime the next concept that we talked about is the tf vars file so this is where i can define the values for these variables so if they're non-sensitive i'm just going to put them in here so for example i'm going to name my instance hello world it's going to use this ami i'm just going to use that instance type you'll notice i did not put the password in here because that one we wouldn't want to define in our code base because that is sensitive and so we're going to pass that in a different way we also can have more than one tfvrs file by default it's going to apply if we name the file terraform.tfrs that would get applied if we have some other file name then we have to explicitly tell it when we do a terraform apply we need to do terraform apply dash var file equals and then path to that additional file tfrs file i also added a couple of outputs to this just as as samples of what you might include for example we might want access to the ip address of the instance once it's provisioned or we might want access to the ip address of that database when it's provisioned finally in the database configuration i specified the username and password as variables so we can apply those we could put them in the tfrs file but because they're sensitive we likely want to pass them in at runtime so we would do terraform apply var and then we can specify for example db user equals myuser we can pass another var equals db pass equals some thing super super secure uh and so those would get passed in at runtime ideally we're automating all this and so these sensitive values can be stored in something like a github secret or maybe stored in aws secret manager and accessed within that git github action et cetera but we want to avoid putting these sensitive data into our configuration itself and the variable and variables are the way that we can achieve that also when we define that password variable the db pass we want to make sure to set sensitive true and that way when we run the terraform apply it won't actually echo that out into the terminal like it would some of these other variables so those are just some basic examples now i'll show you what i went through and changed about our web application itself here are the variables that i added i'm allowing us to choose a region that we want to deploy into for ec2 i define similar to the example which machine we want to use as well as the instance type for s3 i'm allowing us to set the bucket name dynamically with a variable for route 53 we can switch which domain we're actually using for rds i'm allowing us to set the the name of the database the database user as well as the database password so as i was thinking about what we wanted to parameterize for this application i tried to choose the set of things that would allow us to deploy multiple copies of a similar but slightly different web application and this was the set that i that i thought made sense also i'm passing back the ip addresses of our two instances and the ip address of the database instance in case we wanted to consume those in either another terraform configuration or some other piece of our automation within the main terraform configuration i've now updated all those references so that wherever they're being used i'm passing in the var dot and then the name of the variable for example here and by using variables in this way i'll actually be able to down the line deploy a staging and a production environment simply by configuring different variable values within my terraform.vars file i've included the values for each of these i've included the values for each of these variables except for dbpass which i would want to pass in at runtime as i described now i'm not going to go through and actually do the terraform reply here the process would be exactly the same as before we would do a tr form in it a terraform plan terraform apply the only difference would be that now that we have these outputs once we finish the apply step it would actually log these out and provide access to those output variables from our configuration as you saw we were able to refactor our example code to use variables and outputs to make it much more expressive and modular we're able to create multiple copies of the infrastructure just by changing those input variables now let's take a look at some of the more advanced language features and how we can start to use them to improve our code even further i've listed out a bunch of things here i'm mostly just going to call them out as things that exist and you should really use the terraform documentation which is quite good to understand how exactly to use these things but we have access to expressions on the left hand side here so we can do things like template string so if you're familiar with how you can have a string in javascript with with a curly braces inside and reference a variable from within that string we can do similar things here where we can build strings dynamically we also have a number of operators which behave like you might expect from a programming languages from a programming language with multiplication division uh checking equality etc we can use conditionals so it's it's similar to a ternary syntax here where you have a condition and then if it is true you do one thing if it is false you do another so that's a pretty powerful construct we also can use a four list so if we want to let's say have a listing of configurations and loop over them to do something a number of different times it allows us to have a configuration that we don't have to repeat the same code over and over we also can splat so by that i mean take the values in a list and expand them out over some way in which we want to use them there's dynamic blocks there's constraints we can use to check typing and versions of things and so like i said use the docs for those uh we have a number of different functions so things like math functions that we can do on numbers we have date and time functionality built in where if we wanted to have something that is named after the specific time stamp at which it was created or if we need to use the hash and crypto functions to generate a password on the fly etc so there's a bunch of built-in functions within the language that that you'll want to use the documentation to understand how to use these within your code base and we'll see this as we go through and continue to build out our examples we'll use some of these functions to help make our our code better and better now there's also a concept of a meta argument there's a number of these but one specifically that i want to call out here is the depends on meta argument and so normally if there's things that need to happen in a certain sequence if you're like provisioning a server and then you need the ip address from that to pass to a firewall rule or what have you just by passing those data and saying ec2 example.output and putting that into the configuration for the the other resource terraform when you run the plan or plac command will figure out the sequence of events and the the dependency graph there there are cases though where one resource implicitly depends on another but there's no direct connection within the config and so an example here shown on the right is that here if my instance needs to be able to access an s3 bucket i need to have a role policy that can make that happen but there's no direct connection within my config and so i can tell terraform what this depends on key oh you should make sure to provision this role policy before you provision the instance otherwise it's going to fail and so this allows me to give some hints to that the parsing and the dependency graph generation to ensure ordering matches what it needs to be there's also another meta-argument count and this allows me to specify if i need multiple of the same configuration provisioned i can use this count meta argument and it will provision multiple copies and so usually this would be used with a let's say a module where i have a single block and i want to make multiple copies of it and so here this configuration on the right would provision four copies of this instance by passing that count into by passing count into the resource and then i can change the tags using count.index so that each one will be named server 1 server 2 etc it's very useful to use this if you have multiple necessary resources that are nearly identical another important meta argument to call out is the for each meta argument this is kind of like the count argument but it gives us much more control over each resource so whereas count we literally just get one two three four et cetera here we're taking an iterable of some kind and we're using those to create the multiple resources so in the example let's say we had two subnet ids up there on the top we can then take those subnet ids and go through them and do a four each where each of those subnet ids is used in our aws instance config it allows us to very easily define copies of things while still meaning the necessary control to individualize them as needed we also have a life cycle meta argument and this is important because there are certain things where we need terraform to take actions in a specific order and so we can use the create before destroy argument to say if we're replacing this server we want you to provision the new one before you delete the old one and so this can help us to avoid downtime for our applications if we do this properly there are also some times where behind the scenes after you've provisioned a resource aws or whatever service you're using will add some metadata to that resource those can be very annoying from a terraform state perspective because it looks as though you have a change between your state and the deployed infrastructure and so you can tell terraform oh yes that tag exists we don't need to worry about it and you can put that within the ignore changes uh lifecycle meta tag and so otherwise you can end up in a state where you're trying to revert those changes back and forth and you're kind of fighting with the the system the other meta-argument lifecycle tag that i'll call out here is the prevent destroy tag and so this is kind of a safeguard if you have some piece of your infrastructure that is critical to not delete you can add this tag and then anytime if the plan command if the terraform plan or apply would have deleted that resource it will throw an error and so this can help you really lock down some specific core pieces of the infrastructure that you don't want to be deleted another important concept within terraform is the concept of a provisioner and a provisioner allows you to perform some action either locally or on a remote machine there's a number of different type of provisioners there's a file of provisioners local exec remote exec et cetera there's also sometimes vendors publish a specific provisioner that allows your terraform config to essentially call out to one of these other tools and do whatever it needs to do so maybe if we're using that pattern i talked about with terraform as a provisioning tool and ansible as a configuration management tool perhaps you once you've finished your application of your terraform config and you have your server up you can use the ansible provisioner to then go off and install and modify those servers etc and so these provisioners are how you would go about actually making that happen within your your configuration or in a more simple example we could have let's say a startup script that we wanted to execute after we have provisioned our servers and so that could be a file provisioner with a bash script stored there that the terraform configuration could reference so those are just some examples of how these provisioners would be used for this part of the course i haven't actually made any direct changes to the configuration of our web application we will start to use some of these language features as we do things like break it out into a staging and production environment we'll start to look at using conditionals and other language features in this portion of the repo i've just included a short readme that summarizes some of the various features here but really the best place is to go within the terraform documentation here and use it directly because it's going to have the most up-to-date and detailed reference available in this next section of the course i want to talk about how you should be organizing your projects and how to use modules to make your code reusable as well as we'll take a look at how to use external modules if we want to provision something that is pre-configured by someone else so what exactly is a module it's essentially a container for taking multiple resources that we've defined within our terraform configuration and bundling them up in a reusable fashion and it just consists of a collection of our tf our terraform files and or our terraform.json files if we structure them as json files and we keep them together within a single directory so we've already been using a module so far we just put everything into kind of that default module but in this in this section we're going to show how to break those out and bundle them up in a reusable fashion and it's the main way that we can take our terraform configurations and reuse them across projects or across environments or share them externally if we if we think third parties might find them useful why would we care about doing this and so we've got all these software engineers and so we've got our web application we've got some asynchronous processing system we've got microservice a b and c and so here everyone needs to know a little bit about everything and that is not typically the breakdown that we want within our organization we want to be able to have some people who can specialize in infrastructure and other people who can specialize in application development you can't expect everyone to know everything and so in order to make this manageable we need to break down the system into different components and so maybe we have a handful of infrastructure specialists who are really good at terraform hopefully that will be you after completing this course and they can define the infrastructure around how we want to deploy our web app and how we want to deploy our batch processing system and how we want to deploy our microservices and they can abstract that away from the application developers who can then consume it so maybe for my web application that my company is deploying i need a web server i need a database i need some networking config and i can bundle that all up in a terraform module that an application developer can then say i have my ruby on rails app i just need to plug in these specific values provision a copy of this terraform module and now i'm good to go and i can have the best practices that the infrastructure specialist knows and has baked into this config without needing to understand fully what's going on under the hood and learn terraform to to a great extent there's a few different types of modules i mentioned before we've been using modules because the default module is just every tf file in the main working directory so kind of by default by accident we've been using this root module we also can have child modules which is a separate module that we reference from our root module these modules can come in through a variety of sources if they're all in the same file system we can have local paths so i can have a directory a and a directory b and i can reference one from the other so that's the local paths these also can be in the terraform registry so like the providers live in a registry on terraform terraform system we can also have modules that live in a remote registry and we can reference those directly we can have modules that live in github and reference them via a github url or other version control systems they could live in s3 buckets so there's a number of ways that we can reference these but just knowing how to do it for various examples is the important piece here a local path as i was saying you would you would just use a relative path from your root module to wherever that child module lives so let's say i'm i'm building my module web app and it is in a directory that is a sibling to my current root module i would just use the dot dot for parent directory reference the web app directory if it's living in the ter form registry we would specify it like such so the source would be whatever organization slash name of uh module and so that's how we do that you can pin a version so that you don't have any surprises about it changing in the background for github modules you would reference it using either over https like the first example or via ssh in the second example this is github specific syntax you can also do any generic git repo with a username and password as shown in the example at the bottom there similar to how we would pin a version for the terraform registry we also probably want to pin a version for the git sources in previous sections we looked at input variables and how they can be used when we're issuing our commands those were for the root module but each child module can also be passed inputs in a similar fashion so we can specify whatever the developer of the module has exposed as an input variable we can specify this is how we can have one module that is very generic and applicable to a variety of different sources so we can specify things like the bucket name or the domain and we'll see this when we actually refactor the example code and add turn it into a module these are the types of things that we will want as input variables to that module itself you can also then use the meta arguments that we talked about before like if we want to have multiple copies of this we would use the count or the for each meta argument to specify the number of copies and the attributes of those copies similarly we can use the providers and the depends on meta arguments as well now what makes a good module you can put anything you want in a module you could have a mega module that had everything you wanted you could have a tiny module that's just a very thin wrapper over one of the a resource from a specific provider but there's a number of attributes that make a module useful and i've listed here a number of attributes that make a module useful so we want to raise the abstraction level from the base resource type so if your module is just a super thin wrapper over the resource within the aws provider it's not going to provide that much utility to the end user they may as well just use the the resource directly but if we can take a set of resources that should be logically grouped and put them into a a configuration that are usable with just a few input variables that's where you can really get some value of having it modularized you would want to as i was saying group these resources into a logical grouping and have the necessary inputs so that i as an individual developer can get what i need out of it if if i don't have access to a specific configuration that i should have access to that module doesn't doesn't really add any value for me because i need to then interact with that resource directly we also should always try to have defaults for the values that will give us a useful and a usable output having useful defaults will make a the onboarding experience much nicer so that if one of the consuming developers isn't sure what value to use as if they use the default they'll end up with with a working nice setup and then we also want to make sure that if there's any outputs from our resources that would be useful for other infrastructure to go along with our module we need to return those as outputs so that we can integrate our module with other systems and so these are kind of the five things that i would think about as i'm building modules and making sure that they're going to be useful for my team or for external parties that are going to be used using those modules we've talked about the different sources this is the terraform registry and what it looks like in addition to providers you see there's a tab there for modules there's many different modules associated with aws and other providers that you can use as a starting point for example we could have a specific module like the security group module within this it will help us to provision a security group within aws with the necessary configurations associated with it now that we have an understanding of how modules work within terraform i'm going to do a few things within this demo portion first i'll show how we can consume a third-party module and i'll deploy this console module console is another hashicorp tool and we will deploy it using this third-party module it is available via the terraform registry and i just want to show how we can take advantage of them configuring this system with a lot of the best practices and exposing the key uh input variables to us so they've abstracted away a lot of the complexity that would be required to run console if we look at this readme we can see that this is actually going to deploy a ton of resources behind the scenes but they've only exposed the important pieces that we would want to make when actually going about and deploying this so that we can let them handle most of the setup and we can configure it to match our needs so i'll just go ahead and show you this within the the console subdirectory i've got a main.tf again we always set up our backend and providers but then we can just reference the console module via the github repo that it is stored in and so if i navigate to that directory do a terraform in it and then a terraform plan you'll see this would actually provision 52 different resources within our aws account so things ranging from ec2 instances to networking policies to im roles and so console is this relatively complex system for doing thing for automating a lot of the network setup and discovery if you have many different services it would be very difficult for us to go off and deploy this but because they've built a public module we can actually consume that module and just configure it exactly how we need using the smaller set of input variables that they've exposed to us now let's take a look at the web application that we're building and configuring and see how we can turn that into a more modular configuration and so the first major thing that you'll see if i go here i've broken it up into those different portions of the architecture i've put the compute in one place that's going to contain the ec2 instance definitions the database is here storage is going to contain the s3 definition networking etc so i've broken it up into the different components i've kept our variables.tf file i've added a few here that we can take a look at including the environment name so this is going to allow us to split on dev versus staging versus production and avoid some naming conflicts because i'm deploying into a single aws account but with so now our main.tf all it contains is this base block in which we specify that we do need that aws provider and then each of these blocks i've copied and pasted out the definitions of our specific resources into the corresponding dot tf file you'll also see we don't have a tf vars file here we'll define the tfvrs file where we actually consume this module and so i've got the web app module here in this web app directory i've got our main.tf file and i'm going to show how we can consume our module and deploy two different copies of it very easily from within this one configuration so i'm setting up our back end and required providers as usual i'm then setting up a couple of variables that we're going to want to pass into our configuration let's say we want two different passwords one for database one and one for database two these are of type string and are sensitive we can then reference the module using the relative path since these are both in the same file system so i'm just going up a directory into webapp module and then i define all the input variables that that module consumes so as whereas before we had a tf vars file that we were passing in at runtime now this module has those variables defined and when i consume it i define all the different values here except i i once again want to keep my password which is a sensitive value out of my code base so i pass that in as a variable to as a variable to this configuration and then that gets passed through to the module that we're provisioning i can then have a separate block where i define another copy of the web app where everything is quite similar except i've incremented this from one to two and so just by having these two blocks i can consume that module in two places and have a slightly different configuration so here maybe we're deploying devopsdeploy.com and then here we could deploy another devopsdeploy.com i'll go ahead and actually apply this and we'll see two copies of our web application being provisioned into the account make sure we're initialized and then i'll apply because i didn't pass that db password db pass 1 or db pass 2 on the command line it actually is going to ask me for them here in the prompt so i'll just enter some values and that should be enough for now to go off and provision two copies of my web app i'll speed things up while that's creating as you can see instead of two instances we now have four instances provisioning so everything that we are provisioning from before in our web app is now gonna be doubled we'll have two databases two s3 buckets four ec2 instances two load balancers and all the corresponding configuration that goes along with them once again it's the load balancer and the database instances that are taking the longest to create uh okay it looks like all those resources were added the apply is complete and so we have two copies of our web app now uh running across four ec2 instances in aws so hopefully that gives you an idea of now how we can breaking up our terraform code into the relevant section so it's just easier to follow and easier to read parameterizing it with variables so that we can pass in uh the key information to differentiate between different environments abstract a lot of complexity within that module so that as a consumer of the module we just pick the key values that we want pass those in and we can provision one or more copies of that module very easily now i'm just going to destroy and we can continue on now that we've seen how we can take our example configuration and modularize it and deploy it using that module as well as to consume a a complex third-party module like that console deployment you should have a pretty good idea of how to build a configuration using terraform now i want to move into how we use terraform to manage multiple environments so if you think about having your web application like this example that we've been building it's built with these different components and that's great and we've defined those as a terraform configuration that we can deploy but now in addition to a production environment we also want a staging environment we want to have them be very similar so that we can be confident as we make changes we can test them out in staging and see how it goes we also may want a development environment that we're deploying to all the time rapidly changing testing things and so we want to take our single config or module and deploy it multiple times and there's two main approaches that people use when doing this sort of thing the first of which is a concept called workspaces and so this is how you can use multiple named sections within a single remote back end so if we've got our s3 back end or our terraform cloud backend we can use the terraform workspaces command to create and manage at these different environments or workspaces that live as different files within our different state files within our backend and so we could say switch the dev workspace deploy that switch the staging workspace deploy that and so that is one method the other method is to break things out as different subdirectories within your file system and so here we see on the right we can have our modules directory which has the different modules that we built and then we can have a dev staging and production subdirectory which consume those modules in different ways and these two approaches have pros and cons that i'll talk through here in a second now terraform workspaces are good in some ways and bad in others they are easy to get started with it's very convenient because within your terraform files you can reference terraform.workspace as an expression to let's say populate the name of your resource so you could call your database the staging database or the production database and so it can it can make it very easy to handle that it also minimizes the amount of code duplication you have between your different environments so you have one copy and you're deploying it multiple times through these different portions of your remote back end the downside to workspaces is that if you are interacting with these things manually it can be very easy to forget which workspace you happen to have configured and make a change and apply it to the wrong environment so let's say you made some change you thought you were in staging but you were actually in production and you hit the button it causes an issue and that's that's no good so if you are manually interacting with this thing workspaces can be dangerous from that perspective if you've automated a lot of this and it's all happening from within a pipeline that can be less of an issue but it's still something to think about another con is that the state files are all stored within that same remote back end and so permissioning and access to those different environments you can't really isolate them so if someone has access to the development space someone then also generally has access to the production space within the cloud offering that terraform hashicorp provides there is some some more nuanced uh configuration there but if you're using a self-managed backend this can be a challenge also just by looking at the code base you can't tell specifically what's deployed everywhere you can tell we have this configuration but until you've done until you've issued some terraform commands you can't tell oh here are all the places this is deployed and here is those are those specific configurations so it can be a challenge just looking at the code base to reason about specifically what's deployed everywhere now the alternative option that i described is to put these things into different subdirectories within your file system this is nice for a few reasons one you can isolate the back ends you can have a separate back-end configuration for production from staging from development uh which is good from a security perspective you can handle the permissions for those back ends differently it also i think decreases the potential for human error of thinking you're operating in one workspace while you're actually operating in another and then finally looking at the code base it fully represents the deployed state so you can see here's my production config here's the variables i'm using and here's the staging config and here's the different variables i'm using and you can look at it very clearly see we've got these environments deployed and here is here are their configurations some of the cons here are that because now these are living at different places in the file system if we want to apply to each of them we've got to navigate between those subdirectories and issue those multiple terraform apply commands also we have more code duplication than in the workspaces style because each of these main.tfs represents a very similar configuration and therefore that code duplication it can be hard to keep things in sync across them and so that could be considered a downside there now depending how complex our infrastructure is we probably want to start separating things out into not just having a single massive terraform config for all of our infrastructure if you're a small company with with just one or a few applications then maybe having everything in one terraform config is fine but as your organization starts to grow and your infrastructure becomes more complex you probably want to break things out into logical component groups rather than have everything bundled into one section so let's say here let's say we want to break our compute away from our networking if one changes more frequently than the other that can be a good way to think about how to break things up but on the right hand side we've shown how our file structure continues to grow and get more complex as we break things out there's also the ability for us to reference state from a module or a configuration which is completely separate from our current configuration so let's say i had deployed my i keep using this example because it's a good one i had deployed my infrastructure my compute infrastructure so i've got some ec2 instances that are deployed with one configuration and i need to go grab the ip addresses for those i can actually reference that remote back end from another config entirely so that i can pull those var those variables across the two and keep them in sync without them necessarily needing to be in the same project so that terraform remote state is a powerful concept to think about when you want to break things out into individual isolated configurations while still being able to reference pieces of the infrastructure that live elsewhere i talked about pros and cons of each of those approaches and why we would want to use one over the other there's also some sort of meta tooling that can be applied on top of terraform that can make our lives easier and really help to manage some of the complexity that comes with breaking things out into that file structure and keeping our configurations dry don't repeat yourself so we can avoid repeated code avoid having to issue lots of commands across multiple terraform configs there's this tool from grunt works called terra grunt that helps us to do that type of thing so if you start to find that that complexity is growing and you want to wrangle it definitely look into this tool terra grant to help automate and simplify the operations associated with building out multiple environment configs like that it also helps us if we want to use multiple cloud accounts so if we need to be switching our aws account to isolate a staging environment from a production environment that can be a little clunky using a tool like terra grunt can help immensely in simplifying that for this portion of the demo i've gone ahead and implemented the two approaches i've described for managing multiple environments both using the terraform workspaces feature as well as using the directory structure within our project to manage those different environments i'll walk through the changes that i had to make and some of the trade-offs and show you how those approaches work so first let's go through the workspaces approach within the o7 module here within the repo i've got a workspaces directory with a main.tf file this is very similar to what we did before in in the previous portion of the course where we were deploying from the module containing our web application as always we set up our back end with s3 and the required providers we then set up a variable so that we can pass in that sensitive value for the password then we can also take advantage of the fact that this workspace has a name and we can use that to do things like postfix certain resources to avoid naming conflicts so here we're creating an s3 bucket and those are globally scoped so you need to have different names for the two so i'm just appending the environment name to the end of it and this is exactly like we saw before we need to populate all the variables that our module takes in as an input so that it knows how to go off and provision that environment another thing that i'm doing here is using a conditional so because the dns zone is going to be global across the two environments i'm saying if i'm in production go ahead and provision that zone if i'm not in production don't use it and instead if it's staging or development i'll use the dns zone that already exists i can also use my environment name within the database name to avoid a naming conflict there and that is essentially all we need to define an environment then we can use the terraform workspace command terraform i'll just make sure we're initialized and we can use the terraform workspace command use list and it'll tell me okay you're currently operating in the default environment i want to create a new environment called production and so to do that i'll do terraform workspace new production if i do list again now i'm in that production environment that i just created and i can do terraform apply because i didn't specify a tfrs file for that database password i'm just going to pass it in at runtime again if we're actually doing this in production we would want to automate this process and store that secret value in a dedicated secret store okay it's going to add the 17 resources just like it always does i'll go ahead and speed things up while that completes also while that's working i'll point out the other side of this conditional and that is that within the module that we defined in the previous part of the course in the dns portion i'm actually here i have a resource for the zone as well as a data block for the zone and that allows us to uh depending on this input variable of whether the boolean of whether or not we want to create dns zone if yes like we had there in production it will go off and create that with our domain if no instead it will assume that it's already created and look it up and use that as a data source and that's what this zero slash one on the count means if it's a one we will actually create this resource if it's a zero we will skip it here if it's a zero we'll skip the data object and if it's a one we'll look it up and so that allows us to toggle for that global zone that's shared across the two environments whether or not we want to use the resource object or the data object and because for our staging environment we're going to use a subdomain whereas production we're not here in my locals i'm setting based on which environment i'm in if it's production i set this subdomain to an empty string if it is any other environment i use that environment name to set up the subdomain value that gets put into this record so here we're taking advantage of some of those additional language features including conditionals setting this count value so that we do or do not actually use those resources etc now i just updated those name servers again if i were using this setup in in a true production environment i would want to automate the setting of those name servers and there is a provider for google domain so i could either continue to host the dns within route 53 and update these name servers automatically or i could just use terraform to set a records directly on my google domains account i didn't want to go through the setup of an additional provider here but just know that that is how you would handle it if you wanted to fully automate setting up that process okay it looks like our production environment has finished deploying now let's go ahead and create a staging environment so i can do terraform workspace new staging if i list them out it'll show me that i have the default staging and production and i'm currently in staging so if i do terraform apply i'm now going to provision another copy of my web app but now the environment name is staging and those conditionals will be set so it'll deploy to staging.devopsdeploy.com and set all the necessary values accordingly so here we've got our web up at here we've got our web app at devopsdeploy.com as usual as this comes online we should get an additional site that we can access at staging.devopsdeployed.com okay it looks like that record is now created so if i go into the interface here we can see in addition to our devopsdeploy.com we now also have staging.devopsdeploy.com and so if i hit that url staging.devopsdeploy.com it could take a second or two to propagate and now it has propagated and so there we go we've got two copies of our app one running at devopsepplay.com another at staging.devopsdeploy.com and these are two isolated environments that we've provisioned from the same code base using the terraform workspace feature i'll destroy both of these and then show you how we can use the file structure to achieve a same to achieve a very similar result all right so i'll do terraform destroy now with both environments destroyed i'll show you what it looks like to use a directory layout within our file system to organize the different environments so within this file structure subdirectory let me navigate there we have three subdirectories we have a global a production and a staging subdirectory global is for anything that is shared across the multiple environments so in this case it's going to contain a reference to our route 53 zone specifically because that's used for both of these things and so it's got our normal terraform block up at the top and then we're just deploying this route 53 zone within that global subdirectory then within our production and staging subdirectories we have a main.tf file that defines the all the options that we need for that module that we're deploying so it looks very similar to before the one difference here is that now for both production and staging i'm setting create dns zone to false because that's going to be handled outside of these two outside of these two within that global subdirectory so the first step here for deploying this would be going into the global subdirectory and deploying that zone and then we could deploy either production or staging from that point so i'll go into global we initialize here then we'll do terraform apply this is only creating that zone because that's the only resource that's shared across the two different environments okay that's completed so now i can navigate into the production folder for example again terraform in it to make sure we're initialized and then a terraform apply here would apply that resource i could then go into staging and do the same thing you get the idea so why would this approach be potentially beneficial over the workspaces approach one i can very clearly look at my directory file structure and see okay here's everything that's deployed i have a global group i have a production group and i have a staging group i could store my tfrs files within there and very easily see what environments i have deployed and how my configuration maps onto those with the workspaces approach all we have is that main.tf and until we actually have terraform installed and start issuing terraform workspace list etc we won't know what is is deployed so it's much easier to look at the code base and reason about the actual infrastructure we have deployed using this file system based organizational approach one downside is that we do have a little bit more code repetition right instead of just that single main.tf we have a main.tf in each of our environments there's a little bit of duplication there we also are specifying uh when setting up the backend provider we can't use some of the terraform variables features so we can't actually template in things within this backend so we do end up having things like our bucket and our dynamodb table hard coded in these configurations so that's definitely a bit of a downside as i mentioned there is some tooling such as terra grunt that can help automate some of this stuff but i just wanted to showcase the two different options i personally prefer this file system approach especially if you layer on some of that additional automation but either can be a good approach and if you're using terraform cloud they do a nice job of highlighting your different workspaces within their ui and so that can be a good way to get started with doing multiple environments and as long as you're automating most of this workflow and ensuring that you have protections against manually executing a terraform apply or destroy in the wrong workspace it is still a very viable option for managing multiple environments so hopefully that gives you an idea of the different trade-offs of how to go about doing this and you can make that decision within your own project of how you would like to manage this sort of thing with an understanding of how we should organize our code and manage different environments with terraform i want to talk about a concept that is fairly new to the infrastructure world and that is how we can use testing like we would use with software development to ensure that our infrastructure's code configurations are high quality and continuing to perform how we want them to in order to know why this is useful you need to think about the concept of code rot so code rot in general refers to this concept that over time things change about your software systems and if you don't test and use code it will oftentimes degrade over time that could be due to external dependencies changing that could be due to other changes in your code base that are impacting your specific this specific function in particular with terraform and with infrastructure the types of rot that we might see are out of band changes so if i deploy something with terraform and then my colleague goes in and changes something via the ui that is now a misconfiguration that could be a challenge we could have unpinned version so if we forgot to specify a specific version of our provider and it just used the latest one that could cause a conflict if that provider was upgraded in the background let's another example could be if we're depending on an external module or a specific resource type within the cloud provider and that was then deprecated that would be another instance of code rot uh and then the final one that i'll bring up here is if we have made a change to our infrastructure config our terraform config but that never got applied to a specific environment let's say we we rolled it out to staging but we forgot because we didn't automate it how to we forgot to actually apply that to production so that unapplied change now is a conflict between our config file and our state file and so how do we prevent these types of code rot and the way that we prevent that is by testing our code there's a number of different types of tests that we can do there are static checks so we can scan our code base with a variety of different tools there's some that are built into our into the terraform binary itself we can run the terraform format command and that gives us an opinionated formatting of how we should indent and structure the code base making sure that everyone adheres to the same style there's a terraform validate which does a check to see are my configurations using all setting all of the required input variables and that sort of thing or am i passing a name am i passing a boolean to a number variable etc so that's kind of what terraform validate will check terraform plan will go off and tell us if uh if our what needs to happen to get from our desired config to an eventual state of those resources being deployed and so that can be a great way to check if something has changed out of band is if i run a terraform plan command and it says zero change is required that means we're good to go our config has not been modified if i run a terraform plan and it says we need to issue these for commit we need to change these four things that means something happened if unless i changed my config and i wanted those changes if i haven't changed my config that means something happened out of band so often a good check is to just run a terraform plan command once a day or once a week and if it says that there's changes needed but there's been no change to the config that that indicates that something could be wrong and then as we talked about in a previous section you can define custom validation rules around the types of infrastructure that you're provisioning so that can be a powerful way to use the terraform tool itself to do some validation on your configurations there's also some third party tools that we can use to help do some additional checks against our code base this tool called tflint uh there's also some scanning tools called chekov or tf security or terascan which are focused on sort of the security aspects of your terraform config and then the cloud the managed cloud offering offers a tool called terraform sentinel which is enterprise only so you have to have the highest paid tier to use it but it can help you to validate some security configurations and enforce some rules on your code base that that you can get if you want to use that which would be great from a security and compliance perspective if you need that sort of level of guarantees about the configurations that you're you're managing you can always as you might expect do manual testing of things this would just be following that similar life cycle of commands that we talked about many times throughout the course where you're running an init you're running an apply a plan apply and destroy so this can give you a sense of hey does this configuration actually produce a working set of infrastructure and that's great but we would much prefer for this type of thing to be automated so we can take that type of manual testing and we can just automate all those steps with a shell script or whatever other technique we want and so for example we could take our example run this script in ci where we're starting off we're we're going into the directory we're initializing we're applying it we're waiting some amount of time and then we are issuing a curl request against our ip address so we've we've gotten our ip out as an output there on line 17 issue a api request against our endpoint and if we get back what we want that means that we have provisioned our our server and it is running and if it succeeds then we could go ahead and destroy those so this is a way it's kind of a hacky way but just kind of a the most bare bones example of how you could automate that manual testing cycle and turn it into a simple automated test we probably don't want to just have a hacky shell script as our end-all be-all and so there are tools that allow us to define tests uh within actual programming languages to test our infrastructure and make more complex assertions about what we expect to happen so i've taken that same test that we had before and now implemented it in go using a tool called terratest and so we import this testing package we go through and it this is essentially the same as that shell script we uh initialize our configuration we apply it it allows us to specify oh we're gonna try 30 times if we fail we will give a five second wait and then we'll try until we get a success or if we don't get a success we'll fail the test and we use that deferred terraform destroy so that at the end of the test regardless of whether we succeeded or failed we will hit uh hit that destroy command and that will allow us to clean up the infrastructure and avoid any uh resources that will remain and cause us to have additional charges because of that so this is just showcasing how we can use actual programming languages like go to test our infrastructure's code configurations which is a very powerful thing i want to cover a few things within the demo portion of this section first i'll just show kind of the repo layout here for how i would suggest laying out your repo i have four directories at the top level one for modules so this is where we would actually define any of the modules that we are building in this case i have a hello world module that i'll be showcasing it's a little simpler than our web app that we've been using and i'll just be using it as a way for us to test to show how to use some of these different testing techniques so within this module it is just a aws instance with our standard web server within the user data block as well as a security group that will allow inbound traffic on port 8080 so that is the module that we will be testing within the examples directory is where i would suggest having a working example of consuming that module so that anyone coming into this repo can see oh here are the different uh variables that i need to set and how i would actually use this so this looks similar to before we've got our our back end and our providers defined we are then referencing our module since it's in the same file system with this relative path we're going to consume two outputs from that module we're going to take the ip address of the instance and then we're actually going to use that ip address to build a url that we're going to hit to test that it did indeed come online and is serving that web server and we've got the network configuration set up within the deployed directory is where you would actually put your different environments so if you had a production environment staging environment like we showed in the last portion of the course this is where you actually deploy those things i don't have anything defined in here these are just empty files as a placeholder but those are kind of the three directories that we've seen so far and then the fourth one here is this test directory and so i'm going to show a few different types of testing the first of which is just sort of static checks so these are not quite tests but they're still useful to perform analysis against the code base so if we open the preview here there's a few different things we can do one we can use the terraform format command and there's kind of an opinionated formatting structure built into the terraform binary that allows us to see if it is formatted as we would want so if we wanted to actually do that we can go into let's say our modules directory into hello world and do terraform format looks like it didn't actually perform anything or i can do terraform format check and it didn't output anything if i were to go and change something about that module so let's say i had my formatting a little off actually it auto-corrects so let me save without formatting and so now it is still correct syntactically but it has this ugly indentation if i do a terraform format check it will tell me oh instance.tf is incorrect and should be changed if i do terraform format it will go off and make that change on my behalf so this can be a good check to run within your continuous integration to make sure that everyone on your team is following the the style guide that is provided from terraform in terms of formatting your code we also can issue things like the terraform validate uh which will check if it is a valid config terraform plan as we've seen gives us a a listing of what changes would be made if we want to apply we talked a little bit about custom validation rules in the language features portion but here's an example of what you could do you if you had a variable that you wanted to make sure was very short in the variable block you can include certain conditions and so then when you do your terraform plan command it will actually evaluate the variable that you're inputting and if it is too long it would give this value so another example of how you might use this is for a password you might enforce that it has to be of at least a certain length there's also a number of third-party tools that you can use to help scan your terraform configurations from a correctness security and compliance perspective so definitely look into some of those tools now i talked about how you can automate some of the process that you would normally use with a bash script and so that is what this test would look like here we're just going to navigate to the examples directory initialize and apply and this auto approve is how you avoid having that prompt asking if you do indeed want to apply and then while that instance starts up i want to wait at least a minute for that to happen because that's how long it would take this is a little bit hacky and we would actually want to use a provisioner within our terraform config could make this more robust so that we're not just sleeping in arbitrary length but actually waiting on that instance to come up and then i'm going to take the output from my module where i'm actually getting at the ip address and issue a curl request to that ip and make sure that it returns a successful a6 a successful http code when making that request finally if it succeeds i destroy the resources so i'd end up paying for those test resources so let's go ahead and run that so with if i'm within my test bash directory i'll do hello world test and it's going to go through these steps initialize that virtual machine test that it came online and i can get a successful response back on port 8080 and then destroy it on my behalf okay our terraform apply step here on line nine has completed uh we got back our instance ip as well as this url it's now sleeping for a minute simply because that web server is not started up immediately and we want to give a little bit of time for that to be live so it's still not online yet and it looks like on the right hand side that watch command was issuing it every every couple of seconds it is now successfully returning so when we make that request on the left there we got it and because it was a successful request that's going to go ahead and proceed if that request had failed the bash script would have exited with a code other than zero indicating that it was a failed test and so if we were running that within a ci system that would indicate a failed ci pipeline and we would get a notification accordingly now while that's running i can start to describe an approved approach while it's fine to use a bash script like that there are tools that allow us to use real coding languages such as go to define our tests so here within the teratest folder i've defined an equivalent test using the teratest package from gruntwork to define exactly the same thing except it's a lot more robust so we're no longer we're just sleeping for 30 seconds in west and resting here we can actually specify how many times we want to retry how long we want to wait between retries etc and so now that that bash test is complete i'll do go mod download to make sure i have all the dependencies that we are using here that'll go off and fetch all of those and now i can do go test dash v i'll set the timeout to 10 minutes and that is going to run our test script so it's going to find all of the tests within that directory and is using that teratest package to provision our system again we're essentially doing a terraform apply on that example behind the scenes then once it has come online we will issue a number of requests looking for the output from that on the url and that once it returns successful will cause the test to succeed if we don't get any successful responses from that server then we would give up eventually and fail the test and so by using teratest we have many more powerful primitives for defining the nuanced types of tests that we might want to perform for our infrastructure and it allows us to take advantage of the normal golang tooling for actually testing our code we can see now it's making those attempted requests there the server is not online yet and it's not accessible so we're failing but we set the retry high enough in order to get a successful request and then pass the test here it looks like we got out our hello world log and so at that point we got our 200 response status in that request and we went off and destroyed the infrastructure behind the scenes now another powerful test that i like to add to any terraform project is to periodically execute a terraform plan command within your ci cd system and what that's going to do is if there have been any changes either via the cli or via the ui outside of what terraform knows about you can set it up so that it will fail the test and that will notify you so that you can go check hey what's different about my deployed infrastructure from my infrastructure's code configuration and if it was by accident i want to revert those changes if it was on purpose i want to bring those changes back into the configuration okay our test is completed we've destroyed that infrastructure and hopefully that gives you an idea of how to start using testing principles along with your infrastructure as code configurations to make your infrastructure deployments much more reliable now at this point we understand how terraform works how to use the hashicorp configuration language how we should be organizing and structuring our code with these modules how to manage multiple environments how to test our code and now this final portion i want to kind of bring it all together and help you to understand what different workflows would look like both from a developer perspective as well as from automating the operations of using a tool like terraform to ensure that we have reliable infrastructure and can update it accordingly the general workflow that a developer who is using terraform is going to go through is you write and update that code you run those changes locally maybe you have a development environment that you can change without having any issues once you're satisfied that your config matches what you want it to you would then create a pull request so because we're storing all of our configurations in a version control system like github for example we can create this pull request so that one of your colleagues can review those changes issue comments maybe make suggestions for improvements that we would want to kick off a test run from within our continuous integration system so if we're using github as our example that might be github actions so that could run maybe that tera test that we had that i had shown before spin up a copy of the infrastructure make sure things are still working as they are expected to if they are give us a green check mark and tear that infrastructure down and that gives us confidence that when we do end up deploying this to production we're not going to run into any issues if we merge that pull request to the main branch we could have an automated pipeline within github actions again or whatever continuous integration continuous delivery pipelining tool of choice and deploy those changes to staging automatically so rather than have a developer issue a terraform apply command locally on their laptop we want those things to be automated so that we can avoid the potential for human error and so we deploy those changes to staging and then maybe on the next release so let's say we tag a release within uh github that could kick off a a separate pipeline which now takes those same changes that were were made on main and deploys them to our production branch so this is kind of the general workflow that i would recommend i would also recommend having a testing schedule on a cron on a cross street so periodically running just a terraform plan from within your continuous integration tool and if that plan shows any changes from the deployed state to the current the the current config on your main branch to raise an error and so that can be an easy check to see if something was changed out of band and find that very quickly and make sure that that gets reverted or at least integrated back into the config if it was a purposeful change so that's what the general workflow looks like there's also an important consideration here of working with multiple accounts within aws or the the terminology is different from cloud to cloud and gcp they're called projects in aws they're called accounts oftentimes it is beneficial from a security perspective to have one account for staging one account for production one account for development etc having these resources deployed into different accounts makes it much easier to manage the level of granularity for access that you need to within im policies to enforce the controls for the different environments both from the infrastructure that's deployed as well as for these terraform backends we want to isolate the environments to protect against potential issues so if you make a mistake and you blow up the development environment that doesn't impact staging and production so if we can isolate at the account level it makes it much stronger guarantees that making a mistake in one will not make a mistake in the other it also helps us to avoid naming conflicts with things so if we're deploying everything into one account you oftentimes cannot have the same name for an individual resource within that account uh and so you end up having to add all these prefixes or post fixes so maybe you say database dash production database dash staging those changes now need to be templated across any place that's used and that can just be annoying to deal with if you're working with a multi-account setup you can just name the database whatever the application name is and that can be identical across those different accounts and that allows it to you it allows you to avoid having to template that in as many places which can be nice it does add some complexity to your terraform config but in general i think it's still worth it to think about multi-account or multi-project setups wherever possible and as you need to start tightening up your security this is going to be an important way to to go about doing that there's a great talk on the hashicorp website from kobus bernard about how to use multiple accounts with aws and terraform and so i would suggest using that if you want to do a deep dive on how this would look and how you can go about implementing this within your own project i just wanted to call it out here as something that you you likely want to think about as you move forward and continue to grow with terraform i also want to call out a couple of third-party tools uh that are from a company called grunt work that make working with terra form much nicer i talked about teragrant a little bit in a previous section but it is a tool that helps you minimize the repetition of code throughout your code base it also helps with what i was just talking about in terms of dealing with multiple accounts and reducing the tdm of switching between aws credentials et cetera and so it can it can make a lot of that smoother there's also a tool called cloud nuke and it essentially allows you to very easily clean up a bunch of resources so one worry that a lot of people have with working in the cloud is that you can spin up resources and then forget to tear them down and then get hit with a big nasty bill the next month when you get that billing email and it's thousands of dollars and so one method to avoid that is uh let's say you isolate all of your testing to a specific aws account where you don't have anything else except for testing of terraform you could then use this cloud nuke tool to very easily clean up everything within it and it can go through and and wipe out all the resources uh and so this can be a sort of a backstop if people are forgetting to tear down their test configs that they they provisioned man that they provisioned manually you could have this periodically go clean those up i'm also a big fan of just local bash scripts and make files to store off commands that you need to craft that have lots of options and arguments and so putting them into a a repeatable scriptable fashion just helps to prevent human error in issuing those commands so using make files or shell scripts can help to to avoid human error in those situations now i mentioned earlier github actions as kind of the example but there's many different ci cd tools that can be used including circleci gitlab atlantis is kind of a terraform specific one but these are all great and you should take a look at if your organization is using one of them how you can use that for for integrating your infrastructure as code deployment and testing as well now i want to call out some potential gotchas with terraform that can lead you to have a bad day and these are sometimes non-intuitive so the first one is if you change the name of a resource within your terraform config it can lead terraform to think oh they want to delete this resource and create a new one and so sometimes that might be what you want but sometimes it might not be and so you want to be careful when changing names even if it seems like oh i'm just improving the how descriptive this variable name is i want to warn you about that so you could end up deleting an old resource and create a new one so make sure you think about that i talked about this early on but your terraform state files do have sensitive data within them uh so be careful in making sure to encrypt and manage permissions accordingly cloud timeouts so by this i mean you can issue a command and sometimes that command will take a long time for the server to provision or for the database to be provisioned etc and so if there are things that take a long time behind the scenes terraform sometimes has timeouts where it will provision half your infrastructure and then the other thing was taking too long and it gives up so you can configure those timeouts accordingly but just something to think about usually if you just reissue the terraform apply command it will fix that but they can be they can be a little challenging uh naming conflicts so if you're provisioning two things two resources within the same aws account oftentimes they can if they have the same name the second one will fail to create and so you need to think about that as you're provisioning your resources and make sure that they are named accordingly another one is forgetting to destroy your test infrastructures i talked about that cloud nuke tool as a potential solution for this but if you provision stuff and then never run the destroy command it is just running there and and costing you money so making sure to always run that terraform destroy command or clean up somehow periodically is important to avoid surprise bills terraform also has unidirectional version upgrades and by that i mean if i run terraform version 1.0.0 to provision my infrastructure and then my colleague runs terraform 1.1.0 i can now no longer issue a command with my older version of terraform because the state file is associated with the version of the of the terraform binary that was used with it so i would then need to upgrade and so if you have a large team you want to make sure everyone is using the same version of terraform on their local system as well as matching that version in your ci cd system there's also just many ways to accomplish the same configuration so that's not necessarily a con but it's just as you're thinking about things there's always multiple ways to do it and so thinking through hey what would be the cleanest representation of this infrastructure is important to think about there's also some parameters within a given resource that are immutable and by that i mean by that i mean if i provision some specific resource there are certain fields within that resource that i could change and there are certain fields that i cannot and so if there if i need to make a change associated with one of those immutable parameters i will need to delete the previous resource and provision a new one and so that's another consideration in terms of thinking about downtime and then the final one is something that's come up a few times i've mentioned is making changes out of the normal terraform sequence of events and that is something that you just want to avoid whenever possible your team needs to be bought into using terraform as the only means of deploying this infrastructure and you can reap the benefits but if you are partially bought in and deploying infrastructure with terraform but then making changes now your terraform config is no longer accurate and it's actually dangerous because if you apply it it will revert any change you had made out of band so these are just a few gotchas i think in general if you understand these and are thoughtful about how to approach your configuration you will reap many benefits by using terraform or another infrastructure as code tool but you want to think about these and avoid making changes in these ways as you go out and build your infrastructure for this final demo portion of the course i'm going to walk through how i've implemented that described workflow with all the automation within github actions and how that would work for the various different elements of that uh i've got pulled up here the github workflow yaml file uh with three different triggers for when this workflow is going to run the first of which is if we push to the main branch that's going to go through and actually deploy our staging environment if we issue a release so if we tag a new release with a semver version that will trigger the workflow as well and then finally on any pull request is when we're going to want to go through and run our teratest testing script to validate that the configuration is working as expected i've gone ahead and pre-populated these github secrets within the ui containing the values that terraform is going to need to access aws as well as populated a database password that's going to get populated in when we issue our terraform apply as well i'm just setting our working directory to be in the proper location for issuing our terraform commands uh this is a a step that is in almost every github action where you're checking out the code base i'm then using a public github action step from hashicorp that will set up terraform within the repo and use a specific version i'm going to run one of those static checks just to make sure that everything is formatted correctly so i have the terraform format check command there if something is not formatted correctly it will throw an error there and stop we then want to initialize as always if this is a pull request we're going to issue a terraform plan command and that's just a good check to see if the plan succeeds so would we be able to make an apply however if it doesn't succeed i'm allowing it to continue anyways now the reason i wanted to continue is that i'm using the following step within the workflow to pull the results of those of that terraform plan and display it within the action interface so that we'll be able to see the exact changes that would be applied from the ui after we've done that if we have a plan outcome as a failure then we want to fail the workflow so we're allowing it to proceed if it's failed at first but then showing it within the ui before we fail later that just makes it much more convenient to go see what happened i'm then setting up go so that i'll be able to execute teratest for any pull request we're then going to run the test just like we did in the last portion of the course next up i want to validate whether or not the tag that triggered the workflow matches a semver version or not so in this case if my reference is it v followed by a number dot another number dot another number and if it matches we'll set our production environment if instead we're on the main branch then we'll set our staging environment otherwise we'll set it to unknown in this step we actually go ahead and make those terraform apply changes in this step we apply the global portion of our config so that's that dns zone whether we're on the main branch or if it was a release then we can go apply our staging if above our validation check told us we are in staging otherwise we'll apply production because the production config lives in a different directory i'm also setting the working directory here and so this is all the config that's required to set up that fully automated workflow let me commit something and push it to the main branch so that we can apply both our global and our staging environment from github actions so now within our repository if we go up here to the actions tab we should see a new workflow kicked off because i pushed to the main branch let's follow along as that progresses we have our main terraform job here with all of these steps that i just walked through we successfully installed terraform we ran terraform format we initialized because this was not a pull request we skipped the terraform plan steps we're setting up go we skipped the teratest execution we're checking our tag in this case we evaluated this and it should have output staging because we're on the main branch we're now applying our global configuration to create that zone our dns zone was created successfully we can move on to the apply staging stage also moving the execution of these steps since they take quite a long time from your local system onto a remote system like github actions means that you can make these changes to your repo go off and work on other stuff and then come back and check on the results rather than needing to sit here and babysit and make sure that your terminal is executing as expected and worry about your computer going to sleep etc now while this is running i'm actually going to go issue a release within the github repo which should trigger another workflow that's going to do the production apply so i'll go back here to the repo over here on releases draft new release so i'm going to issue a new release i'm going to call it v dot one dot o dot o official course release and then we can choose a tag create new tag on publish looks great publish now within our actions tab this one is still going it's deploying our staging environment should be almost done but we got a new workflow corresponding to that release tag and it should look very similar except now within our check tag we're going to get production and then instead of applying staging we'll skip that step and apply production yep we skip staging and now it's going to go off and start provisioning our production environment now the one other piece that i described is the testing that happens when you make a pull request so let's go ahead and create a branch that we can create a pull request from so i'll do git check out b show test thing on pr that gives us a new branch now i'll just make an empty commit so i'll do git commit allow empty dash m show testing on prs hit push so i'll push that to github and now from within the github interface if we go up here it looks like our deployment to staging has succeeded our deployment to production is happening as we speak and we can go to actions and we need to issue a pull request on that new branch so we'll go from testing to main looks good create pull request and that should kick off the third type of workflow which is going to be that testing sequence and we can see on the pull request it shows us this workflow is in progress and if we want to see the details it'll take us into that job here because it's a pull request we are going to issue those plan commands then we're going to issue that github script that's going to push the push that information back to our pull request in the ui uh it shows us that terraform fi format and style was a success terraform initialization was a success and terraform plan was a success if we wanted to see the results of that plan we can go into the details here under the plan step and see the different resources that it was going to provision also because this is a pull request we're executing those that teratest test it looks like our example has provisioned it is now issuing those get requests to our web server and seeing if they succeed assuming that completes our check tag step will not show production or staging so it should skip all of the remaining steps and finish here shortly okay our tara test execution completed successfully uh our web app came up uh we were able to hit it with a http request and then it was destroyed so that's great uh we got to our check tag step it did not see staging or environment conditions and so we skipped all the terraform applies and then we finished our job so for one last time let's go confirm that our web app was indeed deployed to devops deployed as well as staging.devopsdeploy.com okay devops deployed is working and staging devops and now i don't have any automation set up to destroy this infrastructure so i'm going to go destroy it manually i'll do the same for the staging and the global environments once this completes also to avoid any small changes to the repo from causing that to be deployed over and over since i don't want to actually keep this infrastructure running i'm going to go ahead and within the workflow comment out the trigger on the main branch and then put a comment saying uncomment this portion to enable staging deploy from main go ahead and push that okay now that i have turned that off we should not get an additional action workflow based on that push and our code should reflect the latest great so if we did want to have this setup again uncomment that and that will allow us to have that staging environment deployed from the main branch but for now in case i need to make any updates the documentation and i don't want this infrastructure to stay up on my aws account forever that's what i'm going to leave the state of the code and so there we have it a fully automated infrastructure as code solution with multiple environments using github actions to actually do the deployments as well as to manage as well as execute some code quality checks and tests on our infrastructure's code configurations and with that we've reached the end of the course congratulations if you've been following along you should now be ready to go off and use terraform to encode your cloud infrastructure within your code base and help ensure reliable operations for you and your team if you got value from the course go ahead and hit the like button to let youtube know if you want to check out how terraform can be used with another cloud provider and a different application go watch this other video that i created in partnership with lenone that's it for today and remember just keep building [Music] foreign

Channel: DevOps Directive

Views: 369,629

Keywords: infrastructure as code, terraform tutorial, terraform tutorial for beginners, hashicorp terraform, devops terraform, terraform aws, terraform github actions, terraform explained, aws terraform, aws terraform tutorial, terraform tutorial aws, terraform aws tutorial, terraform devops

Id: 7xngnjfIlK4

Length: 158min 4sec (9484 seconds)

Published: Mon Feb 14 2022