Using Terraform with Azure

hey everyone welcome to this video all about using terraform with Azure and my goal for this video is to really kind of start from the ground up so super quickly kind of the whole imperative versus declarative and why we like declarative a quick 101 about terraform exactly what it is some of the key fundamentals about its installation understanding the importance of state when we use terraform and now make a lot more sense when we actually start going through a couple of demos I want to talk about the structure of terraform files using the actual asier provider for terraform creating resources in Azure through a terraform script how I can interact with Azure key bolt now terraform has its own vault technology but I'm gonna focus here on just interacting with secrets that I have in a jerky vault and then probably don't need to do this and we don't really want to do this very often but if I do need to call an armed template which is Ash's native declarative technology from terraform so super quick the whole imperative versus declarative now I have a whole video on this over on my youtube channel I go into huge amounts of detail about imperative and declarative and how I like to get to a pizza restaurant so if you want the details of that you can head over there but the key point is that when I do imperative I'm telling the system how to do something do this step then do this step hey check fer if this is already there this is how you handle this situation or this situation I have to handle all of the errors handle all of the possible current states of something which actually gets super complex to do it's also harder to compare what exists compared to some kind of decorative configuration so if I'm using imperative I'm saying do this do this do this a hundred different steps it's very hard for me to look at those hundred steps look at the current state and work out well is that matching where's with a declarative technology it's actually super easy to detect drift because we've declarative I just state what I want the desired end state to be I'm saying hey I want a storage account I want it to be locally redundant I want it in this region and I want to contain you in it called this that's the end state that should be there I'm not telling it how to create an account and not telling it to create a container it's its job to work out what's there today what's the current reality what am I being asked to do and it has to work out I ever create the resources maybe modify the resource if it's there but it's the wrong type of resiliency and everything in between so essentially with a declarative technology it's the job of that technology to just make it so you always think ERV kind of Star Trek and Picard make it so that's the job of a declarative technology I'm just saying I want it to look this way go do it so it's much easier with a declarative technology to have a new desired end state I could change my desired end state and just say make this so it's up to the technology to detect the differences it's up to the technology to detect if there's any drift and kind of fix it and when we think about Asia it loves declarative fundamentally Asia has all of its resources provisioned on the back end using some kind of declarative JSON technology so even if I imperatively use the REST API PowerShell AZ CLI the portal underneath everything as you is storing it as JSON it's doing it as a state so yes I can absolutely use PowerShell and CLI I can create things imperative Li but that's really not the ideal I'm making a lot of work for myself to go and check well how do I handle this error what if my script found was halfway through and something is already there how do I now handle a changing desired configuration of the resource I have to write all of that logic in my script so resources are best provisioned using templates now natively Asha uses JSON based on templates now JSON is great for computers it's really not great for humans to edit to troubleshoot it's also as a specific so if I was just using Azure then maybe sure I'm okay with using um but if I'm using other clouds if I'm using VMware on-prem if I'm using other technologies well now I have to do a different syntax for every single one of those with terraform I just have different providers but I'm using the same syntax to use across them super quickly I mean this is an arm template this is JSON you can see hey I've got some parameters that I'm passing in I've got some resources that I'm going to create I'm gonna create the storage account I'm gonna create a container you can see there's a lot of code there just to create a storage account and just to create a container it's also not super nice to read or even try and kind of create this from scratch and yes we are absolutely talking about infrastructure as code we are defining our infrastructure resources using machine readable definition files these are prescriptive I can put these in a source control system a version control system like get and track this through I can deploy this in a very prescriptive manner between test dev between to acceptance to prod without any risk of something being different so this is really the foundation of doing things around infrastructure as code so let's talk about terraform so it is a declarative technology created by Hoshi Corp it uses terraform text-based configuration files with ATF file type now because it is a text file it's easily integrated with source control systems like get I can easily see what's changed between them now there is also the ability to use JSON I can use a dot TF JSON but typically I'm gonna stick to the dot TF format which is both machine and human readable terraform is written in go as are its plugins you can use something else for the plugins but I think 99.9% are all written in go and a plug-in could be something like that provider to make it work with Azure or with AWS or Google Cloud or VMware because it's written in go it's very easy to make this available across platforms now what's interesting about terraform is in this day and age we're used to hey we install things it puts files all over the machine it does things to the registry terraform does not do that if I go to the terraform download page here I can see Hey look there's Mac OS Linux variants windows 32 and 64-bit if I was to download it what I get is a zip file and if I actually go and look in that zip file its terraformed or XE that is not an installation file that is terraform so what you do is you take that file and you copy it somewhere so in my case I copied it to a seeker on slash tools folder there's my terraformed or XE from there you simply add that to your half if it wasn't already for go to my advanced environment variable was my system path you can see I added my tools folder so now terraform is just available to anything running on this system and that's it I don't do anything else that is the installation of terraform as I mentioned there were then plugins so the plugins provide service specific integrations I already talked about whole bunch of them but there are many many others for example kubernetes that would be a super popular one for many organizations and there's a whole registry that I can go and look at and terraform themselves actually maintain a list of all the various providers so here if I look I can see hey look all of these have providers that are kind of official providers registered we have hashey Corp and what that actually means is if I use them I don't have to manually install it first when I go through the various terraform phases and I'll show this but when I do an init it will go and check Hey look what providers am i registering what plugins and we're actually go and pull them down and make them locally available so it's super easy to leverage those or I can manually go and install other providers additionally terraform is actually natively available in the azure cloud shell so if I just start up an azure cloud shell I can just type terraform it's there already I don't have to do anything which kind of gives you a hint of the close integration terraform has with Azure so before we actually go into details about the format's let's talk super quickly about kind of what are the key phases so we have to create terraform configurations now we create these in a workspace a workspace is basically a folder so I have a folder and in there I'm typically gonna have three key files I'm gonna have a main TF which is actually my declarative configuration I have a variables file and I have an output smile we like to separate the variables from the main because for example I may set default values and I may have different values for different environments I don't want to have to change the main TF I want a bush and control that and let that move between test dev and you ATM prod with no modification anything environment specific I can put in the variables file now there are other ways to override that I can pass parameters I can have an actual TF VARs file I can set environment variables there are other things I can do but at a base level I'm just gonna separate out my main configuration my variables and my outputs here in my environment we can see I've got a main TF super simple biller code here using the azure RM provider version 240 11 I've got a little data section I'm grabbing an existing resource the resource group and then I'm creating a storage account and creating a container no this is doing the same thing that that very large JSON template did gives you an idea that it's a lot easier to read and it's a lot shorter and then have a variables file hey look I'm gonna set the resiliency type and I can also have an output file if I jump over for a second we can see the outputs and there's my output what I'm doing is outputting the storage account name so outputs is super useful for conveying information maybe that I don't know before I actually deploy for example if I was creating a public IP address I could return it here because I wouldn't know it until I actually made the deployment so the first thing I do once I've got my configuration working is I'm gonna initialize terraform so that's the terraform init command now this is going to basically take that workspace that's got all my files and is now gonna pass through them and it's gonna kind of create an initial state and it's going look through my files and firstly look for what plugins am i referencing for example I'm using the Asha plugin it will automatically go and download those two ADOT terraform a subfolder I only run this once I don't have to run this every time I have a slight change to my configuration now if I added new providers if there was a new version I would run it again I can even do a dash upgrade to kind of force it to upgrade any even existing providers but I'm typically running this once in my workspace again unless I I kind of changed that provider configuration so what I'm gonna do now is run that terraform in it now notice there's a terraform fmt command this will basically take my files and if I had it poorly space they are etc it would actually fix that for me so it would fix all of the formatting for the files in my current folder so if I ran that so she's showing me hey look my outputs TF file it fixed it made some better formatting for me so I'm gonna run the terraform in it now if I look comment Liat my folder there is no dot terraform subfolder I have no stapes is the first time I'm actually running this additionally if I actually go and look at where I'm going to deploy to the resource group is currently completely empty there is nothing in this resource group so we're gonna run the terraform init command so it's checking and you can see it was looking ok looking for available plugins Hey look the azure RM plug-in has she called as your RM 2.11 if i now look at my file system we can see i have a dot terraform folder where it's port the azure RM provider so it's actually started to do that so that's done that initialization it's checking it and we're kind of ready to proceed so the next thing we want to do is the terraform plan now this is going to create an execute um plan ie is gonna work out what actions it actually needs to perform to take the current reality I in my current state and make it match the desire configuration in my terraform Falls now this is not gonna make any changes this is gonna show me what will be done but won't actually do anything so it's actually a good way to check hey what's gonna happen it's a good command to run for example before I'm gonna go and check in my code into source control now I can actually add a dash out to my command and I could then use that as part of an apply or when I run the apply it will just go and kind of work it out again so there's a potential of a difference but it's pretty unlikely now remember terraformed plan is comparing against the state file if there is one this is the first time I'm running it I don't have one yet but if the state file was wrong ie I made a change outside of terraform it's gonna skew what it thinks it's actions required ie its execution plan will not be correct and when I stress this later on but we start to get an idea that once I use terraform I only use terraformed provisional things I don't provision with terraform then go and create some other stuff then modify things in the portal then try and do something else in terraform it's not gonna work as I expect so I'm gonna run the plan so now it's gonna go and look let's say okay what do I need to do okay so it's actually complaining hey look I have not actually logged in yet so the first thing I have to do is actually log in with AZ so I'm gonna quickly do that so I'll do my easy login again this is a new machine I started everything fresh and blank so I could show you everything so it's saying hey look you don't have this installed so I'll install the AZ so here's the azure CLI so I can download and install this that's installed now we can actually jump back now it's installed I'll run the AZ login again so it'll make you pop up a browser and put in a code and we'll be good to go so here we go yeah pick my account and I'm logged in okay so now I have that AZ installed and I've done the login now no I logged in so I'm running this interactively as me the other thing you can do is actually create a service principle and as part of the provider configuration actually specify and identity to leverage this would also be a managed identity but if you actually look at the documentation for terraform and Azure it does walk through hey I could actually go and create a service principle I do show that up here in this file and this is that in my github repository so you could actually go and do the AZ add service principle crate for role based access control and scope it a subscription or resource group whatever you needed and then that output you can even add kind of an SDK office it formats it nicely I could use that as part of my Asha provider configuration but for this I'm just using my interactive account so now if I run plan it's because there's no state it's gonna go and generate that and then look at well what is my desired configuration in those files and what I'll see is will actually create some files in my folder when I kind of proceed with this okay so here we go it's saying what actually makes a little bit bigger quickly what are these green things are things it's going to create so it's going to create a storage account for these various parameters are note some of the things are known after applying if they're things like cared about I could have that as part of the outputs file so it actually show me and also it's gonna go ahead and create the container I want now at this point if we refresh there's still no state phone yet that's going to happen when we actually go and do an apply at that point we'll start tracking actually what's different but it's telling us hey here's what I'm gonna do so in the next phase is to actually kind of make that the reality so I can actually run a terraform apply and it will now take that desired state and apply it on to my actual Azure subscription it will prompt me for approval I can skip that if I want with the kind of - Otto - approve so that's what I'm gonna do here I'm gonna say - Auto approve remember currently if we go back to my resource group it is empty so let's go and hit that apply so once again because I didn't save the output of the plan that execution plan is gonna kind of go and work that out again is gonna create that storage account and it's gonna create that container so now I can see the storage account one is being created and then I'll see the container created as well and that's completed so here we can kind of scroll up so it finished Creighton the storage account then it created a container it added two things changed nothing destroyed nothing and then the output was my storage account name which is what I'd specified here hey show me the storage account name if I now look at my folder well now I have a TF state file now yes this is JSON so that makes you think technically you could edit it let's say there's some state drift really dump that's a far more advanced thing and there's actually better ways to handle that anyway now this is declarative now if I go back to my resource group so it takes a little while to catch up often 30 seconds to a minute so there's my storage account if I go and look at it I can see hillocks in that resource group its LRS so just focus on that for a second now I can look at the state file yes it's kind of JSON if we go over here I could take that state file and I could say open with code and we can see hey look there's all the resources so I can actually read it from here but I can also do terraform show if I do that it's actually gonna go through that state file and show it in a little bit of a more friendly format I can even list all of the resources it knows about in that state file and look at specific ones so hey my storage account and my container now what if this is where to Kyra TVs super powerful actually that storage account I created as LRS I want it as GRS now I could change the variables file but in this case I'm actually just gonna pass it as a command line parameter so I'm saying hey plan what you would do if the replication type was now GRS as part of that configuration so it's gonna look at the state file remember it's not looking at what's deployed is looking at the state file which is why it's really important that we kind of keep those things synchronized we don't go and make changes outside of terraform but it's gonna go and look and it's if we go through oh there's an update in place this yellow squiggly account replication type it's gonna go from LRS to GRS and also it's sent it's gonna remove a tag which I'm applying through policy so again I'm not applying that through Tara forms is gonna remove it but then it's gonna go straight back again so if I actually applied this now what I actually see is that account will actually transform into GRS I didn't have to write something to say hey if it currently exists if it's not matching the type then use these modify PowerShell command it's saying I'd have to do imperative my desired state is just different go and make it so terrible so I can see that is complete so it has made that change can see it's modifying the storage account so now if I actually go and look at this storage account notice is LR s currently now it is gr s so it made that configuration change there were some other kind of neat commands I can visually look at the state using terraform graph and output it to a dot file and that dot file like actually read into a graphical visualization utility like graph is and I've got the kind of link which could actually go and get that so it's kind of some nice thing as you can see now I mentioned the state the terraform core is responsible for a number of things one of the most important ones is that state management so terraform stores the state of those managed resources in its state file now it actually does create a backup so what I should have now is a TF state dot backup file so because I made that additional apply with the new type of resiliency it backed up the old State for me now this provides that mapping of resource reality to my desired reality so the configuration in the config files compared to what it believes is the reality it's not scanning Asia every time I do a plan or apply it's tracking it in that state file so it's super important I really don't mess with it outside of there now those state files also track any kind of metadata that could be kind of terraform specific now that terraform dot TF state should not be manually edited yes I showed you it's JSON and maybe more advanced people may be tempted to go and do that but it's really not the right thing to do resources should be managed by terraform and I should not change things outside of that so the best practice once I've started to engage in using terraform don't provision things outside of that don't change things outside of that now one of the challenges could be if you're working as a team I have ten different people working on the resources they all might be modifying the terraform configuration and applying well you need to ensure I'm using the latest version of the state file now what this means is a number of different things Eve or I take that state file and I make sure maybe it's checked in and other people check it out but sometimes I don't want to do that maybe they're sensitive day to in that state forward maybe don't want to put it in a source control system now they were backends by default with terraform I'm using a local back-end that's where I'm actually storing the state where I'm running these configurations I can also create remote backends so that would have a remote state I've used an add a storage account for example as a remote back-end so that could help with having a consistent state used across multiple people and our option would be I could have a jump Server and people always run the terraform from the jump servant the jump server has the state file the larger teams I'm probably gonna use terraform Enterprise that gives me better collaboration governance out-of-the-box pipeline integrations really stronger locking to help enforce and control that concurrency it's gonna mean sure the integrity of my state files so if I'm going beyond just cool of myself and I'm working as a team I probably want to look at a richer kind of option for that state and again be careful even when I have secrets or maybe they're stored or sensitive they may end up and likely will end up in that state folks you want to be careful what I do with a state file what if I do get state drift if the infrastructure changes outside of terraform we need to update the state file there's a terraform refresh command I can use that is also a terraform import for example if I had existing resources and I want to manage but just be careful with both of these these are not really something for a beginner so the important thing when I'm getting started is hey I'm using terraform don't use anything else everything I want to create I'm gonna create via terraform let's look at the files in a little bit more detail so the terraform code is this HCl this is the hashey called configuration language this is the declarative language used in those terraform files and it really is documentation as code I create these configuration files with documents the configuration as well it's fairly easy to read as I showed you before it's generally about 30% less code then if it was an arm template so there is less stuff it is easier to read now I can do comments there is expressions I can use they're a little bit more advanced but I have all of those kind of capabilities says look at the anatomy of a terraform file and the resource part of a file now resources use the same format no matter what the provider so doesn't matter if it's a show or AWS it doesn't change the format of a resource it's gonna be the resource keyword the type of the resource the name of the resource and then parameters so that could be hey it's an azure M storage container the container name in this case container 1 and then some parameters around it in my example file we can see hey I have two resources so my resource section I have my storage account that's the type of the resource I have the name of the storage account then a number of parameters for that then I have a second resource that is the container so as a complete file I'm going to have my provider section we have optional version information so in my file I have the provider keyword is using the azure RM provider and I want version 2.1 and i have to have this kind of features block it can be empty but i have to have it or it will fail then i'm gonna have a data source section this lets me reference existing resources in my case here i'm referencing the resource group that already exists and then I have my resource section where I define these things I then have variable files so this is where I can define variables that I want to be able to pass in and the goal of this is that it lets me separate anything that is environmental specific so again I have a variable file for test they have an UAT and production this is the variables TF file this let's meet version control the main configuration file I don't have to change it as it then moves between different environments or instances all I would have is different variable files these variables are declared and I can set default values if there is no default the user will be prompted and I can also set variables as part of the command line which I showed earlier I can have a terraform TF bars file I can set environment variables so have all these different things and in order of precedence a command line flag will take precedence then something I have in a terraformed TF files file then an environment variable then stuff that's in my default variables TF and then finally I'll be prompted so I will lose different layers then I have my outputs file these are messages and data I want to be displayed post the apply and again the example I gave is maybe something I don't know before the application and that will be created as part of that application for example hey you're creating a public IP here's what that public IP is that's gonna be my output dot TF file now the azure resource provider currently it is at the second major version now as I showed earlier and this is the reason I start with blank environments for my demo to make sure I remember to mention things you must have the AZ CLI installed and then one of the big things we did with the v2 is in the past I could accidentally import existing resources the things that already existed and then modify it with terraform now it will not let me do that it will force me to do a terraform import before I can actually modify them now one thing that often comes up is well hair up to date is this so there is a very close partnership between Microsoft and hashey Corp as it's multi-million dollar relationship mic stuffed has its own terraform dev team that helped manage this provider they work closely with hashey Corp so many features will actually be supported day one for the terraform Azure provider a lot of major as our customers use terraform as I mentioned already I have to import resources that already exist to be managed and this is an official plugin this means it is distributed by Hoshi Corp I don't have to manually install it now when I create our resources there are a number of quick starts actually available for me and just like arm templates my recommendation is go and look at these quick starts and find one close to what you want to do so here's the quick starts page we can see there's ones around Web Apps Linux containers aks I'm using helm of aks and static pages virtual machine scale sets aks Enterprise so you can go and look at these and notice it's got the main TF the variables TF and I can go and look at hey this is what it's doing there's the version just gives you a hint of kind of how to get started so going look at those templates find one that's pretty close to what you want to do for the more advanced authentication to get an idea of all of the types of resource that are available just go and look at the documentation so if we jump over to the azure provider it talks through hey look how I can authenticate using and managed identity using a service principle even actually give me examples of using all of those types of commands it has a list of all of the types of resources I can actually leverage and examples of their use so the documentation here is kind of super useful and I can also use environment variables to actually set some of these values and it kind of talks through some of these Hey look I can set and arm underscore subscription ID to set the subscription I want to use which cloud I'm using what's the tenant ID certain types of secret or certificate so you can actually pass things like have it as part of my profile for example to set these values it could be part of an application as part of a deployment pipeline which will then get used by terraform so you can take a second look at those things and they really are kind of super useful now what about interacting with key vault so as your key vault I can store keys and secret certificates and I use a secret typically when I have some sensitive data I don't want in a variable I don't want it in a terraform fault I don't want it in my state file now you have to make sure the account that terraform is running as be it myself be it as that managed identity be as a service principle has rights to secrets in the vault but then I just referenced the secret like anything else it's really not special so over here in this same I have a key bolt and I have my savile vault and I have some secrets one of them is this kind of simple password nothing fancy if we go and look at it we can actually say show the secret its password I think that great and what I have actually closed these vowels down for a second I have another sample file what I actually do is I'm going to change folder super quickly and in this you can see I'm using the azure RM provider again this time I'm looking for a variable called K V I'm not sending it to anything and what I'm going to do is actually show you using an environment variable to set that value then I'm using the data type to reference an existing Keeble secret ie sample password and I'm passing in the key for ID via this kv variable so it's dollar curly braces VAR kv because there's my variable kv then we use output to show you the value of that so really pretty simple so what i'm doing here is again i have to do an initialize I've not done this before but it's gonna go and pull in my Azure RM provider then I have to do a plan now that is it saying what I need a value I've not said it anywhere somewhere as you're gonna cancel out of that because what I forgot to do was set this environment variable so in PowerShell I'm just gonna do dollar M : now I'm putting TF underscore VAR underscore so I put that in front of it that tells terraform to come and look for these things then the name of the variable is just kv and i'm sending it to my ID of my key vault my sub idea I've already got set as a different environment variable so there I'm setting that now the plan should see that environment variable and use it as part of the plan now it's not creating any resources it's literally just gonna go and reference that existing bit of data and then output it but I would have reference it as anything else if I did have resources that I wanted to use that password maybe I'm creating a VM and that's local admin password on joining the domain I was just reference that dollar var kV so now if I run the apply it should my output spit out the secret I don't have to actually change the key volt to say they allow arm templates if I'm using a secret for an azure arm template I have to set the volk to allow arm template interaction I don't have to do that for the terraform it's just going in via the regular API so it's going it's checking the state and it should spit out that password really just show me how easy and there we go a secret from key volt there's my password so that's how I can interact with kibo again whatever security principle Telefilms running as would have to have rights via the policy on that key vote to the secrets if you have a useful components now obviously in asia i can use extensions i can use the run command custom script extensions lots of other things there's also a file provisioner there's part terraform that can be useful to copy a couple of files into a remote machine there's also a remote exact provisioner to trigger commands on a remote machine this is for maybe just doing a couple of things but again I could use the add a specific extensions to do this as well if it was more complex I probably use say like PowerShell DSC chef puppet ansible etc pipelines hopefully you're not just manually running these like I am so in the best case scenario I'm checking my code in somewhere I have pipelines to actually do provisioning of the infrastructure of my application there is a marketplace extension for Asia DevOps that adds terraformed tasks to my pipelines and there is also a Hoshi Corp set up terraform github action well that does is it actually adds terraform to the path of github actions send all of the regular commands I can trigger as part of a github actions pipeline so I can do the apply in it planned etc etc so it's very easy to integrate terraform with your pipelines finally if there was a scenario where hallo guy I have this feature I want to use maybe terraform provider doesn't support it yet or I just have a ton of arm templates I don't want to recreate them all yes I can call an arm template from terraform but it is discouraged remember how she caught worked very closely with Microsoft to ensure the Azure provider for terraform is up to date there shouldn't really be many things that I can't do natively with terraform because the doubt side is yes there is this Asha RM template deployment this lets me take an arm JSON and push it with terraform but terraform is not aware of the resources that are being pushed through that template if I was to destroy for example it wouldn't delete the resources it's just gonna delete the reference to the template so I really want to try and make sure I have kind of native terraform objects but if I'm absolutely stuck then yes technically I can call an arm fall from terraform as I mentioned the code that I showed here is in my random stuff github repository if you go and look in my terraform with Azure I've got those basic files and I've also got kind of referencing that secret and just think may be useful for you I've kind of got my gear ignore file as well because one of the things you probably don't want to do is there was sensitive information it's for example send your TF state into a repository I also don't want to upload the plugins so my get ignore file is kind of ignoring order the doc terraform content ignoring any files and if I created kind of a a plan I use plan out so I don't want to upload that either so that just may be useful if you want to go and look at the code kind of I've used here so I hope that was useful if it was please kind of like comment subscribe and until next time take care

Channel: John Savill's Technical Training

Views: 28,095

Rating: 4.9577041 out of 5

Keywords: terraform, declarative, azure, azure resources, hcl

Id: JKVkblsp3cM

Length: 45min 0sec (2700 seconds)

Published: Thu May 28 2020

Reddit Comments

[u/assemblyman]

Hi John, great stuff. I love your Pluralsight courses. I struggle with using Infrastructure as Code. I appreciate it's purpose, but haven't seen how it would ever help in the typical Azure deployments my company does(SMB MSP space). Maybe it's the painstaking process of looking up every variable, resource group ID, possible values for settings, etc. I find it easier to have a GUI with all options on the table, as I'm typically opening the GUI to find the name of the resource group, or valuee to copy into the .tf file anyway! Maybe Azure Intellisense would help(forgive my ignorance if this already exists).

[u/mdnbcfpr]

Personally I've never understood the point of using terraform over native ARM templates - it's a json abstraction on top of a json abstraction that's even more fragile due to the state file.

[u/NotMichaelReeb]

Looks great!!