Terraform And Azure DevOps - How To Configure

[Music] hello okay so let's get into uh how this terraforming azure devops works in my environment so first of all let's show you what it actually does uh to show you what the end result should look like at the end so if i go into my azure environment here as you can see i've got no resources at all deployed in here currently the only thing that is deployed um are some management groups and some policy definitions and as you can see they're all built out here and look very similar to enterprise scale um but we're not covering that today on this video um in the subscriptions you can see i've got three subscriptions um and if i go into all resources again you'll see that there's nothing deployed in here um if i then go into azure devops and you'll see i've got one project in here um enterprise scale terraform community um but if i then look into the repositories and my files you'll see that i've got a terraform folder in here and i've got all of my uh management groups defined in this module in this file here and that's what's deploying everything into my environment as you can see i've got a terraform plan build pipeline and i've also got a release pipeline here as well to actually deploy the infrastructure individual but instead of just showing you this and we'll dive into how this is created later let me show you first of all what it looks like from an end user experience if i want to deploy something into my azure environment so if i dive into vs code you'll see i've got the same repository cloned here locally on my laptop and let's just create a new terraform file inside this terraform directory so let's just call it a demo.tf and inside here let's just go and create a resource group so let me go and copy the code for a resource group in true blue peter fashion here's one i had earlier which removed some properties off of here that aren't going to be required you'll notice that i'm using the provider and the alias functionality inside of terraform here because i want to deploy this to a specific subscription my connectivity subscription and if you look inside my providers file i have this subscription hit defined here and with the alias of connectivity so it should appear in my connectivity subscription um cool all right let's just create a resource group here called rsg v1001 going to go into north europe so let's save that file now actually before we save that file really important thing to note is that we're currently in the master branch here on my repository and the way that i've got this configured is that you can't actually commit code directly to the master branch it has to go through a pull request process so what we first of all need to do is create a another branch put our code into there and then push that branch up to our azure devops project and inside the replay where this live so let's create a new branch and let's call this uh demo one and that's created that branch there and now as you can see down here vs code has changed into this branch but if you wanted to select into that you can click on it select demo one that you've just created now we can save that file into there and then if we could see over on the left hand side the git extension is now reporting there's one file to change and it's our demo.tf file so let's go and add that and stage the change and let's just say create demo resource group and if we click this button here that will do a git commit into our branch and then right down the very bottom now we can click on this little cloud icon with an upwards arrow that's going to push this branch up to the remote repository in azure devops so let's push that and now that's completed if we go back into azure devops and back into our files on repos you'll see that it says you've updated demo one branch just now and if we actually click branches here on the left you'll see that we've got a demo one branch here that's just been added just now by me so if that's what we need now need to do is create a pull request to put this code back into our master branch which will then kick off all of our pipelines and all those requests so as you can see we can click create pull request here and what we're saying here is that the demo one branch needs to be pulled into the master branch and this is the pull request template so we can fill in here so we can call this uh demo1 pr create resource group and we can put the same in the description but obviously you can fill this out in your environment whatever you want and as you can see here i don't need to specify any of this i've got some branch policies that we'll we'll dive into later that put all these options into place but if you want to add some reviewers whether they be required or optional you can add those in here and that will work just fine i can also link this to a work work item but i haven't got that configured in this environment and i'm also just going to add the tag that i'm using of demo just so i can keep track of these you can see the files that are going to be included so it's that demo tf file and it's the resource group that we said is going to be created and you can see the one commit that we put into that as well from vs code so if i now hit create that's going to create that pull request and as you can see a few things have happened here in this pull request so first of all you can see that it says jack tracy must approve even though i didn't add myself as a required approver my policy my branch policy automatically added me as a required reviewer and we'll dive into how that's configured later and you'll also see that there's a status check running so i've got another build pipeline that runs that effectively does a terraform validate and the terraform plan just to make sure that the code is all valid and syntactically correct and it'll also give us a bit of an overview of what it's going to deploy so that whoever's reviewing this pull request obviously wouldn't be me in an ideal world i wouldn't approve my own homework but they can review what's going on and what it's going to deploy and then choose whether to allow that or not so if we go into uh the required check succeeded here and we click view check and you'll see that this terraform status check build pipeline completed successfully so if i click into that let's go and have a little look at what that did so it kicked off just now it took 29 seconds and if i dive into the status check job here and as you can see it did a load of things so download some secrets from a key vault it checked out the repository installed terraform did a terraform on it and connected to the remote state which is also stored in an azure storage account um and then it did terraform validate and we got a success there which is all good and we also did a terraform plan as you can see all my management groups from modules and policies that are deployed but if we scroll right to the bottom we've got down here one to add and it is that resource group that we added earlier so it looks good you know it hasn't deployed this it's just terraform plan phase and as you can see the job stopped but that's reported back into our pipeline that's already into our pull request that everything's good and now the person who's reviewing that can make an informed decision as whether to approve that pull request or not so let's go back into our pull requests and as you can see it's sitting here still waiting and we can put a note in here that says yeah it looks good to me we can add that as a comment and we can mark that as resolved as well because that's a good thing to do on your pull requests and we can now click approve um because we're happy with what it's doing you can also click in here and from the pull request and see what's being changed and how many commits have happened and that's it so if we now click approve brilliant that's now approved and we can now complete the pull request which will actually push this code from the the demo one branch back into the master branch um and then the build pipelines and the deployment the release pipelines will all kick off to actually deploy this into azure because if we look in azure right now and we're going to the resource groups and hit refresh you'll see there's still nothing in my azure environment here okay fine so we're going to complete this merge for this pull request so um there's no associated work items so we don't need to tick that um and i'm going to customize the message if i wanted to for the commit message but that looks pretty uh uniform to me and we're going to say delete the demo one branch after merging as well so let's go and complete that merge okay and we can see up here it's merging that pull request and great that's completed this pull request is now show as completed and if we now go back into our repository and we are on the master branch here you can see that just now our pull request commit has gone in here so the merge of the code from our demo branch into the main branch has happened and we can now see we've got this demo tf file here on the left and also it's just listed here and we can see it says just now and it's come from the commit of our demo so that's great and actually if we go on to that file you'll see that it's exactly what we typed in vs code and if we actually go back to the main repository you'll see that there's this in progress icon up here now remember i showed you the build pipeline earlier of terraform plan well actually this has automatically been triggered by new code being committed into the master branch i thought we've protected that from being done directly and it has to be done via a pull request with branch policies and we'll dive into that later so if i click on this in progress here you'll see that the terraform plan pipeline is running so let's go and check out what's going on in there cool so we can see this is actually just completed it looks like it's just finishing up but if we go into the jobs here you'll see terraform plan and again it looks very similar to that status check that i had the only difference here is that once it completes the terraform initialization the validating the plan phase and again if we went into the plan we'd see exactly the same thing going on here so one resource to add and it's just that resource group all we do then is archive the resource we basically do a terraform plan to a file so as you can see here it says terraform plan and out tf plan so it's going to put it out to a file named tf plan um it then archives that up so puts it in a zip folder from my terraform directory and it'll publish that zip folder up to a pipeline artifact and that artifact will then be used further on in the process so if we go back to the pipeline we can see that terraform plan for the commit that we've just made has been complete um and actually if we now go into our releases and i will have actually got an email about this because i've got email alerts set up here saying that actually as you can see i'm pending approval for the terraform apply stage so this is sitting here there's still no resource group being deployed to azure so even though we've done one approval we've let it now run and you know and we've approved that pull request it's still not deployed because i've configured my azure devops pipelines in such a way that actually you sort of get two to approval chances so if we click into this release here that's pending as you can see this has been triggered from an artifact from uh the the build pipeline that's just run and if we clicked on that it would take us back into that build pipeline so we could see its history and we now can see that it's sitting here at the pending um approval for me to say let this go approve and run in my azure environment so they'll actually deploy this resource group so if i was to uh wanted to see what had happened before and what triggered this and what's going to be deployed i could click back onto to here and i could go actually let's have a look into the pipeline run that happened and it'll open up the build pipeline again and i can click back into the terraform plan at a build out job go and have a look at the bottom of the tf plan uh job stage and as you can see uh it's just that resource group to add so i'm fairly happy that it's doing what i want it to do and nothing else so i can go back into here and click on approve i can go yeah all okay and we can get that to approve now i could select this box here and defer it to deploy a different date and time if i didn't want it to happen now and that's quite a nice reason why i've got this second approval here before anything's deployed because you might want to say yes i approve this during the day but it can happen tonight when you know we've got a change window approved or something like that so i'm going to leave that unchecked now if you want to happen right this second and if i hit approve that's now going to kick off my terraform apply uh release pipeline so if we click on the logs of this you'll see we're just waiting for that to be queued and it looks like it's it's found a hosted build agent and off it goes so this is now gonna run so it's downloaded the secrets again we're downloading that artifact from the terraform plan build uh job that we had it's extracted those downloaded some key vault secrets now we've installed terraform done the terraformer knit to the the remote back end and now it's actually doing a terraform apply based on that plan file and that's it it's all done very quick if we just drill into that terraform apply step and we look at the terraform apply uh step in the job you'll see that apply completed one resource has been added and nothing's been changed or destroyed and if we look at the uh terraform apply command here that's used we're saying terraform apply auto approve and input full so we don't get any prompts at the uh the the terraform apply stage because obviously this is running in a uh hosted build agent which we can't interactively um interact with and we can't put any prompts in if it asks for yes or if it asks for a variable input so we need to put those options in to ensure that it just runs straight through without any prompts and when we say tf plan here that's using the tf plan file from that build stage that we we created earlier so we now go back into the azure portal and i hit refresh we've now got our resource group and with any luck we can see it's into my connectivity subscription as well so it's gone where we needed it and it's been deployed into north europe as well so all from creating a pull request and writing this piece of terraform code going through that process inside of azure devops and sort of our ci cd process we've now got that deployed and i actually didn't touch the azure portal at all so that's what it will do and that's what the the end result should look like let's dive into how we actually make that happen okay so our first step is creating an azure devops project maybe an organization as well if you haven't created one of those um if you haven't created uh an azure devops organization or haven't got access to one you can create one for free by just going to dev.jaw.com and creating an account there or log in with any valid microsoft account and create your own organization from there completely free to do for up to five users i won't go into the billing details here but for small demos and lab environments for less than five years is completely free to use um so go to dev.azure.com and create yourself an account or log in if you've already got one if you've already created yourself an organization like you can see down the left-hand side here mine's called jay tracy microsoft or msft and i've got one project already within here and this is the one that we we dived into in the demo just a second ago so wind's got a new project so let's click new project in the top right here and let's just call this tf azure devops demo and we can make this private so that it's only accessible to me and the people i give access to and i can hit create by creating this it will automatically create a git repository and azure repo within this project automatically so if we now go over to the left and click on repos you'll see that there this is an empty replay right so with the repo is already created here tf as azdo demo um what we're going to do is clone this into into vs code so uh you if you've got vs code installed uh which i'm assuming most of you will do by now you can automatically clone this into here by clicking the clicking the buttons up here but one important step that i think is really important and really nice of the azure devops repos option here is we can initialize this with an initial readme or a git ignore file so we don't want the readme file that's fine we don't need that but we do want this git ignore file um and actually there's a pre-built one for terraform so if you look for terraform in this search box you can automatically add a dot git ignore file for terraform now uh the git ignore file basically tells uh git what files not to upload to the central repository um and in terraform that is very obviously don't upload the state file don't upload any tf files files all of those sort of things so it will automatically add a preconfigured dot git ignore file so you can be safe in the knowledge that you won't ever be able to commit any of those to the central repository so don't click any of these options up here come down to the the bottom section here select terraform in the git ignore section and hit initialize and as you can see this repo now has one file called dot git ignore and if we browse into that file um as you can see it's just got all of the exclusions in here so exclude the local terraform directories exclude the tf state files and exclude the tfs files um all really important steps to make sure you've got done to make sure that you don't expose any secrets into a centralized repository and it's all shown in plain text um so if we go back to the the repository or we can click on the repository up here or we can go back to replays and click files and we can now clone this repository onto our local machine to make it easy for us to edit and upload files so if we click clone and i'm just going to copy this link here and then if i go into vs code and i hit ctrl shift p i'll get the prompt to pop up and if i type git clone you'll see that there's one already in here and i can select git clone and it's now saying provide the url which we've now got on our clipboard from what we've just pasted from azure devops and i can then hit enter now it's going to ask me where do i want to put this repository so i've got a place on my machine where i store all my repositories so i'm just going to put it in the right place here i'll say yep you can go there and now it's cloning that repository now if it's the first time you've done it it might ask you to log in and authenticate against that repository um but because i've already done it mine's taken me straight through so now that's finished cloning i can say yeah open that and now it's going to change vs code to that directory that we selected just a second ago and as you can see our i've got that one file that dot get ignore file which is exactly the same as what we had in azure devops if i go back here it's exactly the same file and that's what it is so we've now got these the local folder on my machine and if you right-click here and say reveal in file explorer so if inside my machine if i browse to the c git repos folder and then i've got one for all my azure devops j tracy msft projects um and this is the tf as as your devops demo project i created and this is the repo of all the files in the repo stored in here as we selected when we did that git clone process so that's everything we won't show in this step of the video we've got our repository cloned and synchronized with the azure devops project and the new replay that's been created inside that project um in the next step we'll dive into how we need to create a few uh prerequisites to store things like the terraform remote state some key vaults to store some secrets for terraform a service principle that azure devops can use to log in and access that key vault and access our azure environments and also creating a variable group inside of your devops so i'll catch you in the next step okay so before we can really get into configuring azure devops and terraform and linking them all together um we need to set up sort of some prerequisites for terraform now because we're using it in a uh devops fashion and we're not running it all locally from our machine we need that state file to be stored in a remote back end as it's referred to inside a terraform so what we're going to do is uh configure that to be an azure storage account and that can just be a blob storage account so pretty low cost obviously the state file is pretty small and but obviously it will grow however big your your deployment with terraform gets um but obviously by putting it as your storage account we can control it without the storage control access keys we know that it's encrypted by default um using the uh azure storage encryption services um so it's a good place to be storing it and obviously it means that you know the azure devops pipeline we can configure to get access to it and we'll walk through all those steps but first of all we need to create a couple of manual things now you might be thinking why aren't we creating this with terraform that's a great question um we absolutely could do this but it is a very much a one-time task so we need to create a storage account we're going to create a key vault we're going to create a service principle um they're really the only three things that we need now we absolutely could create them with you know a terraform uh deployment or an arm template um but as they're very very basic resources to create and we're only creating the ones um creating them manually in a portal is absolutely fine so first of all let's just create a resource group and i'm going to put this in my management subscription that i've got here and i'm just going to call this tf as do rsg so that we've got all of our things in the in that resource group and i'll put this in north europe uh i'm not going to put any tags on it yes put what you like in the comments um yeah i know that's bad practice um we're just going to create that resource group fantastic all right that resource group is created and then we want to create a storage account so we can look for a storage account great we will have one of those please and we just call this tf as doe uh demo storage account storage double a1 we'll pop that in north europe as well uh and we can leave it as uh v2 we can set it to lrs because we don't need anything higher than that we're going to leave the networking as public we won't dive into how we configure this with private endpoints or service endpoints um it's not something that's very easy to do we're using the hosted agents in azure devops anyway leave all these options there we're going to say turn on soft delete for blobs and keep that for seven days this is quite handy in case your state file becomes corrupted um you know we've got a way of getting it back and we're also going to turn on versioning for blobson for that case as well um so you know she's anyway delete the state file we can get it back for seven days and also every time it changes we're gonna get a version of it so we can roll it back as well so some this is a really new feature but blob versioning um but it's great to take advantage of this scenario as well uh go to the advanced tab we're going to make sure that secure transfer required is enabled we're going to make sure that tls version is 1.2 we're going to leave disable allow public blob access so nobody can get into this by uh just guessing the url and we're going to leave the access tier as hot by default and that's all we need to configure here again we're not going to add any tags um we'll go to the review and create page and hit create great all right well that storage account is going to create that shouldn't take too long if we dive back into our resource group we can create a key vault as well and this we're going to use to store all of our secrets that we want our devops pipeline to know about so things like the storage account access key our service principal login details all of those things so if we create a key vault and pop it into the same resource group so in that same resource group and we'll just call this tf as though demo [Music] key vault double a1 north europe again i'm going to leave that on the standard pricing tier um and then we're gonna enable purge protection and we're gonna say uh retain the deleted vault for 90 days and soft delete is automatically enabled now for us uh on the access policy um it's gonna add myself so that i can access and manage these keys in here we will need to come and amend the access policies later to make sure that the service principle for azure devops can actually read the keys but we'll come back to that we again leave the it to be publicly accessible from everywhere because your dev ops is running the hosted agents we know where it's go we don't know where it's gonna be coming from all the time um we don't need to add any tags uh for this demo and again we can hit create and pop that keyboard in as well so that's creating and let's go back to our resource groups because i'm pretty sure our storage account will be done which it is here's our storage account so in our storage account we need to add a container and let's just add a container here so we can call this uh terraform uh hyphen state because that's what it's gonna store make sure i've spelt that correctly and hit create okay so we've got a uh container now in our storage account called terraform state and we don't need to do anything there um and one thing we now do need to do inside of our storage account is just take note of the access keys so i'm gonna just open up a good old notepad window and we've got that here i'm just going to say these are the storage keys and key one and key two and the reason i'm taking these down for the moment is we need to put these inside our key vault yes i could use uh manage storage accounts in terms of the key vault um but for this demo i want to show you sort of the bare bones and obviously you can add on and change this slightly but if you understand the concepts you can do what you like to change these steps out as you please so let's get the access keys now obviously they're going to be blurred out for you but uh let me just copy those and i'll blow these out in the video as well so you can't get into my my storage account all right okay so we've got those copied to our notepad file here and we're not going to save this notepad file so you know be careful and because we don't want to store this anywhere we want to delete this once we're finished with it it's just a temporary storage location so that we can put these into our key vault all right so let's go back and find out that said key vault one thing is also worth taking down is the storage account name as well and because we're going to need that later so we're going to take that down and pop that there and the resource group that it also lives in is also something we're going to need later as well so we'll just take down the resource group name all right so let's go into our resource group and hit refresh because our key vault should be there now and it is fantastic and let's create a couple of secrets now these secrets are going to be the keys for our um storage account that we just created so we're going to give these a name and i generally give them the same name as the storage account and then uh append them with either key one or key two so i'll do that now we should set some expiration dates and some activation dates but you know for the ease of this demo i won't and we'll make sure that it's set to enabled and hit create and we're going to do exactly the same for the second uh key as well in here so let's go and get the name again and put key and go back to my contemporary notepad and get the keys make sure we're not copying any spaces or anything like that in because that will catch us out later and hit create perfect all right those keys are in the store or in the key vault from the storage account and what we now need to do is just dive back in to azure devops so let me find that so back inside of our tf asdo demo project what we actually need to do is create a service connection from here which will actually just create a service principle for us so what we need to do is go into project settings and if you didn't see that i'll just dive back into the project and show you how to do that so you go right to the bottom left hand corner next to the cog and you've got project settings and then in the new windows that appear on the left hand side you've got an option under pipelines called service connections and it says look you haven't got any but do you want to create one yes we do so let's create a service connection and as you can see we've got one in here for as your resource manager because that's what we're ultimately connecting to at the back and we're connected to arm so you can hit next and we can say this is automatic now this can actually load the subscriptions that this account has i've got access to so i don't have actually have access to um those subscriptions that i've just been creating from the account that i've logged in with my azure devops now if you've done this you can follow this process through and create it from here and it'll be nice and easy but i don't have that privilege so we're going to go back and say actually we're going to do this manually so we're going to say it's the is your public cloud and it's going to have permissions over a subscription and let's get the subscription id so let me dive back into my azure portal and go on to my management subscription and get the manage subscription id and let's get his name and then i'm actually gonna need to pass in some details into here so i'm gonna need to pass in the service principal id uh the service principal key the tenant id and click verify now it won't actually create the service principle for me so we're gonna have to go and do that in the usual portal manually so let's go back into azure and let's go to azure active directory and let's go to app registrations and click new registration and let's just call this tf as do demo spn and we can leave it with the first option here and we don't have to put in the urls that's fine because it's all good we can hit register and that's it so we need to copy the client id and again i'm going to use my trusty notepad to store all of these details temporarily so we can call this spn details so this is the client id we need to copy the tenant id so tenant id uh copy the object id and we need to also get a secret for it so to do that kick certificates and secrets and then under secrets you can hit new client secret um choose what how when you want this to expire and obviously you know this will fall into your secret rotation policies and stuff like that so the shorter the better and i'm just going to call this a secret one for the demo purpose now when you hit add here it's very important you don't click away from this page because the secret is only shown once so if i click add you'll see that this value is is blurred out for you but i'm going to click copy to clipboard and the secret is actually there and i'll just put this here and again this we've learned out for you but make sure you copy that secret out there because once you click away from this page the secret will still be valid but you will never be able to retrieve it again you'll have to create another secret so we've got all of our details now for our spn so let's go and fill in the azure devops wizard so it's asking for the service principal id and i'm pretty sure that that is going to be our client id so let's grab that and pop that in there uh the key which is also the secret so let's get that and pop that in there the tenant id which is our directory tenant id which we copy out as well and that's it and we should be able to say in a service connection name so we can call this tf as do what we actually call the service principle let's use the same name because then it makes everything match up uh if i could copy that there we go and we could put a description if i wanted to um and that's about it i think let's hit verify make sure it works uh oh and it didn't so it says i write so it doesn't have permissions over the subscription very good point that's why the verify button is very important we all make mistakes so we've created our service principle but we actually haven't given it any permissions over anything in azure so actually let's go to our subscriptions let's go on to our management subscription and let's go into access control and i am going to role assignments and as you'll see my other spn for terraform is in here and it's actually inherited from my management group structure another one that i've set up in the past but um the one that we've just created hasn't got any permission so let's go and add a role assignment in here and let's make it a owner because we you might want it to create uh our back assignments on it on the subscription so you're going to need it to be an owner and let's search for its name so let's go for tf as dough and see what pops up and there it is click add and click save and then that is going to give it that our back permission now that should be fairly quick um however it can take up a sort of 10-15 minutes to apply so let's go back into the portal and click verify great that's already taken effect so it nails confirmed it's got permissions over the subscription that we specified up here that's all looking good and we're going to say all of our pipelines can access this permission as well so let's verify and save and great that is now created so we've got that service principle here and you can see it's been created and we can click on manage service principle it will take us to where that is um but all good we don't need to do anything that's there now the final piece we need to do is put all of those service principal details into our storage account as well sorry not into our storage account into our key vault as well so let's go and do that so it's back into the azure portal back into our resource group that we created find our key vault and we need to create some secrets so we're going to call it the name of the uh the service principle again just to keep everything uniform and i would recommend keeping everything lower case um especially if we're planning to use linux build agents just because uh case sensitivity will come into play um so we're going to need to add that there and copy that to our clipboard because we're going to need that more than once in a second um first one will be the client id so hyphen client hyphen id and let's go and get the client id here and again make sure we're not copying any secrets or uh not copying any spaces or white space into here uh i haven't had that and we're gonna need to do the same again so let's go back into here get our prefix for the secret and this one will be the tenant id maybe way wondering why i put all of these details into the key vault it's just a habit i think it's it's good that everything's sort of contained within the key vault if it's going to be queried um because then we know it's all being accessed securely it's all been accessed over sort of tls uh when those certificate those keys are being retrieved um but you know you could choose only to put the really important stuff in here like the secret um completely your choice again i'm showing you the full step here how i do it you can choose to do this any way you see fit once you understand how this all works uh so let's pop in the object id and let's get that id there and pop that in and the final one will be our secret and the most important one and uh hopefully i've blurred these all out before i publish the video uh that's out there now for the secret we obviously set that to expire in a year um so that would be probably quite a good one just to add in quickly so uh that's saying 2022. that's actually wrong so let's say 2021 and let's set it for that then and let's just set it for say i don't know one minute past in the morning uh there you go 1201 a.m uh and that's it so we can query this and we can get alerts based on when that is going to expire and let's hit create and great all right so we've got uh six things in our key vault as secrets and we've got one with an expiration date because we know when that is um and they're all in there and we're going to come back to these later the final thing we need to do remember what i said when we created this key vault is amend the access policy to ensure our service principal has access over uh to see the keys so if you click on access policies you'll see that my account who created this got permission to see the keys and add keys we actually need to add an access policy and we're not going to use a template we're just going to go into secret permissions and hit get and list so we're just going to say that the only thing that the service principle that we're going to use from azure devops the only thing you can do is get a list of secrets and so we can select there and select our principal and if we look for tf as do spn we can add that in there and then we can click add now the most important thing to do that a lot of people forget is to click save up here otherwise you'll click away and it won't actually do a take effect so we can click save and that's it so that's it for setting up uh the service principle we've got one more thing to do inside of azure devops which is create a variable group which is going to link the azure devops variable group to the azure key vault that we've got here based on that service principle we've just connected so it's going to use that to talk to the key vault and create a variable group which we can then use in our pipelines later so if we go back into azure devops and if we go over to the left hand side and click under the rocket and then library we can click on there and click on that and this is what we want we want a variable group so let's create a variable group now i call this the same as my key vault just so that i know what it is so let's go and get the name of that key vault in that case i'll keep all the naming consistent just so that it makes it easier for me to work out what's going on now we're going to allow all of our pipelines to access this and the important select role radio slider here is this link secrets run as your key vault as variables so it means we're not just going to put anything in the the variable group here we're actually going to go and get them from the kit the key vault and therefore we know it's secure because we can control the access policies on that key vault so we can select there now it's going to ask us well where is it so if we click this drop down now because we created that service connection earlier we should get a drop down of here of our service connection to use and we do so our available service connections here is this tf asdo demo spm one that we created um and now that's gonna let actually query the uh azure apis and it's found the one that it's got access to uh if you haven't done the permission granting it's not gonna be able to see this so we can select that key vault because we've done all of that and now we can click add variables and now it's going to query that key vault and here you can see you can see all of these um secrets here actually match the secrets that we've got with inside of our key vault so it's queried these and brought it out and it's even brought across the expiration date as well as you can see for one of these so we want all of these so check all of them so the visual devops knows about how to get these and we can click ok and that's it hit save so now we've got a variable group that we can use in our pipelines later that automatically links to our key vault so as your devops doesn't have the secrets it has to go to the key vault so you've got one place to store and manage all of those secrets and keys if you change it it's over there this part in azure devops the variable group purely links as your devops to that key vault so that our pipelines can use that okay right that's that's this section i'll up stop here and we'll get into the next step in a second okay so in this step we just need to set up some prerequisites for uh terraform as in we need to put some terraform codes we need to put a provider's file together and a provider block together to tell it we're using azure rm provider from terraform's perspective and we also need to forget the back end to make sure it's going to use that storage account um even though we won't be able to test it works um because we're not going to specify the key um we're just going to put those files into our git repo and commit that up to our azure devops repo inside of our project to make sure that the code's there so that when our pipeline's run they can see that code so we need to go into vs code and then first thing we need to do is create a folder because i like to put all of my terraform code inside of a folder inside the repo just in case we you know we could like the git ignore file we want that in the root leave that there and you want to put anything else any other structure in here you haven't got to move stuff around or changing your pipelines start from the beginning by putting everything in all your configuration files in a folder so let's call it terraform great and inside this folder when you're going to create a new file and let's call this uh providers dot tf okay so first thing we need to do is uh specify what provider we're going to use so in true blue peter fashion i'm just going to copy one from here i had earlier um so i'll just zoom in as well to make that a little bit bigger so we've got here we're going to say the provider is your rm um the version actually let's change that to use the latest version uh because that's saying anything equal to or later than that version there i think it's equals and let's say let's use the latest version i think that's 22.37.0 um you can check that on the terraform as your rm docs provider page and it'll always show you the latest version at the top i'll put our features block in here and i always specify the subscription id to just to make sure that this is using that subscription so this subscription is actually the subscription id uh even though you can't see it because we've blurred out this is the subscription id of the management subscription i have where we've created that storage account and the key vault and all those things and obviously the service principle has got access over so that's there the other file we're going to need to create is a backend file and most importantly make sure you save your save your file um so we save that let's create another file let's call this backend.tf and again i'm just going to go and copy this out and we can then um amend the the values so we're saying the back end is your rm again now we're going to need to put in the name of our storage account and uh let's go and grab that so let's just go into our resource group into our demo into our storage account so we need that let's pop that in there uh we also need the container name which was just terraform state and a key now this isn't the key to access the storage account key is actually just referring to the the name of the tf state file file it's going to create so let's rename this to tf as do demo dot tf state and that's just going to be the name of the file now there are some missing properties out of here on purpose being the the main one there the the access key to get into that storage account um but we don't want to store that in our code we don't want to commit that to our repo that everybody could potentially see so we we put that in our key vault and i'll show you later on how we're going to get that and run that at the build pipeline time to make sure that um it's not stored in our code it's injected at runtime and nobody ever knows what that is so again save that that's all complete and then the other file we need to create you may have saw on our providers file we've got these variables here um saying we're going to take these as inputs because obviously again we don't want our client id or client secret and our tenant id um stored in our code either that's again in our key vault that we're gonna use as part of the pipelines to to inject when they run so the next file we're gonna create is uh we can just call this variables.tf and again in a blue p to fashion i will copy ones i made earlier so i've created a variable for spn client id the client secret and the tenant id which matches up to these sp variable names here again let's save those and once they're all saved we just need to commit these up to our repo to make sure that they're in the usual devops repo and they're not just sitting locally on our own cloned version of the git repo so we're going to stage those changes by clicking add and then we're going to say uh and tf initial files and click commit now that doesn't actually put them up to azure devops just committed them to our git repo locally here on my machine so down at the bottom you'll see that i've got this uh sync at sync icon and a zero saying there's nothing to download um but i'm actually one ahead so i need to push something up to my repo so if i click that button there that will sync it with my repo in in azure devops and once that stops syncing we can go back into azure devops and if we go to our repos and files and there we go we've literally just just now we've created this we're going to here we can look at terraform back end terraform providers and it looks like i didn't create the variables file um because it's not showing here or if i did create it i put it in the wrong location so let's just double check that so let's go into here and yeah absolutely i put it in the root of the directory not in the terraform folder so if we go back here look it's sitting in the wrong place so let's go back on my machine and let's just literally drag and drop that into the right folder and move it now that's going to show us a couple of changes in get because technically it's a delete and an ad back into the right place so we can stage both those changes again and we can say uh fix far file location and hit commit and again push those changes and if we wait for that to finish and go back into azure devops and do a refresh we're going to a terraform folder as you can see they're now all in the right place we've got our variables file our providers file and our backend file so all looking good everything's in in the right place and we've got these three files set up ready to start building our pipelines which we'll get onto in the next section okay so we've initialized azure devops we've created some prereqs for in terms of the storage account for the remote backend the key vault we've set up the service connection azure devops we've also set up uh the terraform files to be ready so that we can inject uh those keyboard secrets at runtime we now need to get onto actually building some pipelines right so the bits that you've probably all been waiting for so back into azure devops and then we need to click on the rocket and the pipelines on the left and click on pipelines now i am going to create these in the classic way um so yes we could create these in the yaml format but i really want you to get hands-on and understand exactly what's happening at each step without having to learn the ammo pipelines at the same time so yes you could do this all in yamaha pipelines and if you can fantastic great but for this example i'm going to use the classic pipelines just so you get a really hands-on experience on how to do this so we're going to click create pipeline and then really importantly when this loads up don't select any of these options here we need to click use the classic editor down here so once we click use the classic editor we'll go a different prompt and now we can start using the options so it's going to say where's our where's our source code stored well it's actually in azure repos git it's inside of this project and it's actually this repository and we want to trigger use this branch right so this is the branch we want to use our main branch hit continue and then again we want to start with an empty job so let's just hit empty job and great this is our main editing window for our build pipeline so the first pipeline we're going to create is our terraform plan pipeline right so the one that when somebody approves a pull request the first thing we need to do is a plan and make sure that what's going to what are they going to do and is it you know are we approving of what they want to do so it's creating that plan file doing the terraform plan action and spitting out a plan file so let's rename our pipe build pipeline to terraform plan and then what we also need to do here is say where our agent pool is now i'm going to use the azure hosted agents because i don't have any of my own private self-hosted agents so we're going to say the azure pipelines and i'm also going to select from the list to use ubuntu 18.04 um this is just personal preference you could use windows i just like using linux because i find it a lot faster for these sort of actions cool so now if we click on the agent job one over here we're just going to rename this agent job again to terraform plan like that and then we're going to say use the the agent pull from from the pipeline which we set back up here to ubuntu so back onto here and we now need to add a job so if we click or add a task sorry to this uh job so if we click on this add symbol here and we've got all these options here now we actually need to the first thing we need to do is install terraform now it is actually installed on uh the hosted agents automatically but we just need to make sure it's the right version that we need right you might be using a different version of terraform to the one that we've already pre-installed so it never hurts to make sure that the right version is installed so if you look for terraform uh ins tool installer you'll narrow this down to one list and uh you'll see it's from us that we it's the microsoft corporation um marketplace item for this and we can click add now great that's been added now i've already approved this from the marketplace and added it to my uh devops organization if you haven't you might have an option on that uh plane there to select get and it will take you to another window where basically due to the azure devops marketplace shows you the same extension and says do you want to get this for free you hit you know install and you would say you choose the organization that you want to link it to which in our case would be in this example the jjc msft or devops organization and you'd say you know what one that goes into you and then it can be used by all of your projects with inside that organization so that's already done for me so i've got that here and we want to you know install the right version so the latest version of terraform is 13.5 and as you can see it will change the display name up here for me so let's say we want terraform version 13.5 to be installed so that's our first step and these all work procedurally so it's in order of the list and we can drag these around in a second but you know do it as as you would want it to run step by step now the second step we actually need to do is um do a terraform init so let's get a command line option here now there are pre-built terraform marketplace extensions let you do this but i find that doing it yourself gives you a really good understanding of what's actually happening behind the scenes and you remain in full control so select this command line option here and again it's from the microsoft corporation um and this may say get but and if it does say get click get come in here and then hit refresh you'll be able to add it back in once you've got gone through that install process um but it's already added to my organization so i can click add and it's got command line script here so this is going to do our terraform in it so let's give it that name so it's very obvious on what's going on and this is where we put our script so i will just grab one that i had earlier and we can replace the bits that we need to replace so as you can see here saying terraform in it which come on you should all be comfortable with and then we're passing in this back end config now if you remember inside of our devops sorry inside of vs code inside of our backend file we had the storage account name the container name and the key but we didn't actually specify the access key as in what's what's the secret to getting and do stuff and put files in there or read the files into that storage account so to do that um we are actually injecting effectively another line into this back end block with this command hyphen back end config and then equals access key now as you can see i've wrapped this in speech marks and then i've got a dollar sign um and then another brackets now this actually means go and get another value it's an interpolation syntax right so it's actually going to go and get this from a variable and we're going to link this up to our variable group at the end but what we're going to do is we need this to be match the key vault secret name that stores the access key so if we go back into our key vault in azure and let's just go back to our resource group into our key vault and into our secrets we've got this one here so let's use key one for this just in case we want to roll it over we've got key two there in case we need to change this at a later stage very easily so let's grab that and let's pop this inside of those brackets here so we're saying that the access key is actually stored in a variable and at runtime it will go and get that from the key vault via the variable group that we set up earlier and inject that into into this string instead so that's one thing here one thing we need to do as well is that all of our terraform files are actually stored inside of that terraform folder inside our repository so we just need to tell that this command needs to run inside of there otherwise terraform will look and say well there's no files in here what do i do so if we click on that and we click on our terraform folder and we just click okay on top of the terraform folder and that means it will run this command within the context of that terraform uh folder inside of our repo so that's great it's now gonna initialize get access to our back end all good first step done download providers all those sort of things the next step we want to do a terrible validate just makes for sure that it's all syntax syntactically correct so let's download this task so you can right click on the the task here and click clone and that will keep all these properties in the same and just saves you a bit of typing so let's change this one to terraform validate and the command we need here is very very simple it is literally terraform validate so that is the only command we need there and again just check under advance that our working directory is still that terraform folder inside of our repository we then want to clone the task again because then now we need to do the actual terraform plan so let's rename this to terraform plan and uh let me go and get again the terraform plan command that we need to use so quite a long command but let's break it down step by step so terraform plan a command that we're all used to so do that plan stage tell me what you're going to do hyphen input full so don't prompt me for any input variables at runtime and obviously because this is running in a hosting container we're not going to type anything in so we need to make sure that this is actually happening this this doesn't happen we don't get any prompts to put in input variables so if they don't exist um or they're required it will just ignore them and carry on with with null values um hyphen out equals tf plan so we're saying output the tf plan so the terraform plan to a file that we can use later so that we can save what was going to happen in this terraform plan action and use it later so out into a file called tf plan and then we're injecting a load of variables so var spn client id var spn client secret var spn tenant id now you might think we've created these earlier and put them in the key vault and that's absolutely correct and if we also go into our terraform configuration in our providers file these are all these variables here's right so we're specifying these values because obviously they they are just variables at this stage we need to put this to real values we don't want to put them in the code we want azure devops to inject them when this pipeline runs so we put these all in our key vault so again what we need to do is change the values in the brackets so the first one we need is the client id so let's go and get the client id from my name of the secret inside my key vault and that's the one we need there and change the relative the respective one inside the brackets and there we go make sure we've got no spaces around it the second one we need is the secret and instead of going there because we put in a good naming scheme for our secrets this will be very easy for us so we can replace that there with secret and this one here we can swap out as well and change that to tenant okay so that's everything there and then under advanced just make sure we're in that terraform folder again right and now we need to add an archive action so we've done all the terraform actions the terraform plan has happened it's output a file called tf plan and now we want to capture that tf plan and all the files into a zip folder and then publish that to an azure devops pipeline artifact so that our release pipeline can go and get that and run it and extract it and make sure it's only happening against those files and not any new ones that have happened in the repo since so we need to add another action so click on add here and we need to add this one that's called archive and as you can see we've got archive files and again it's from the microsoft corporation and you might need to do that install process and we can click add and let's put it in here so i call mine archive terraform plan files so let's just rename that here and we now need to uh put in the folder we want to basically um zip up so that's our terraform folder in here so if we click on here it's going to be in this folder because that's where we run those other commands above from so we can click on okay and then we need to change the archive type so because it's running linux let's use tar and gz compression and then what we need to do is just specify the uh file that we're going to call this our this zip folder to or sorry not zip this targey zed file um to be called so let's just paste that in there make sure we've got no spaces and these variables are automatically built into um azure devops so it's saying the build artifact staging directory and then create this file here so that's fine um you can look at these all up in the documentation there's lots of them in terms of this and pre-defined variables that you can use and that's fine we can leave that there we can leave these check boxes as they are and that's that one and then we need to add another action which is publish terraform plan publish the terraform artifact that we just created so we go publish and publish pipeline artifacts add that one in and again this one i'm going to call publish terraform plan artifact and the directory for that is going to be the same as the uh archive here so it's where we're creating this archive here that is exactly the same as the directory that we want to publish so we're linking those together and we're going to give our artifact name uh just the build id so the build id is the the run number for this build pipeline so that will change every time it runs it will just go incrementally up and then hyphen tf plan and we probably want to publish that to visual pipelines and that's it so that is that is it for this one so there's a couple more steps that we just need to do so click under the variables tab and then go under variable groups and what we can do here is a link variable group and here's the variable group that we created earlier at this link to our key vault and we can click link and that will give this pipeline access to that variable group and therefore access to the key vault using that service principle we created earlier and if we drop that down you'll see that all the values are here and it doesn't know them but it will go and get them at the time of run and we just need to set our trigger as well so click on triggers we need to click enable continuous integration and that means that when anything changes in the main branch run this pipeline we're going to protect that with a branch policy to make sure it happens via a pull request also we're just going to actually add in here that it only happens on files that change in the terraform directory because if it happens outside this directory it's not related to terraform and therefore we don't care about it okay so that's that so we can hit save um don't hit save and queue there's no point we don't need to run it right now we can just hit save and you could select a folder if you if you've got different folders for your build pipelines but we can just say uh the default root directory and hit save and that's it that's saved if you now click on pipelines again on the left you'll have a terraform plan here and it's got no runs if you ever want to go back in here and edit it you can click on the ellipses here and click edit or you can click on the actual pipeline itself and click edit up here now because i've clicked into that pipeline we actually want to clone this for our status check stage so if we click on there and click clone it's going to clone this pipeline and our status check is the one that runs whenever we create a pull request just to check the code syntactically correct and the reviewer can check what it's doing now we can just rename a few bits in here so if we call this terraform status check and i'm just going to put in brackets what it does so it's a validate and plan with no help file and we want to use all the same options and i'm just going to copy that name and put it here as well and then we just need to delete a few things in here so we still want to install terraform we still need to do the terraform initialization um we still need to do the validate and we still want to do the plan however we don't need to put the out file in here so we can take that out we still need to pass in those variables and we don't need to do these these archive steps so we can remove that task and we can remove the publish artifact task as well because we don't need that we just want it to run and give us the outputs um check that it's still running inside of our terraform folder and then we still need to make sure it's got access to our variable group which it has if it hasn't re-linked that and we need to go into triggers and we actually need to disable the continuous integration trigger for this pipeline run because we don't want this to run every time new codes committed we've got our terraform platinum pipeline to do that this one should only run when our bill when our pull requests are created and we'll get on to how to configure that later so again hit save leave it in the root folder and just hit save so now if you go back to pipelines we should have two pipelines in here with no runs and that's it we've created our first uh build pipelines that are going to do the terraform plan stages and we'll get on in the next section on our release pipeline and which will do the actual deployment on there and how we can control the approval gates and all those sort of things so catch you in the next section okay so we finished creating our terraform build pipelines for our terraform plan states and our terraform status check that we're going to use with our pull requests so now we just need to finish uh creating another pipeline in our release section so back into azure devops again under the pipeline section we have releases so let's click on releases and again we're doing this the the classic uh classic way um we could do all this inside of a young pipeline um but again we're going to do this uh the old-school way so you understand truly what's happening each step of the process um so we've clicked on releases and we've got no release pipelines let's create a new one and when we create that new pipeline it's going to pop up here with again all of these options let's just select empty job and we need to give the stage a name so i call mine terraform apply um and i can be the owner of this stage and you can just click away or click the little close button button here to move that away now what we also need to do here is link up the artifacts so at the end of the terraform plan stage we export that artifact and we need to make sure that we're using this to trigger this pipeline uh so if we click on our add an artifact and it's from our build artifact in this project and if we drop down this source we'll see that we've got our two build pipelines here so let's just select our terraform plan one because that's the one that exports an artifact and as you can see it spits out all of this here and we can leave all of these options as default and click add and then what we can do here is click on the little uh lightning bolt in the top right hand corner of that terraform plan artifact and we can enable the continuous deployment trigger so we can say that every time that there's a new build available you can kick off this pipeline and obviously we're only going to allow this to be triggered because the terraform plan build action will only ever trigger once code is pushed pushed or merged into the master or main branch and that's going to be protected by a branch policy and we're going to enforce that with pull requests to make sure that everything's reviewed and approved before it actually happens so we can enable that ci cd trigger there and that's uh all we need to do there when you hit the little red cro the cross in the top right hand corner um and then we need to actually configure the steps of our job so if we click on um [Music] the little blue where uh one job zero tasks uh link beneath our terraform apply section and click into this and it looks very similar to the the previous build pipeline creation steps so we're going to do very similar things again so we're going to change our agent job to run on ubuntu 1804 we're going to change the name of the agent job to terraform apply and then we're going to add some tasks in again this time we're going to add an extract uh files task first so when we've got that it pulls that artifact in automatically as part of this pipeline because it's used as the trigger and then we need to extract it because it's in a tar gz format as it stands so the command we're going to need to pop in here or the file pattern is as per the screen now that i've just pasted in so this is the alias of the artifact that was configured when we linked the artifact on this release pipeline and then this will look very similar to what we put in at the previous stages um when we created the build pipeline for a terraform plan so pop that there and these will all be linked to in the blog post compliments of this um video so don't worry about taking a note of all these these will all be available for you um and then in the destination folder we just need to dump it to the default working directory of the of the hosted agent so we can select that there and again i'll put all these in the in the uh blog post um we need to uncheck this clean destination folder for extracting so we don't there's nothing in there already so we don't need to clean it it just slows it down um and that's it we then need to go get into the terraform install process so we can look for our trusty terraform um tool installer again and there we've got it and we click add and again we want to make sure that these versions match so 13.5 was the one we used before and then we need to add a command line task so we look for that in the marketplace and click add and this is going to be terraform and nip and it's going to be the same commands that we use in the build and the status check pipeline so let me just go and grab those and actually if i open up a new tab i can go into one of our other pipelines and go and get the commands to save me having to change them so click on telephone in it copy the script and then back into our release pipeline and pop that in there we're also going to need to make sure we're running this in the right folder this time it's a little bit of a different uh path so we're extracting the files to the system default working directory variable and we need to just make sure it's going in there and then inside the terraform folder that's contained within that zip file so we can paste that in there so inside our default direct working directory and then the teraform folder within there and then we need to uh add another action for terraform apply so we can copy as we did before select tariff form apply in there and then all we need to do here is um add in the parameters for the command here so terraform apply hyphen auto hyphen approve which means uh if you run terraform apply normally on your machine it actually does a terraform plan first and then it will say this is what i'm going to do this is what i'm going to change this what i'm going to delete this is what i'm going to add do you want to do this and you type yes obviously because it's running in a hosted agent as discussed before we can't do that so hitting uh putting in this parameter hyphen auto approve um stops it prompting for yes it'll automatically approve it and go go to the next step and then we also need uh hyphen input equals false as we discussed before we don't want any uh sorry equals there let's make sure we put equals to make sure that no input parameters uh that are required are weighted for and prompted for the terminal because we won't be able to put them in and then we need to put tf plan now that's the tf plan file that we export during the terraform build or the terraform plan build pipeline that's the artifact that's exported and we're saying we want to commit or we want to do an apply based on what was in that plan file so not what was in the repo now because that may have changed we wanted it based on what was in that plan file at that time because we can ensure consistency throughout the approval process and we never get into that stage where you may approve a release pipeline to run it goes and gets the latest code from the repo and that might be different to what was initially proved and approved in the pull request so by making sure we're putting the terraform plan from the build pipeline out as a file we can ensure that that file is kept all the way throughout this process consistently and it's only ever the what's in that plan file is committed and applied to the environment and again we just need to make sure that's going into the same directory and that's it for our apply stage now actually we need to link our variable group again so we can go into variable groups and we can click link variable group add that one in for our key vault again and hit link and if we drop that down we can see we've got all of our variables in there again um that they can be used when we're uh the tasks query for them so they'll be swapped out um and that i think is pretty much it uh apart from our approval gates so at the moment this will just automatically run and won't ask for approval now to add that second step of approval before it deploys if we add an approval gate so if we click on the lightning bolt on the person on the left hand side of the terraform apply stage we can say actually there's a pre-deployment approval required we can select that to enabled and we can put in the people we want in here so let's put myself in for example and we can set how long this approval can wait so let's say it can wait for seven days and after that um it can disappear and it'll be just be denied um and we can also set some other options here like the user requesting a release or deployment can't approve it now obviously i would select that in a best practice world because there should be multiple people in the environment but in my world i need to leave that unchecked because uh i'm the only person in this environment uh we can select revalidate the identity of the approver so force mfa tokens if they're configured and all those sort of things to happen um and also we can select this option here that says skip the approval if the if the same approver approved the previous stage um that's not relevant in this in this environment so we can select that there and that will basically when this pipeline runs when it gets to the terraform apply stage it will trigger another approval and until that approval is granted the steps that actually commit the terraform applied changes to your azure environment won't take place so that's all good there and we can just rename this pipeline as well to terraform apply and that is it so we can hit save and again leave it in the root folder okay all right so we've got all of the pieces now falling together so we've got one final section and let's talk about our branch policy so i'll catch you in the next section okay so we're at our last stage now we're going to enable a branch policy to ensure that the code going into our main or master branch is only coming via a pull request and can't be directly committed from a merge or a push from any of your git clients so to enable that we need to go on to our repos and select branches then for our default branch and this could be any name but mine is main in this example yours may be master but you can call this whatever you like when you create your repository so make sure it's the default one and if you go right over to the right hand side and click on the three dots and click branch policies and when this loads you'll see we've got a number of options to enable here for the main branch so the first one we need to enable is require a minimum number of reviewers so you should leave this at two or more depending on how many people are in your environment but i had as i'm the only person in this environment in this demo i'm gonna set that to one and because there are not two people who could manually approve this um i also need to select this allow requesters to approve their own changes because i am the only person in this environment i would suggest leaving that unchecked in most scenarios um i also need to want to select when new changes are pushed reset all code reviewer votes that means if we got a current pull request in place and somebody puts their you know submits the pull request based on a certain commit of their branch and then they suddenly add another thing into that pull request if i are pushing code into the same branch again we want that completely re-reviewed because that may be completely changing what's getting deployed into azure via terraform so i'm going to reset all the reviewer votes and they have to vote again um then we we can limit merge types now this is personally just a personal preference i only allow squash merges so that we only get one commit per pull request uh going back into the main branch to keep things tidy um but you know use whatever your personal preference is there and then under this build validation step and what we want to do is add one here so we can add a new build policy and if we drop down our build pipelines we've got a terraform status check which is the one we want because we don't want the file to be created the out of it but we still want to see if the terraform code is valid and also that what that code is going to deploy so we can review it in the pr stage i'm also going to put in a path filter here and we're just going to put in here star dot tf because that's all the terraform configuration files i will be using as a file extension so we're going to set that there and we're going to set the build expiration on here to be immediately when when main is updated and we're going to set a display name check here uh or a display name to terraform status check and then we just put in here the same as what it's called so validate and plan no out file and just quickly to unders make you understand why we're saying the build expiration immediately when maine is updated if main is updated there might be multiple pull requests going on in your environment and if somebody you know submits one and it goes through and gets committed to maine that's going to change effectively what your code is going to deploy so we need to basically scrap that and do another build and see how that that your current code changes would impact what's in uh the main branch so it's best practice to do that in in this sort of scenario so we're going to hit save there and one final thing that you can choose to do as well is this bottom piece so automatically included reviewers so instead if you have to specify every time when you create a pull request you can set a policies effectively saying these are the people that i absolutely always want to review my um pull requests and without having to specify them every time so i'm just going to add myself in here and say i'm a required approver and we can say requesters can approve their own changes as well and let's hit save okay so that's that's it there's no save button on this option it's literally just enabled the sliders and and it's on to test that if we went back to our uh main repository now instead of going into vs code and committing something pushing it up we can do exactly the same here so if i just go into the git ignore file for example and do an edit and let's just add test test comment i shouldn't be able to commit this because it's not gone through a pull request based on the branch policy so if i hit commit and say update git ignore and hit commit i get an error message saying push to this branch and not permitted you must use a pull request to update this branch and that's because we've put that branch policy in place and that would work exactly the same if i was using vs code or visual studio my machine to push that git change back up um it wouldn't be allowed it would deny because it's not being done via a pull request so i'd need to create another branch put my commits into the other branch and then create a pull request to put it into the main or master branch okay so that's everything done here uh the next step is giving it a go so i'll catch you in a second we'll give it a go and hope it works okay so let's test out all of those things that we've put in place in azure devops and make sure it works so first off let's create a new branch inside of vs code so click on down here on main and click create new branch and let's call this um test one and we can see that that's changed now to that branch inside of vs code and that means git has changed to that branch as well let's create a new file inside of our terraform folder and let's call this main.tf uh and let's just create a resource group and see what happens so resource azure rm resource underscore group and let's give it a name so test rsg01 and let's say its name is test rsg001 and its location if i could spell location equals um north europe and that's about it uh let's format this file so uh shift or f if you've got the terraform formatter installed which i don't in this example but we know let's just make sure all of our equals signs are aligned as good practice um and that's it so save the file and then you can see on the left hand side we've got something to commit to get so let's go in here and it says main.tf that's the file we've it's got updates so we can stage that and let's go add test rsg and commit that to our branch and then push this change to azure devops so that's pushing now and if we go back into our azure environment first of all let's just go into our resource groups and as you can see we don't have a resource group called test rsg001 um so that doesn't exist at the moment so we we don't see that here in orange environment but hopefully after this process we will um so let's go back into our azure devops environment and click back on files and let's just say leave on that option and you can see we updated test one branch just now don't want to create a pull request yeah so let's create a pull request and let's just call this test one create test rsg and we could put in here testing and then we can say creating tech test rsg01 um we can leave all these options as blank because we've set up our policy so that i should automatically add it as an approver here and we can hit create and there we go okay so we can see a couple of things here so our terraform status check validate and plan is on its way that's getting run and you can see i've been added as a reviewer so let's go and check out this so if we click on our terraform status check and go and inspect what's going on in here so it's installed in terraform it's doing the initialization it's done the terraforming it and done the terraform validate and it's doing the plan now it's just checking what to go on and great that looks like it's worked let's just check what was in the plan so the plan says we're going to add one resource group called test double test rsg01 in north europe fantastic so that's what we wanted and the most important part here is if you look at the terraform plan and input where we've got our commands remember in our spn client ids and these secrets that we're passing in from the key vault you'll see that actually we don't see ever what those passwords or secrets are we just see star star star and that's because your devops is aware they're a secret because they came from the key role in the variable group so you can see the very first one of the very first jobs is it downloads those secrets from our key vote using the service principle that it has access with via the service connection we created at the beginning so as you can see it's downloaded all these secret values and put them in as environment variables they're only last as long as this hosted agent container is running and as soon as it finishes that's it it gets blown away and created and you know rebuilt from scratch so it'll have no knowledge of these again so let's go back to our pull request so if we go into repos and pull requests we'll see that so our required check completed and if we click view check that's that status check that happens that's all good if that failed we'd get a red cross and we'd be aware of that and now it's saying jack's got to approve it so you know if i was approving this i'd come into it and look at a few things i'd first check the files and see what was being changed so i can see yeah it's just one file and it's that here and then i'd also go back and check that this required check had completed by clicking on there going into the the job and looking at the terraform plan stage and just checking the bottom of this file to make sure what was being deployed is what uh the pull request was stating was being deployed and you know it wasn't going to break anything or delete anything that i wasn't aware of and once i was happy with that i'd go back and i'd go back again sorry back to the pull request and you know if i'm all good here i might put some comments if i need anything changing or i can say yeah it looks good to me and i can say let's resolve that comment so that doesn't get in our way and i can say yeah approve and now i can say complete and based on our policy that we set we said only the squash commit could be be allowed and as you can see the rest are forbidden um and i'm just going to customize the merge message here so that it you know we'll just see what it's going to create and i quite like this so it's saying merge from pull request eight based on the pull request title and it puts in the the description notes here as well uh so i'm going to say complete merge and it's also going to delete that test one branch from the azure devops repo automatically for me once it completes so we can say merge there and as we can see it's now merging that pull request so it's putting that code back into main and that's completed so now if we go back into our repo and click on files we'll see that we're in the main branch here and that's the only branch that now exists because the other one's been deleted because the pull request completed and we can see that in progress now if we click on that we can see it's the terraform plan stage that is running so our plan build pipeline is running so let's go and check out what's going on over there so we go into that terraform plan and as we see it's already all the way pretty much all the way through this so it's done the same thing downloading the secrets installed terraform done the terraform plan and it's adding that same resource it's done the archives are creating a zip folder and it's published that um that artifact to the pipelines so that it can be used in other jobs so we'll wait for that to finish we can see that here and monitor it from here and as you can see that's just taking like around 50 seconds so i wouldn't expect it to take much longer than a minute two minutes obviously that would depend on the size of your terraform code um you know the more you have the more more it's going to have to do but you know it's pretty quick because we're using the linux build agents as well so that's now completed now if i go into the releases you'll see that i've got a pending approval now i would have again i'm going to get an email message saying i need to go and do this i'm going to review this approval but if i come into this release one that is waiting for me to approve and i can see it's waiting for my approval because that's what we set on the approval gate and we can see that it's come from this artifact 28 from the terraform plan stage and if i wanted to go and look at that and if i was reviewing this as a reviewer i would go okay it's come from here what is this number 28 let's go and have a look at that click into that and i can see okay it came from this build job from pull request eight and i could now now go back and look at pull request eight if i wanted to um but as it's already approved i shouldn't have to worry about that and i can come in here look at tf terraform plan and go okay oh we're adding a resource group that's all fine and just to prove that resource group still doesn't exist at that stage at this stage and if we went back into here and we went back and back uh and went into our back onto the other tab for our release pipeline and we say okay it's still waiting for me here okay and i can say yeah it looks good to me let's do it i could defer it to a later time if i wanted to but i won't and say approve so now that's going to go and cue this job to run on one of the hosted agents and hopefully deploy our resource group to azure in a second so let's have a look at what's going on here let's have a look at the actual job itself so downloading the secrets again downloading the artifact uh downloading those secrets now installing terraform extracted the files doing a terraform on it and now it's doing a terraform apply and hopefully well there you go it's finished perfect okey-doke so if i go back into azure and do a refresh and uh wait for arm to to update and wake up we should see my resource group appearing here in a second obviously there is a a little bit of a delay sometimes on on this portal so we need you to wait let's do a full refresh and see if it will appear and as we can see uh the the full refresh did the trick and we've got our test resource group here and that's been created in north europe based on the subscription that we linked the provider to in our terraform code so all good that's all worked all managed from a devops pipeline and now you know how to build that yourself with all the approval matrixes so i hope it's been useful leave some notes in the comments feel free to tweet me and i'll leave some useful resources on the blog post as well thanks for watching [Music] you

Channel: Jack Tracey

Views: 8,835

Rating: 4.9766083 out of 5

Keywords:

Id: AWXOYS-SBfY

Length: 92min 7sec (5527 seconds)

Published: Sun Nov 22 2020