Day-8/16 Azure DevOps Terraform Pipeline | Azure DevOps Zero to Hero Full Course | CICD Pipeline

hello everyone welcome back to my channel my name is f and this is video number eight in the series Azure devop 0 to hero in this particular video we'll be looking into infrastructure as a code and automating the provisioning of infrastructure with the help of terraform and Azure devops so without wasting any time further let's have a look at the agenda all right so I am in my GitHub repository you should already have the link by now if not you know where to find it and I have scrolled down to the section where it says infrastructure as a code with terraform and Azure devops which is day8 and in this demo we'll be provisioning an Azure virtual machine it includes uh Azure virtual machine vnet subnet Resource Group and these many resources and to authenticate ourself securely we'll also create a service principle and we also have a look at what exactly is infrastructure as a code why do we need it you know um what is terraform and how it works the entire workflow of it we'll create the terraform configuration files for Azure virtual machine and all the resources that we have discussed and we'll execute it manually from the command line and then we'll implement the same from the Azure devop CI and CD pipeline so this is the plan for today and without wasting any time let's head over to the demo and let's start working on it all right so let's start with the basic introduction of infrastructure as a code what exactly it is we need to understand first what challenges does it solve or why do we actually need it right so let's assume you are a developer or someone who's working in an ID organization and your task is to maintain servers infrastructure and all other services in the cloud or on premises right so you've been tasked to create a test environment right it involves a virtual machine it involves a VPC subnet a load balancer and a few other resources right so what you will do in a traditional way you will just log into the console like whatever cloud provider you are using you log into it and you know provision the infrastructure one by one all the resources or you can use you know like CLI commands if it is azure you can use a CLI command if it is gcp you can use g-cloud command if it is AWS you can use AWS CLI commands right or even scripting languages such as you know bash Powershell and so on right so this is when there is a small environment which can be created and which can be destroyed as per the needs and requirements what if you have to provision 50 search environments like that how would you do it and how much time it will take if you go down this route or if you go down that route of scripting it like what would be the complexity of it let's assume there were 10 resources and 50 environments so that makes 500 so you have to provision 500 such Resources by going manually or you have to script the logic for it it would be a tedious task and this is not something that anyone would want to do right and what if you have to make any changes let's say you have 50 virtual machines running what if you have have to make one single change to one of these machines how would you do it with the help of script or if you want to do it manually then also it would be not advisable right so you need something that avoids all this mess you know and automates everything you need some sort of mechanism that takes care of your infrastructure provisioning destruction uh updates and all the management of it it has to be be sorted right so that's why infrastructure as a code comes into the picture right it takes care of all those things all the challenges that we have discussed and it provides you a better way of organizing your infrastructure so basically instead of scripts so you write something called as configuration files you pass different parameters to these files and based on those parameters it will provision your infrastructure make the updates to it and it will destroy the resources if you want right so it makes the management so easy and hasslefree and example of such tools are terraform know and pulumi every cloud provider also have their own set of infrastructure as a code tool such as AWS has cloud formation then Azure has actually Azure has two type of tools uh which which is arm templates and another one is bicep bicep is a recent Edition that was introduced I guess couple of years back and gcp also has deployment manager but now gcp is also moving to terraform right because of the increased popularity in the market because of how companies are adopting it and how easier it is to use and learn so terraform is one of the most popular tool these days in the market maret so this is what we'll be focusing on terraform and we'll be implementing it with the help of azure devops so let's have a look at the terraform workflow how it works actually um you have your terraform core over here you can use terraform cloud or even terraform CLI we'll be using CLI for this demo so this is your terraform core and you write basically terraform configuration files right uh files such as main.tf variables output we'll see everything but these are a few of the files or you can create one single file and name it anything but it has to be ended with the extension. TF this is the format that terraform supports just to avoid the complexity and just to make the code more manageable it is recommended to keep the separate modules and separate files right that's why um we'll be using this format to divide that into multiple files we also have some ADV Concepts in terraform like using modules custom modules and lot of other things but we have a time constraint we cannot learn everything about terraform in just 1 hour or 30 minutes so that is why we'll be covering the most important part so that you can just get started with terraform and you can just try to use it in Azure devops right uh then we have a concept of back end there is a file. TF State and we store that into storage account um so Azure has a storage account this is similar to AWS S3 bucket or if you are familiar with gcp gcp also has a cloud storage bucket right so this is similar storage account within Azure itself so you write these terraform configuration files and these configuration files will provision the infrastructure for you with the help of terraform providers Azure AWS open shift data do kubernetes gcp these are one of the thousands of providers available within terraform so these providers are maintained by the provider itself like AWS provider is maintained by AWS Azure provider is maintained by Microsoft and so on this is not maintained by terraform this is maintained by their vendor company itself right and with the help of these providers it provides you an interface to interact with their services right so let's say you have to provision some virtual machines within Azure right so how would terraform interact with Azure right it will interact with the help of Provider so Azure RM is one of the provider so this is how it interacts with the cloud provider to provision the infrastructure to make the changes and and so on now let's have a look at the workflow how does it work as part of the first step you have already created the terraform configuration files and everything you have created your TF State file and stored it in in the back end now the next step is you would have to run a command called terraform init this init will initialize the provider right here is the provider this will initialize the provider and download the plugins and dependencies in the local directory right if you are working on a Linux machine it will uh download the Linux dependencies for that provider if you're working on a Windows machine it will download the windows plugin and dependencies for that provider and and so on right so terraform in it will be the first step when you are working with it and yes basically yes you have to install the terraform first uh without that you cannot use it so yes first step is always the terraform install and then you initialize the infrastructure so after you have done initializing it the next uh step is to run the terraform plan command terraform plan command what it will do it will compare your actual infrastructure with the expected infrastructure right so let's say you have two virtual machines already existed then in a cloud let's say in Azure and as per your configuration you should have three virtual machines so it will compare these two things it will see that yes it should have three virtual machines but I could only see two virtual machine so it will provision one more virtual machine right so that's how it works so it basically compares the expected and the actual results and how it will compare it Compares that with the help of TF State file you know it it does not actually Compares uh the file with the infrastructure itself it compares the configuration file with the TF State file TF State file will have all the information of your existing infrastructure so this file is really important this is really critical that is why we store it securely in a storage bucket so that it cannot be accidentally deleted or the secret information inside it will not be accidentally leaked outside so we secure it we make sure that versioning is enabled and we make sure that it can be recovered if deleted right so this is really important file so that's what we do as part of The Next Step terraform plan and it will not actually make the changes it will like show you what are the differences like what exactly will be done when you run terraform apply what all changes will be made to the infrastructure it will show you a plan this is a dry run this will not actually Implement those changes right once you have that plan now you can select whether you want to apply those changes or not so if you want to apply those changes what you have to do is you have to run terraform apply Command right and when you run the terraform apply command it will then apply the infrastructure changes with the help of Provider it will provision all the resources that you want or make the changes that you want as per the configuration files over here and then it will also update the TF State file now TF State file will have the latest state of the infrastructure earlier if it it was two virtual machines over here and three in the configuration file now we have three virtual machines over here so it will update the same in TF State file as well now you see the TF State file is matching with the configuration so when you run the terraform apply again without any changes it will not do anything because the actual infrastructure is now equal to the desired infrastructure and as soon as you make any changes in any of these files and apply the changes it will find a difference in the TF State file and the infrastructure and it will go ahead and make the changes right so this is how it works now let's have a look how does our Azure DeVos pipeline fit in this diagram right why do we need it so all the commands that were running we were running it manually we were running it from command line or from terraform Cloud we were running it manually there were a lot of manual interventions even though we are calling it automated provisioning of infrastructure we still have some manual work and we still have dependency on someone to exactly go and make the changes right so that's why we wrap everything all the commands within Azure devops pipelines you know cic CD pipelines and it will make the changes accordingly so whenever you make any changes in the infrastructure in any of those files let's say it will detect the change with the help of the trigger that we have and as soon as it find any change it will trigger the steps within the pipeline ter form inate plan apply and so on and it will make the changes automatically so we don't need any more person to go and manually run those commands right now this is actual automation of infrastructure provisioning end to endend in which we don't need anyone to go and run the commands manually yes there'll be a manual intervention like someone will have to approve the request or someone will have to approve the pull request but but that is is the part of the process other than that it is fully automated okay so I hope you now have gotten some understanding of terraform and we'll get to know it more once we do the actual demonstration so let's head over to the demo now okay so as part of the first step I'll go to my web browser and search for Azure RM terraform right and go to this link which is registry. terraform doio and it has the Azure RM provider documentation go over there and you see this is the Azure provider and first step is to authenticate yourself to Azure like there are several methods to authenticate you know you can use Azure CLI command using aasy login and then you basically enter your personal credentials and login but it has some issues like when you are working for an organization you don't really provision or manage resources by your personal ID right so you need a manage Identity or you need some something called as a service principle which has specific role to perform certain actions right so that is why we'll be using authenticating to Azure using a service principle and a client secret you can choose any of those but let's use this one I'll open this in a new tab the reason I'm showing you the documentation so that you are aware uh like how do you get yourself started with it right so I'll scroll down and this is basically the steps to create a service principle it says first if you are in China or US Government Cloud you have to do you have to run this command but if not like first you start with a login so you have to authenticate yourself as personal user before you actually create anything so I'll go to my vs code I already have the a CLI installed in my system so that is why command will work for me uh if you are facing any issues please make sure you install install it first and then you can run a CLI commands such as a login if I do AZ login it will open uh the browser for me and it will redirect me to authenticate using the Microsoft account so I'll choose my account and it says you have logged into Azure because the credentials were already saved in my browser and now I can close this window and go back and it says I'm authenticated with this particular subscription right and this is the tenant ID you have the subscription ID as well all the details are here right so after that let's go back and follow the documentation now I can check if I'm part of multiple subscription I can check a account list but since I only have uh one so it's fine and then I can also set a account set subscription again if you want to set a default subscription I only have one I don't need to run this then I can run this command to create a service principal with the contributor role right so this command will create the service principle and attach the contributor role to it so that it will have necessary permission to manage the resources that we're going to create right and the scope is at the subscription level make sure you update your subscription ID over here so let me copy the command first and I'll just go back maybe I'll open a new file new text file okay and I'll paste it over here and my subscription ID was from the command that we ran earlier there is a field called ID so this is the subscription ID copy it and let's update it over here okay now I'll copy the command and I'll just run it over here the error that I'm getting is this inv valid schema no connection Raptors for found so this is a git bash specific error there is a work around for it so you just basically have to set this environment variable and I will share the details again in the GitHub repository so set this and run the command again okay now it's creating and now it's created the service principle and these are the details like password app ID tenant ID so store it somewhere safe we will be using it later on let me save it over here okay so now our first step is completed we have now created the service principle now let's go back to the documentation and okay we were over here so we have created it says the command will output five value we have got those five values now we have to do the AY login with the help of service principal so let me copy this command from here okay and it says these values map to the terraform variables like so so client ID is your app ID and client secret is your password and tenant ID is your tenant ID so let me go back I'll paste the command over here okay and let's enter the client ID which is app ID okay I can just export it to a variable but it's okay I can just do it like this because this is a onetime thing okay and then we have client secret which is password okay and then we have tenant ID which is this one so now let me run the command okay now I am logged in with the service principle so this is the user detail right now I'm not logged in with my personal ID okay let's go back okay so this is what we needed to do if we want we can just do a log out as well and now we followed this step we authenticated ourself to assure with the help of service principal go down and let's configure the provider you see there is a sample terraform file it has a virtual Network Resource Group and provider required providers but with we'll keep it separate in different files so let's create first a file for provider so I'll copy this okay and I'll go back create a new file let call it provider. DF yeah I'll set the over here let's create a folder EF code okay and in fact let me open the folder over here EF code okay let me save those uh so that I don't lose it Secrets okay so I have one file provider. DF over here right in that I will paste the content that we have copied from the documentation now it says the provider that we are using is azure RM asure resource manager the source is Hashi Corp Azure RM it is there in the Hashi Corp repository and the version that we are using is 3.0.0 there is this detail over here where it says skip provider registration equal to true if you read the comment it says this is only required when the user service principal or identi running terraform lacks the permission to register the resource provider right but instead of that we'll be using our service provider credentials over here so let me delete this line go back to the documentation and go down scroll down a bit okay after we have seen till here and that's okay go down so you can export the secrets in these environment variables and that can be used or you can place inside the terap configuration itself placing inside it in the terraform configuration is not a good habit so you can you should use asure keyword for that or any other method so let's use this method which is uh exporting it via variables so yeah so we'll we'll use this one okay so I'll just copy it first okay let me open this file only I'll paste it over here and make the changes according accordingly like we have client ID so I'll just copy these commands and run it from our terminal window so that it'll be exported in our current uh bash session okay so I'll just run those commands Okay those are now exported all right so this is our provider dotf now let's go back and and we were here so we did that now now let's copy the rest of the things like Resource Group virtual Network we need actually subnets and aure virtual machines as well so for that let's search it over here virtual machine okay and Azure RM virtual machine here it is go down you can copy the details from here um let's copy it from here and we'll make the changes accordingly and create one more file inside the same folder let's call it main.tf okay let's paste the content over here okay so the first one is a variable we'll create a separate file for variable so let's copy this and cut this let's cut this create a new file and call it variables. TF past it over there save it and let's go back to main.tf try to understand the format of it right so the first starts with resource resource is your resource block and then the name of resource name of resource is azure RM Resource Group so this format is already specified in the documentation we have to use the same format and this name this is the reference name through which will be referencing this resource internally this is not the resource Group name this is a local name given to this resource so that we can reference that inside this configuration I will show you how we can use it then this is the fields name and location so name is v. prefix hyphen resources it is appending this variable with the resources and this variable is already finded in variables. TF so uh the format will be variable block and then the name of the variable then we have a default value you can leave it blank if you want and we can specify it in another file so let's do that I'll just remove it from here okay and let's go back and then we also have location so location let's take it out from here and go back and create a variable for location as well so variable and name of the variable let's give it location right and give it a default value value is EUR bestest or you can choose any value that you want okay let's save it and go back main.tf now over here we can use the reference of that variable VAR dot location right there is one more thing this is over here I have removed the default value over I have specified it so let me take it out from here as well and instead create a new file this file we'll call it terraform do tfar so any file with the extension. tfar will be loaded as the environment variable by default so the format in this file we can specify is as key value pair so you provide the variable name location and the value the other variable that we had was so let's give it a prefix let's call it demo okay so now this value will be demo hyphen resources right let's go down so this is our this block will create a resource Group let's go down we have now resource Azure RM virtual Network internal name is Main and it will be demo hyphen Network address to Space is this one location will be Azure RM Resource Group now we are taking the location from the resource Group itself so how we are referencing the resource Group first with the resource block Azure RM Resource Group then the name of the resource Group which is do example and then the name of the variable which is location so it will go to Azure RM Resource Group then example and then find location field and then it will find the value of location from the variables and it will assign it to this location right if you are following along with me and if you are trying to write this it'll be much easier to understand than just listening to it so I would highly recommend doing the handson you can pause the video and you can do the handson on the site right that will be a good way to learn it the same way we are taking Resource Group name from the resource Group that we have created so Azure RM Resource Group do example and do name name is this one right similarly we are creating Azure RM subnet we are creating Azure network interface IP configuration so we also have some more variables Azure RM subnet it's taking it from there and we'll see if we are using any other variable but for now it is just referencing the internal variables then uh Azure RM virtual machine this is our virtual machine with the prefix with the name as demo hyphen VM and it will have location Resource Group VM size then we have few other details related to storage disk everything is hardcoded over here but instead of that we use variables but I have already shown you how to use variables so that's okay for now right and then we have tax environment as staging so that's okay so now we have created main.tf provider. TF terraform TF was and variables. TF the other file that we have to create is backend. TF otherwise what will happen is in fact let me show you what will happen so first first let's do a terraform in it let me create the alas TF equal to terraform so this will help us save some keystroke now I don't have to write terraform every time I can just reference it with TF alas so let's write TF in it to initialize the provider right so as I've mentioned it is now initializing the provider plugins it is finding the latest version of azure RM installing the Azure RM version 3.8.0 so it is initializing the back end with all the files that we have the provider information it will get from the provider. TF file and all other details so it will load all the files with TF and TFR extension right TFR extension is a separate file which is just use as to Source the environment variables right okay after a few seconds it says the initialization has been successful now if you look at our code repository it has created a folder terraform you expand this it has terraform provider Azure rm. exe because I am running this on a Windows machine so it has downloaded the windows based plug-in for that particular provider version right and it also created a terraform log file and yeah that's it basically Bally so when we run the terraform commands it will also create a cachier inside the same terraform folder so make sure you put those files in git ignore like terraform folder terraform do log that so basically anything which is not with the extension. TF you should put that in G ignore even TFR files because this is specific to your project you don't want it to be uploaded to GitHub repository right so we ran the first command terraform in it it initialize the provider now let's run TF format this will format the file if there is some indentation issues right it will make those changes EF fmt so it shows two files that means it corrected those two files automatically let's clear the screen and run TF validate this will validate if the file has if any of these files have any syntactical errors let's run this okay it says configuration is valid yeah ideally it should not because we copied it from the provider documentation itself now let's run terraform plan and see what does it show okay it says insufficient feature block at least one feature blocks are required that means we have to go back to our provider. TF and I guess earlier I forgot to save this provider file that is why it downloaded the latest provider version but uh now it is asking me to upgrade the provider so we can either you know upgrade it using this command or we can change the value over here to 3.8.0 83.0 so anything is fine let me upgrade it okay it will initialize the back end again and it will download uh the correct version of azure RM provider which is 3.0.0 instead of 3. 83.0 which it downloaded earlier okay it did that now run TF plan okay now let's see what does it show it says plan five to add 0o to change Zero to destroy that means after running terraform apply it will create five resources it will make changes to zero resources and it will destroy zero resources right so this is what the plan will show that is why I called it as dry run if you want to see which resources it will create you can just scroll up you know and you will see all the details over here and or you can just run a simple command this and then pipe crab and will be created this string should show us what all resources will be created so let's see okay now it is showing it will create a network interface a resource Group subnet virtual machine and a virtual Network so these are all the resources that we have specified in our terraform configuration file now let's go ahead we have validated everything now let's go ahead clear the screen and run terraform apply okay hit enter it will prompt us and it will ask us whether you want to continue or not so we can uh use that option or we can add another flag which is hyphen iph Auto approve it will skip that prompt um so now let's okay let's do that no over here and cancel this operation and use this command instead terraform apply hyph hyphen Auto approve and this will basically skip that prompt and it will apply all the changes okay so it says it is now creating the resources it will take a couple of minutes so I'll just pause the video okay so it did not take much time and it created all the resources it says fly complete resource 5 added so let's head over to our Azure portal and see if it actually did anything or not okay so I'll just go to my assure account go to Resource groups and here is a resource Group so it is created demo resources okay and inside that Resource Group there are four resources one virtual Network one Nic one VM and one storage disk and the fifth resource was Resource Group itself so we have all our five resources created inside the same location and with the same naming convention every configuration is same so now if we have to make any changes we can just make the changes in terraform configuration and apply it and the changes will be applied over here some changes are destructive in nature for example if I change that location of the VM in the main.tf itself it will recreate the resource but it will change the location eventually but it will recreate the resource for that so that means it was destructive in nature some operations for example if I'm just updating tags of a resource then it does not really uh recreate the resource it will just apply the changes instantly right so let's go back now that our resources are completed um because we did not initialize any backend so it created a terraform dotf State file over here right and if you see that it will have a lot of details it will have secrets subscription ID um it will have a lot of details that we don't want to expose plus we have to secure this file we have we have to make sure that it is not corrupted or it is not accidentally deleted right let let's say if I delete this file from here now I have lost the access to my environment from terraform if I delete the file and run the terraform apply again it will not find any resources because the terraform State file is basically blank it will try to create new resources or it will do something that it is not supposed to do right so it will just corrupt the environment it will be hassle for us to import all the resources and create the configur ation again and which is what we don't want that is why we have to initialize it with the back end now let's see how do we do that so first we have to go to Azure portal and create a new resource let's use the same Resource Group we have to create a storage bucket so search for Azure storage okay go to storage account so let's create a new storage account create okay select the subscription uh Resource Group let me use the same Resource Group storage account name it has to be unique check tutorials with P 101 it cannot have Hyun so let's just keep it this way okay and region let me select the same region which is West Europe okay select the performance standard is fine I don't want any redundancy is just for demo okay select next uh enable storage account access key that is fine networking for now just enable it from all network but ideally you should select this option or from a private access point okay so everything looks okay and yeah other thing that we have to do is we have to enable the versioning for blobs because we want to avoid any accidental deletion of our uh TF State file so if if we enable the versioning it will maintain all the versions even if it is deleted we can recover it from the previous version right so hit review and okay it is still validating it hit create okay it says deployment completed let's go to Resource now we have to create a storage container inside that to host the TF State files so go down where it says data storage select this option containers and it by default created one container let's create another one by clicking over here give it a name let's call it broad DF state it did not like the underscore so let's use hyphen and uh select the access level as private okay hit create okay so this is our container and this container will have our file now let's uh go over here in the browser and search for Azure RM backend terraform and again go to this link uh developer Hashi corp.com it will have a sample backend configuration file so this is what we need let's copy it and create a new file call it backend. TF paste it over here now Resource Group name uh Resource Group in which um the bucket resides so which is demo hyphen resources we cannot use variables in this file so that is why we have to hardcode it I'm not sure if it is a limitation or if it is as per the feature implementation but this is what it is now let's put the storage account name storage account name let me copy it from here it was Tech tuto with P this one how did I forget that okay container name is Broad hyphen TF State okay and key is the file name that you want for TF State file right so let's keep it default let's hit save and now let's run TF init again because we have to initialize the back end again hit TF in it and now now it says it is initializing the back endend earlier it just downloaded the provider details and other things now it is initializing and now it says do you want to copy existing state to the new back end it has detected that even though we have a remote backend but your current TF State file is in the local backend which is in the same directory so do you want to migrate it to a remote backend yes obviously that's why we are doing it so hit yes and it says successfully configured the back end and it has initialized it right now let's clear the screen and now let's see it has created a backup of TF State and let's see the TF State file now you see it is blank now because it has migrated over there and it is blank so if we go back and this was the container go inside that you see there is a file that's been created just now prod. terraform dotf State now every time time I run terraform plan or apply from here it will remotely refer to that file instead of referring the local backend file we don't have anything in backend it will refer it from there and it will make the changes accordingly so it don't have to download the file even right so this is the benefit of using a backend now we have verified everything everything looks good and we have all the files created we have an output. TF file in which we can specify if we want to Output anything so let's say if we want after the completion of these resources maybe just provide the name of the resource or name of the tag as an output so in that case we use output. TF but it's okay for now um let's not use it and this is fine when we are done with our work when we are you know let's say we have done with the demo or we have done with the work for which the VM was provisioned so so we can just run TF destroy hyph hyphen Auto approve and it will destroy all the resources created with terraform how it will know that what all resources were created with the help of terraform TF State file so TF State file will show you that okay five resources were created from this configuration so it will delete all those uh now it is showing uh so five resource to destroy this is what it is doing and it is destroying it okay so the resources have been destroyed after a couple of minutes but I guess my bash terminal is frozen so let me open a new terminal get bash okay and I'll just create one more file which is G ignore so in the same folder let's create do kit ignore okay and add these files over here because I don't want terraform folder and uh lock file TF State there is a backup file I don't want the these files to be uploaded to the GitHub repository including the TFR files so let me add that as well star. tfvs and that's add this one as well secrets. txt even though I will delete it after this demo but let's just do that save so now let's push it to a remote G repository so that you can also use the repository as the sample repository if you want initialize the Repository commit first commit now select this we only have one branch okay let's get remote add remote add from GitHub this is the GitHub name is origin okay remote has been added now let's publish the branch says try pulling it first first so yeah let's pull it pull from origin main yes try pulling first get pull origin main now let's force it with this flag allow merging unrelated histories enter okay it's been pulled now we should be able to publish the branch okay it says the branch has been published let's go back and hit refresh so yes uh our terraform configuration is over here I will add the details in this Rao so that you'll get the idea how do we work with it okay I'll update this repository URL as well uh wherever it is needed okay so I guess you have understood the basic concepts of terraform how does it work how to create the terraform configuration file what is the workflow and how do we run it and manage it so now after this we'll be implementing the same logic the same files in Azure devop cicd pipelines so let's head over to it but before that if you're are getting value out of it if you are learning something from this video I would really appreciate if you gives it a thumbs up and subscribe the channel if you are new here okay let's head over to Azure devops I'm sure this diagram might look scary at the beginning at this moment but trust me if you have understood the concepts of terraform and Azure devop cicd pipeline this will be just a piece of cake for you so let me walk you through a diagram first and then when we implement it then you should be able to understand the end to endend flow you should be able to implement it by yourself you can follow along with me and you can pause the video whenever required or you can do it after you understand it whatever is convenient for you but trust me it will be a simple implementation if you have understood all the concepts that we have discussed till now till this video okay so let me walk you through it so the flow is same you know uh developers check in the code with the help of an ID create a poll request it gets merged and approved uh in an Azure repos git branch and all the dataform infrastructure file will resides in this particular repository itself it triggers the azured build pipeline CI Pipeline and it has below tasks first it will get the source that means it will download all the terraform configuration files then it will do the terraform install on the agent then it will do terraform in it again on the same agent then few other commands terraform validate terraform format terraform plan terraform archive so we are not applying the changes over here in this we are doing till plan and it will show you how many resources will be created modified or deleted right it will just show you the dry run what will happen when you run terraform apply so it will show you all those details and once that is done we can just archive that file into a zip folder and then publish this archive to be consumed by the release pipeline so this process is same what we have done so far it just the steps are terraform specific steps once that is done it will trigger the aure release pipeline which will download the artifacts first right and then it will trigger the first stage the first stage is the deployer stage right deploy stage uh the first task of that stage would be to download the artifacts from here right because these are the separate agents so it will download the artifacts it will extract the file it will again run TF install terraform in it and terraform apply we are not running plan at this moment because we have already uh did that while the build stage so whoever approved the release so there could be a you know quality gate over here or a manual appr approval here so manual approval can go and download the build or checks in the build like what resources are being changed in this particular build and based on that that approver can approve or reject right once this is approved it will trigger all these stages and then finally terraform apply which will provision the terraform resources so resources that it has as Resource Group aszure vnet asure virtual machine um Network interface card and then there is one uh subnet as well right these five resources will be created and it will be done with the help of a backend so there is a back end so there is a TF State file already stored in the back end bucket and that it will refer right once that is done you know let's say they could be multiple use cases after that or using this considering just one use case at this moment was once that is done let's say this was a test environment Provisions for some testers right or QA once they do all the changes when once they test everything out the environment needs to be destroyed right otherwise it will just accumulate the cost so for that we have added one approval step before the next stage which is destroy stage and let's say whenever the work is completed the approval will approve the changes and then it will perform these steps to destroy the infrastructure right so whatever has been created over here will be destroyed with the help of destroy stage and these resources will be then destroyed so it will be a total cleanup you don't have to manually clean up anything that's one of the use case the other use case would be you know along with um along with the test environment provisioning it will have another stage to deploy the code AS as well then followed by the Destroyer stage and there could be many more cases so there is no right way or wrong way of doing it it's just based on the specific requirement and based on how it works in a particular company um you can implement it right yeah I guess we have covered everything now let's go ahead to the Azure devops portal and let's apply everything so I'm in my Azure devops portal and let's go ahead and create a new project let's call it day8 and we are doing let's call it terraform day8 terraform hit create okay the project is created let's go ahead to repos and import the repository that we have pushed to GitHub so scroll down click on import let me get the URL GitHub so all the URLs everything will be there in the day8 folder of the repository so do not worry about about it okay I have copied it let's go here paste it and then click import okay this will import the terraform configuration file it just it does not have the TFR files so that's what we have to create so I'll just copy it from my local directory I guess this was the one so let me change the location to Canada Central I don't want it to be created in West Europe now if you are worried about the azure resources cost in this particular uh demonstration do not worry about it because you can use Azure VM that is part of free tier as your virtual network is free Nic is free and then Resource Group is free the storage bucket is chargeable but if you will be done in like within an hour or two there shouldn't be any cost associated with it right so let me create a new file call it terraform dotf bars okay I'll paste the content over over here and hit commit now let me set up the build pipeline go to repos and set up build so build pipeline can be created from pipeline section itself or from here like if I want to create build for this repo I can just click over here as well setup build it's all the services within Azure devops they are lightly integrated with each other and this is so easy to use okay and let's start with starter pipeline because we'll be creating it from scratch so let's delete it and I will copy the first few lines from my previous project CU this is what we have been doing for now um project was that's day seven okay pipeline add it okay so I will copy till here okay and past paste it over here uh let's add the first task so if we go back what we were planning to do first we have to get the source okay so that is done automatically in a CI Pipeline and in the CD pipeline itself so don't worry about it then go ahead with TF install so go back and on the right side where it says show assistance click over there and search for terraform if you don't don't find this in the search history then I'll show you what you have to do you just open this in a new tab okay and go to organization settings from here okay and go to extensions so this is nothing but a Marketplace extension that we are using and okay so I've already installed it that is why it is visible over here but if it's not there what you can do is you can go over here browse Marketplace and you can search any extension from here let's say if you want to use sonar Cube which we'll be using in one of the later videos so can search over here and get it free and it'll be installed in your organization so this is at the organization level if it installed in the organization it will be available throughout all the projects right similarly you can install terraform extension from here right I have already installed it so let's go back um yeah so select this task and we need to do terraform in it so provider is aure RM command is in it and let's use the configuration directory as system. default workking directory we'll be using some system variables in this demo so I will also share this with you whatever variables we have used so system. default working directory yes so this is the default directory where an agent executes the task right so that's what we'll be using so configuration directory is this we don't need any additional arguments now we need to add the Azure RM backend configuration so for that click over here and select your subscription right if you have not created the service connection yet it will ask you to authorize and create the service connection for you which will have permission to deploy resources to Prov your infrastructure in azzure like what we have done in previous uh videos as well so it will take few seconds okay it's done then once the service connection is created it will ask you the backend configuration so what we can do is you go to your Azure portal okay go to storage account and let's create a storage account for hosting the backend resource right so I will click over here create okay Resource Group let's create a new Resource Group grou let's use the same that we have specified in the backend. TF so I'll go back me open this repos in a new tab okay backend. TF so our Resource Group name should be this okay make sure the configuration matches otherwise it will not be able to find the storage bucket okay demo resources storage account name should be this and location it does not have the location so container name location let's use the same one that we have used for other Resources Canada Central okay Canada Central and standard I don't want any redundancy okay that's good review and create okay it's done go to source and go down to container section over here and then create a new container container name again should match with the backend bucket so this was over there and hit create so our container is also created so our back end is ready okay now let's go back over here and in this uh step over here let's select the resource Group that we have created or it's not visible because we have just created it so let me add this task again okay let's search terraform okay so terraform in it default working directory this should be your service connection Resource Group now it's visible search for demo resources this was the resource Group for back end let's search the storage account and the container that we have created now the key key is the name of the file let's take it from backend which is prod. terraform dotf state so this is the name of TF State file hit add so this task has been added over here now we can add the display name as well so where the task is there let's select display name and there is an indentation issue so I'll go over here and hit a tab it's fine now so let's give it a name and we are doing terraform init so let's call it TF init okay now select another task so we have done terraform initialization now the next is terraform validate so search terraform again and select and validate let's keep the default configuration click add okay so this is added let's add the display name as well so TF validate okay let's add another task the next task is TF format so Terra form it doesn't have it so you can select the custom command as well select custom and command is fmt Okay outputs to console that's okay as your subscription the same service connection click add okay let's add the display name and call it DF format let's go down okay so validate format those are all done now terraform plan terraform and then select plan okay select the service connection add the task add the display name plan okay after plan it was archive and publish so this should be archive build archive files okay so let's change this root path from build. binaries directory to build. sources directory again uh this will be over here build. sources directory is where the agent check out the source code by default once you do this by your your own you will get used to it you will get to know what these directories are and why do we use it so you will get uh habitual of it and it's a good practice to use the pre-build director so that you don't have to hardcode anything in the path or in any of the variables so this is what we'll be using over here build. Source directory and archive type is zip now archive files to be created it has to be created inside artifact staging d Dory now this directory is where all the archives will be stored by default right all the paths will be expanded in the logs itself but over here we'll be using the environment variables okay so this is the main directory then the build would be created with build. build ID dotzip so build do random number and then dotzip right so build ID is again an environment variable so it will be created in inside that there'll be a zip folder with that name okay so that's everything and then hit add so now we have added the task to Archive files now we have archived the files the next step is okay let me add the display name first display name archive files all right the next is publish the artifacts right so search it over here publish build artifacts this one okay so path to publish let's use the default path artifact staging directory and we'll customize the artifact name so let's take this one from here so the zip file name that was there and then we'll also append other string build along with it so artifact name would be let's say 42 hyphen build or one hyphen build something like that and before that I missed a dollar sign so that's good now publish location let's keep it default and let me put my cursor over here okay and hit add okay so this completes our build pipeline let's verify if I have covered everything so we have get Source terraform install in it validate format plan archive and publish okay so we have covered everything but I guess um yeah so if we go back in that form plan we can add one more command one more parameter to it so let's go over here and let's see which variable we can use we can use command options I guess this one and then enter this which says hyphen out I'll have to put that in single inverted commas so which says hyphen out build Source directory again the same directory and and create a TF plan file so when we run terraform plan command it generates the output on the screen right these many resources will be created modified or deleted the all the details about it but we can output it to a file as well so that we can reference this later okay so in this build what all resources were created or deleted or changed that's why it's a good habit to include this terraform plan in the build artifact itself okay so everything looks good now let's let's hit save and run okay let's go to this it says the pipeline needs permission to access a resource before this can continue so click on that and let's assign it a permission this is basically the permission for the service connection okay let's hit permit and it should trigger the build let's hope it does not fail okay meanwhile it is doing it let me show you the logs so it did the checkout okay then it ran TF in it and it says initialize the back end it initialized the remote back end that we used then it perform terraform validate and it says configuration is valid it did TF format and then TF plan and if we scroll down it will show that okay five resources to add zero to change Z to destroy so this is what we needed we have five resources in the main.tf so you see see what it did we have five resources in main.tf and TF State file was not being created because we are running it for the first time so it compared these five Resources with this state file and find out that we need to create these resources now TF state will have five resources and this will have five resources once we run terraform apply so TF State file makes sure that we have no discrepancy between the desired and actual state of the infrastructure right and then it archives the file let me zoom in a little bit so this is the default artifact directory right uh home vsts work 1 a so this is a temporary location and then name of the zip that we have specified it has all the files that we have uh used it by default added the dogit folder as well and terraform folder as well but you can exclude it with the help of a uh command within the pil plan step itself then public build artifact it publishes the build artifact and created one file from this folder artifact staging directory to a container so this is the container name says 83 hyphen build 83 is the build ID and then hyphen build so yeah it publishes the artifact now the next is we have to create the release pipeline so that we can apply those changes so let's head over to releases and create a new pipeline now select the empty job stage one is deployment okay before that let's add the artifact and sources build um this is okay and then we have repository version is latest Source alas is starts with hyphen and this is build definition environment variable so we have used this variable also click add okay so let's add the trigger so that it should be released automatically I will enable the release trigger and let's add the default Branch it's okay okay now go to deployment task okay stages deployment run on agent and select asure pipeline agent specification me select UB 2 and this step will by default download the artifacts okay let's add other steps so now again same way search for terraform first we'll do the terraform installation so terraform tool installer this task okay I guess we did not install it for build I guess it's part of the agent itself but it's okay you can install it and anyways right so we can install the latest version over here because some agent we are using ub2 agent it might have already the terraform version installed but if we are using some other agent it might not have so it's a good practice to install it beforehand so we have this agent has already downloaded the files the build artifact now we need to extract it so let's add one more step extract files okay and let's drag it before this let me add the file pattern over here which is this one so build definition name which is what we saw already then build do build ID 83 it was something hyphen build so 83 hyphen build and then there'll be a zip folder inside that and then a destination folder let's use this one over here which says system. default working direct we'll be using this folder for most of the cases because this is where all the task runs by the agent so file will be extracted after this let's add another step for terraform init terraform init so select terraform and should be after installation so in it change the display name tform in it and it'll be running in the default working directory let's select the service connection and these are the backend details as well so Resource Group was demo resources and then select the storage account container and key name was I always forget brad. terraform dotf State this is the key okay terraform init is done now ER of form apply we don't need to run plan now because we have already verified it so terraform apply and let's change the command over here as well okay and apply in that we have to add hyphen ien Auto approv so that it will not wait for the prompt we are not running it in interactive mode so hyphen iPhone Auto approve and then select the service connection okay should should be good so we have extract file installation in it apply now I have saved it let's go back let's rename the pipeline first PF CD okay hit save okay go back and let's add one more stage for the destruction right so we can add it from here or we can reuse the task that we have used inside it and clone this this so there'll be an option over here so if you hover your mouse it says clone or add so let me clone it added the replica of it so let's go ahead and make the changes let's call it destroy stage okay and go inside the tasks instead of apply just change it to destroy that's the only change we have to make rest steps will be the same so change this as well and Destroy should be also Auto approv so let's hit save okay so our pipeline is almost ready let's add an approval step before the Destroyer stage okay so pre deployment approval select this user I'll assign it to myself and it says time out in 30 days let's select it for 2 days which is more realistic timeout days right and that's it close this and hit save now we have completed everything that's that's what we discussed over here we also have our backend ready we have plan ready we have already executed it let me make a simple change in the git repo and let it trigger end to end right from CI to CD so let's go to repos okay let's see what change can we make let's make the change in in read me file okay let's open this hit addit my first change okay hit commit commit okay now let's go to pipeline okay it triggered it already so let's go and see what it's doing in the logs if there are no changes in the configuration file it will not do anything it will just I'll show you what it will show so if we run TF plan it will still show five resources to add but if we would have applied the resources already it would have shown over here zero to change Zero to create right so it created another build archived the file and this time let's see let's go down publish build artifact the build number is 84 okay build number is 84 over here and let's see if it triggered the release pipeline as well because release pipeline trigger was on this build yes it did trigger it you see the deployment is in progress and it failed as well so let's have a look why it got failed okay it says failed in the terraform apply it says the directory contains no terraform configuration why is that so let's see so maybe we are running it from a different directory other than where we have extracted it so let's see let's see it initialized it that's okay then it downloads the artifact downloaded the artifact to R1 a folder now download it and then extracted extracting file so the same file that it downloaded it is extracting basically the same file so that is also okay and it is extracting this in this folder which is default archive directory and then it ran it doesn't matter where it R terraform because basically it's in the same folder same basically it's running on the same agent so that's okay then terraform in it so terraform in it is saying that the directory has no configuration file that means this is not running from the right directory it says it initialize an empty directory which it should not so let's go to the edit pipeline again go to this so it extracted correctly ly destination folder is system default working directory okay I'll copy this and terraform in it is also running in the same system. default working directory maybe I have to add the single colons I'm not sure let let's try this I guess I figured out what the issue is actually the issue is in build pipeline so you see over here I was I spent almost an hour debugging this so you see over here when it is archiving the files it is archiving the folder along with it so s is the temporary folder that it is archiving and over here as well it says archiving file s so this is not right so let's go ahead and edit the build pipeline let's click on edit Pipeline and we have to go down to the archive step over here and include root folder let's call it false we don't want to include the root folder although it doesn't matter but let's uh add this as well because it's asking root folder or files so IDE it should not uh treat this as a file okay let's hit save and let's hope this time it is successful like this was already successful but it will trigger the release pipeline as well so let's see uh what does it do this time so let me pause the video for a brief moment okay this time it archived it correctly if we see it says found 12 files and over here as well it is actually adding the file instead of that folder so this looks good let's go to releases and yes it did trigger the release let's go to that and let's go to logs so it initialized it it is downloading the artifact okay and installing terraform terraform in it and now it initialized the back end you see over here it says initializing the back end and it found the file this time okay so now it initialized the back end correctly and if we see terraform apply it is applying the changes it says 5 to add and it is actually creating the resources if you see over here so it will take around a minute let me pause the recording till then so yes this time it was finally completed you see terraform apply made the changes now let's head over to our Azure portal and go to Resource groups let's Okay so demo RG this was the resource Group name and it has those four resources fifth resource is the resource Group itself so yeah that's what we wanted to test and that's what we uh were able to now let's do the final step which is destroy the resources now that we have created it so if we go back to the pipeline it should have already triggered the destroy stage it's just it will not proceed further until I go and approve it because we have added the predeployment approval so let's hit over here approve yes yes destroy and then hit approve okay now now it has cued it and it has gone ahead and it will destroy the resources okay let's go over here oh it says zero destroyed okay so this one also failed uh like it actually did not fail it just uh the terraform it did not find the configuration in this directory did we miss any step over here so we extracted the files okay in the same folder home vsts work r1a so this is the same folder that we have extracted the files to and after that we installed it we ran in it and in it also did not find did it take the previous build build 85 and let's see what other stage took yeah so it's the same build I've compared the configuration of both the stages and the only difference I could find is the missing single codes and the slash sign so let's try to add it I'm not sure if it will make any difference but let's do that so over here and Flash and same for Destroyer as well okay let's hit save and this time I will run just this stage so create the release and run destroy strage only create go to release logs looks like it is initializing the back end and it is taking few seconds so I guess it is destroying the resources now yes okay so if you see over here right it is updating the logs and it found the resources it found the configuration files five resources to destroy so it is destroying the resources and now it will be destroyed so I'm not worrying about it now it should be done U within 2 3 minutes or so so yeah that's it I guess uh I've spent past 4 hours recording this video because there were lot of issues that I had to fix now I'll be spending another four five hours editing it and hopefully I should be able to publish it as per a schedule so I will see you in few hours with the video thank you so much for watching the video I hope this video was somewhat helpful in understanding the concepts and you were able to implement the solution by yourself by following along this video if you have any issues feel free to join our Discord community and feel free to post your question or your query in The Help Azure devops Channel and someone will help you out including myself and you can also Post in the comment section below and I will try to reply as soon as possible if you found any value in the video give it a thumbs up and subscribe the channel if you are new here and I will see you tomorrow with the next video take care and happy learning

Channel: Tech Tutorials with Piyush

Views: 16,244

Keywords: azure devops terraform pipeline, azure devops tutorial, azure devops pipeline, azure devops full course, azure devops terraform iac, azure devops terraform pipeline yaml, azure devops and terraform, azure devops with terraform, azure devops with terraform and stages, azure devops and terraform tutorial, azure devops terraform tutorial, azure devops ci cd, azure devops zero to hero, azure pipelines, azure devops, azure tutorial, azure devops tutorial for beginners, terraform

Id: wq_sn9ey4KM

Length: 84min 42sec (5082 seconds)

Published: Sun Dec 03 2023