Importing Infrastructure into Terraform

it's uh it's a nice infrastructure you got going there so uh yeah i'm gonna go ahead and need you to rebuild that entire infrastructure using terraform and yeah i i'm gonna need you to write your own instruction manual for that all right great thanks bye now hey what's up everybody it's ned bellevant's ned1313 on twitter and hey welcome to terraform tuesday it's tuesday and it's going to be a good time with terraform we're going to be talking about the method by which you can import existing resources into terraform and it's not all sunshine and rainbows unfortunately it's well it's a little more difficult than i might want it to be but we'll walk through what you need to do so at least you understand the concepts and what you would have to put in place if you do have some of that existing infrastructure some of that intricate manually crafted stuff that now needs to be put under management of terraform if that's the thing you need to do then this is the video for you that rhymed that was fun okay so before we get into that two quick things one as always i have my terraform certified guide available on lean pub if you're thinking about taking that associate cert well check out my exam guide and it'll help you pass i've gotten a lot of feedback from people saying yes this did in fact help them pass it was a good way to study and brush up on key terraform concepts before they went in there's a link down in the description definitely check it out number two this is kind of exciting so i have brought back the daily check-in in pawn in podcast only format so previously this was a youtube slash podcast short 10 minute thing that i was doing it was sort of the precursor that eventually turned into terraform tuesday and a bunch of other things that i did i brought that back in a podcast only form it's under 10 minutes if you're interested in that there'll also be a link down in the description it's just kind of my thoughts on a general topic and it's relatively brief so if that's of interest to you definitely check that out now like i said the thing we're talking about today is terraform import and let me get some of the foundational stuff out of the way first the idea behind terraform import is you have existing resources in one of your clouds or even on-prem and right now you're not managing those resources using a terraform configuration rather you're just managing them through something else or manually you'd like to take those resources and put them under management of terraform which means that terraform is now responsible for the life cycle of those resources the way in which you do that is by using the terraform import command but there's a little bit more to it and i think it would be instructive to first look at what actually happens when you're using a terraform config to deploy infrastructure and then we can take that and sort of reverse engineer it into what if the infrastructure's already there and and i just need to push it into a terraform config the bad news is there's no automated tool for that yet but we'll get to that in a minute so let's take a look at what's actually going on when you have a terraform config and you want to use it to deploy resources all right so in a typical situation where you already have a terraform configuration and you want to apply that configuration to a target environment let's say we're building up an azure v-net what happens there well the config gets fed into terraform and terraform interprets that configuration it also inspects the state file if that exists but let's say this is brand new we don't have a state file yet once it's done inspecting that state file or realizing that there isn't one it's going to go out and tell azure in this case to go provision these resources it's going to use the provider plugin we've talked about providers before in a previous video it's going to use that provider plugin to talk to the azure api and tell it to go create these resources and let me know when you're done when it's done terraform is going to write that information to a state file what is in that state file well the state file lists out every resource that you have in your configuration and then once a resource is created in the target environment it has the id the unique id that refers to the actual object being managed in your target environment in this case it would be an azure v-net with one or more subnets okay so that is basically what's happening when you do a terraform deploy let's take a more concrete look at this through vs code and a terraform configuration file all right here we are in visual studio code i have the terraform tuesday files open and these are always available on my github account there's a repository on there a link should pop up right about now with where you could go get those files if you want to follow along and the link is down in the description so you can just click on it if you don't feel like typey typey typing it now you'll notice that i've started changing the folder naming a little bit because as suggested by a commentator it's really hard for you to understand what each folder is about if it's just a date so i started adding a little more description in this case we're talking about importing infrastructure so it has the date so you know which terraform tuesday corresponds to but it also has the topic importing infrastructure okay so now we're good now let's take a look at the folder structure here i've got a complex import example and a simple import example and we're going to do the simple import in this one now i have a terraform subfolder in here and let's go ahead and expand that out i also have a readme file in here which will sort of walk you through what i'm about to do and i'm actually going to maybe copy and paste some of these commands we'll see what happens now the first thing i want to do is actually deploy this infrastructure using terraform so let's take a look at what's in the main.tf file in the main.tf file if we scroll down to resources don't worry about the variables right now we are creating a resource group we're creating a virtual network and then we're using a count loop to create two subnets inside that virtual network the count is being set to the length of the var.subnet names and subnet names is a list of subnet names so however many are in there is however many it's going to create and each one also needs an address space so if we scroll up into the variables subnet names are then two names of the subnet we have subnet 1 and subnet 2 and subnet prefixes are the address spaces for those subnets so that's what we're going to start by creating so let's bring up the command prompt here and i'm actually going to be doing this in windows subsystem for linux because i'm going to do some bash stuff it's a little bit easier to do it from there the first thing i'll ask you to do is to log into azure so you're going to do that you already have to have the azure cli installed do a z login and because i'm using windows subsystem for linux i need to open up a separate browser and paste that code into that separate browser it'll ask me which account i want to use i'm doing this in a separate window and then once i've done that successfully it should return control back to me with a list of subscriptions that i have available now i want to select a specific subscription so i'll go ahead and copy this text here and paste it down here and i'll set it to my subscription called mass i don't remember what that stands for but it's called mass now all right now that we're logged into azure we have the capability to interact with azure that's one of the ways you can deal with the provider so i'm going ahead and grab this terraform init command and paste it down here we'll initialize our terraform configuration get that azure rm provider downloaded and then we're going to run terraform apply auto approve now one of the things that you'll note is once we start running this it's going to create a state file in the directory where our terraform configuration lives so if we look over to the left there's now a terraform.tf state file and there's a lock file that lets terraform know that an instance of terraform is currently operating on this state file so don't try to change it that's what the lock file is all about now this might take a moment to provision the virtual network and the resource group and the subnets so let me go ahead and time lapse to when it's completed all right excellent it has completed our four resources the two subnets the one virtual network and the resource group let's take a look at what's in our terraform dot tf state file now i know you're not supposed to go in and manually edit this so please don't do that but you can go in and see the general structure of the file and we can see we have a line in here called resources and resources is a has a list as the objects that are inside resources the the key resources corresponds to a list of values in that list of values is each resource that was created from the configuration file so if we look at the first entry that's in here and scroll down a little bit we can see that the type is azure rm resource group the name of the resource group was v-net and that's the name that we refer to it within the context of the configuration if we go back to main and scroll down to our resource azure rm resource group we can see we gave it the name v-net to refer to it within the context of terraform okay cool we know that now let's look down at instances so this is instances of this resource group resource and because there's only one there's only going to be one instance in here and the id under attributes refers to the resource id in azure that corresponds to this resource group so it's got the subscription the subscription number the resource group tag and then the name of the resource group that is created it's called tacos all right so we're cooking with gas now if we scroll down a little bit more we get to the next object which is azure rm subnet and in this case we actually have more than one instance so now under our instance it says index key and it's index key 0 this is the first instance of 2 and under attributes if we scroll down we can see there's an id associated with with this subnet resource starting to get an idea of what's going on here basically what the state file is is a mapping between the resource id or the general unique id of that resource wherever it lives and the address of the resource in the terraform configuration so that address is made up of the type of resource in this case would be azure rm underscore subnet and then the name of that resource in this case subnets and then because there's multiples we would do square brackets and then the number zero or one of that resource and that would be the address to this resource inside the state file okay now what happens when you want to import something well before we get to that let's go ahead and tear down what we just created by using terraform destroy auto approve all right so in the background we're going to go ahead and let that all get torn down and let's bounce back to the slides so i can give you kind of an idea of what happens in an import operation when you have an import operation you already have the infrastructure there it already exists so let's assume that somehow our v-net already exists in the resource group and it has two subnets this whole thing already exists somewhere now our terraform state does not yet exist we don't have a terraform state file what needs to happen during the import process is we have to first create a terraform configuration that maps exactly to the resources we want to bring under terraform management that's the bad news terraform doesn't do this for you you have to write this configuration yourself now for something as common as a v-net with a couple subnets that's not super difficult right and we actually already have this configuration ready to go but if you're working with a more complex environment you're going to have to create a more complex configuration the good news is you don't have to import everything right away you can do it piecemeal first maybe doing the virtual network and then adding the subnets and then adding your network security groups and then maybe adding virtual machines you can do this progressively it doesn't have to be an all-in-one kind of thing but once you have that configuration now you're going to need to get those resource ids from azure somehow or whatever your target environment is give those to terraform and then also pull in the configuration you wrote into terraform terraform is going to take those two things merge them together and write that information to a state file and create the structure that we already saw where it's the resource within terraform and then the id under attributes of where that object exists in the real world now before we get into the command let's go back to visual studio code and get a v-net deployed but this time we're going to use the azure cli and an arm template to do it okay good our resources have been destroyed so now we're ready to redeploy and this time we're going to do it using an arm template so first let's go up one directory and down into our arm directory that's where the template is azuredeploy.json so this is an arm template that deploys exactly what was in our terraform configuration except it doesn't create the resource group inside the arm template instead we're going to do that first so we're already logged into azure we don't have to go through that again we're going to set a couple variables here we're going to call the resource group name tacos and set the location to east u.s and this is the same as what was set in the terraform configuration so that's all lining up now we're going to create that resource group through an imperative command az create the name of the resource group and the location we want it to be created in go ahead and copy that now paste it down here and next we're going to run the command az deployment group create we give the deployment a name we give it the resource group where we want to deploy this template and then the template file itself now what's in that template file i'm not going to go through the whole thing i'll open it briefly but basically if we scroll down to resources we have a virtual network here and the way that subnets are created in an arm template is they're actually a sub resource of the v-net so they're created within as a list inside the v-net so here's our two subnets that we're creating and that's it and then i have some outputs here that will be very useful in a moment so that's everything that's in the deployment template it's super simple right let's go back to where we were and we're going to go ahead and kick off this group create using that template now that's obviously going to take a moment so let's go back to the presentation and i'll walk you through what you need to do to import resources into terraform the command we're going to use to import those resources is terraform import and it has two arguments that go with it one is the address and the other one is the id and i always get these two confused and sometimes i look at the help to figure it out but basically the address portion that addr that is where is this resource in the terraform configuration i need you to give me the full address of that resource in your configuration and that could be in a module or it could be directly in the root module either way you need to be able to give that address and the second thing it needs is the id the unique id of where this resource exists on the provider in the case of anything to do with azure that's going to be the resource id now we're going to get that information from our arm template as an output let's bounce back to visual studio code and see if our deployment has completed yet all right good our deployment has completed successfully that's great the next thing we're going to do is we're going to query this deployment for its outputs so we're going to use the command az deployment group show the name of the deployment the resource group where that deployment exists and then we're going to use jq was a json query to query for properties.outputs and i want all of the outputs and the value of each of those outputs now remember if we look at our deployment in the outputs i have the resource group id the vnet id the and each of the subnet ids those were my outputs and i'm going to need those to import them into terraform so i'm going to go ahead and run this command here copy it and go ahead and paste it down here and that should spit out the resource ids for each of those objects awesome all right we are we're getting there we're very close the next step is to import that into our existing infrastructure so let's go back to our terraform directory we'll go up one directory and into terraform there we go okay and i'm going to go ahead and delete the dot terraform directory out of here we're going to get rid of anything that's stored in there and i'm also going to delete the tf state file and the backup file so we're starting basically from scratch here we're starting from the ground floor we have nothing deployed okay so we're going to have to run terraform in it again we'll go ahead and do that first all right so that'll initialize everything that we need and next we're going to set a variable for each of the things we want to import based off of the settings based off of the output that we got from that previous command so i'm actually going to open up my readme file directly and scroll up a little bit here and i'm going to paste each of these values into the readme just for now so i can run the commands so we've got our resource group we've got our first subnet we've got our second subnet and then lastly we've got the virtual network itself so go ahead and copy all four of those there we go and then i'm going to go ahead and run these four so now they are listed as variables awesome that's all we needed to do let's go back to the preview of readme and the next thing we're going to do is run this command and it's terraform import just like we saw in the presentation the address of the resource group is the type azure rm underscore resource group underscore group and the name dot v-net and then we're going to point it to the resource id of that resource group which we've stored in this variable so i'm going to go ahead and run this command now and if we look to the left we'll see terraform is going to create a tf state file and it's going to add that entry to the tf state file once it's finished let's take a look at this new state file that was created okay the import was successful if we look at the state file under our resources we now have an azure rm resource group we have the name v-net and under attributes we can see the id corresponds to the resource id that we just submitted awesome all right so let's go ahead and run through the rest of the commands we're going to run import on the v-net itself so i'll go ahead and copy that and paste it down here now the subnets are a little tricky because we used account loop to create them so the way that you're going to reference those is after the type and the name we're going to have our square brackets because we know it's a list of resources and we're going to specify 0 for the first one and one for the second one we know that 0 is going to be subnet 1 and 1 is going to be subnet 2. so let's go ahead and copy this well we can just copy both commands that's probably easier we'll go ahead and copy these two commands and we'll go ahead and paste them down here once those commands are complete we've now imported everything that we had deployed and it's now under management of terraform so we can make changes we can destroy resources if we want it's up to us because it's all managed under terraform now if we go and look at the terraform.tf state file and go ahead and scroll down here we can see there's the subnet there's the instances we've got index key 0 which corresponds to the proper id and subnet 1 and if we scroll down some more we get index key one and this is subnet two all right everything looks good in the state file now the real test is if we go ahead and run a terraform plan in theory because we've properly imported everything terraform plan should come back and say no change is needed you've imported everything successfully i don't have to alter anything here so let's go ahead and wait for the plan to come back and let us know if any changes are necessary to our infrastructure awesome no changes infrastructure's up to date so we successfully did an import of this v-net that we deployed with an arm template into terraform now obviously this is a very manual way of doing things and if you have a lot more infrastructure that you need to import you might want to try to script out some of this information like being able to grab all those resource ids from azure or grab it from aws or whatever provider using being able to grab all those dynamically and then line them up with the configuration i'm sure there's some cool scripting magic out there that you could do to properly import resources once you've done it it is done now everything's under management of terraform but i have to admit it's a little more clumsy than i might want it to be now if we go back and look at the files a little bit more you can see this was the simple import i've also got a more complex import and this is a bit of a challenge for you if you're interested if you want to dive in further and try to get this working i have an arm template in here that deploys a virtual network two subnets a virtual machine a nic on that virtual machine in each subnet and network security groups and then i started building the terraform configuration that you would use to import all that stuff does it work i don't know i haven't tested it to be honest so your challenge if you choose to accept it is to actually go through this process deploy out what's in the azure deploy and then try to get it to import based off of a configuration that's in this terraform directory and like i said i haven't tested this i'm not certain it's going to work but that's a challenge for you that's a fun thing you can do to get a little more comfortable with the terraform import command so that's all i have for today hopefully you've gained a better understanding of what terraform import does what you would have to do to import your existing infrastructure and some of the gotchas out there in doing this import process i did mention you can use modules as an import target so you could create and craft standard modules to import your existing infrastructure and maybe make it a little bit easier if you've been enjoying these videos i have a patreon and i've got some fine patrons that have helped support me in making more of these videos if you want to be one of those awesome people there's a link down in the description where you can support me if that's not your cup of tea well you can always subscribe and share and that is also very much appreciated until next time i'm ned bellavance this is terraform tuesday stay healthy stay safe out there bye for now hmm you know it it looks like um it looks like in in step five you you have to put the piece in the jeez everybody's a critic

Channel: Ned in the Cloud

Views: 3,835

Rating: 4.9292035 out of 5

Keywords: Terraform Import, Importing Infrastructure on Terraform, Terraform Tutorial

Id: YYQ6W90ZsY4

Length: 25min 31sec (1531 seconds)

Published: Tue Feb 02 2021