Getting Started with GCP and Terraform

you know i do a lot of videos about terraform and it's usually working with aws or microsoft azure but people have been asking me ned where's the gcp love and in this video i'm going to show you how to get started with the google cloud platform using terraform [Music] [Music] hey what's up everybody it's ned bellevants bellevantsned1313 on twitter and welcome to terraform tuesday today's topic is going to be all about using terraform with the google cloud platform aka gcp but before we talk about that i've got two quick things for you number one my terraform associate certification guide has been updated for terraform 1.0 so if you've been thinking about getting your terraform certification and taking the exam check out my certification guide the link is down in the description the other thing that i want to mention is if you are interested in sponsoring my terraform tuesdays hey i am totally open to that i've got opportunities where you could sponsor an entire episode or you could just do some ads at the beginning or in the middle of the episode so if that's of interest to you please hit me up on twitter you know my dms are open and let me know that you're interested and we will get things started with that out of the way let's talk about using terraform with gcp now you know i've used terraform with a lot of different providers obviously and most commonly i'm using it with microsoft azure or aws because that happens to be the clouds that i tend to work with and also the ones i've been requested to do stuff for but there are a number of viewers that have requested for me to put something together for gcp so that's what this video is about now there's a few things you need to understand about gcp and a few things you need to understand about the provider we'll start with the stuff around gcp in general the first thing you need to know is the way that gcp is organized what it has is a construct called projects any resource that you create inside gcp is going to be part of a project and that project needs to have billing enabled in order for you to create resources now what does that all mean let me try to boil this down if you are most familiar with microsoft azure then a billing account is kind of like a subscription a subscription has some sort of payment tied to it and a billing account has some type of payment tied to it when you create resource groups in a subscription in azure they're associated with that subscription and that's what get gets billed in google when you create a project initially it's not associated with a subscription or billing account so you have to associate it with a billing account so that's one of the things we're going to have to do projects also allow you to do a bunch of other things you can use them as a hierarchy to determine permissions and roles so that is one way just like with a resource group you can set permissions on a resource group and then resources inside that resource group it's a very similar sort of setup you can also create folders that have projects in them and create a hierarchy that way and they all fold up into an organization if you're familiar with azure that would kind of be like a management group and then in your management group you have subscriptions and then your subscriptions you have resource groups so it's a similar sort of tree in gcp you've got organizations and then folders and then you have projects inside those folders so that's the whole sort of comparison between that and microsoft azure that those are the two that i've had to do this with the most so the ones that i really understand but there's also parallels to draw to aws so that's out of the box a couple things you need to understand about the way that google's set up now let's talk about the provider a little bit out of all the providers the google one's kind of the weirdest because there's actually two providers it's not just one there's a regular google provider for all of the services that are in ga and then there's a second provider the google beta provider which is for all the services and resources that are in beta now sometimes there's a version of a service that's in ga and then a different version of the service that's in beta if you want to use both of those you can create a provider for the regular and a provider for the beta and then when you're instantiating that resource or that data source in the provider argument you reference whether you want to use the regular google or the google beta provider and obviously you can use aliases and all that kind of good stuff too but just the main thing takeaway here is that there are two different providers so if you're working with something that's in beta you're going to need to instantiate the beta provider in addition to the regular provider i found that very confusing at first so i just want to get that out of the way right now other things the provider needs well it needs a project to work with and then it will use that project for any resources created within the configuration unless you override the project within the resource configuration so it's going to inherit that project from the provider unless you explicitly change it to something else so that's another thing to bear in mind the last thing that i'll say about the provider is authentication obviously you need to authenticate to be able to access resources through the gcp provider and the way that you authenticate well there's three different ways so let's talk about those when it comes to using the google provider there are three different ways to approach authentication and this actually applies to the google beta provider as well as the regular because the two have basically the same settings in them the first way to do authentication is to use cli based authentication now what does that mean it means if you have the google cloud sdk installed you can run gcloud in it and that configures credentials for your gcloud cli and then you can run a second command that will cache those credentials for terraform to use this is very similar if you're using the azure cli you can do azlogin to log in and then terraform picks up on the fact that those credentials are cached same thing with the aws cli if you run aws configure it will cache those credentials so in the in the same way as those two you can use gcloud init and then another command to cache the credentials for terraform to use and it'll just discover them you don't have to do anything else in the provider configuration so that's the first way that's good for if you're running on a local workstation and you just want to try something out real quick not great if you're running in some sort of ci cd context or if you're running on a machine that's running in google which leads me to the second of two authentication methods the second one is by using machine identity so this would be a situation where you're running some sort of compute instance in the google cloud platform when you're wanting when you're running one of those instances you can give it a service account and that service account can be assigned roles and permissions within gcp what that means is when terraform is running inside of that compute instance in gcp it can discover that service account and use that for performing actions within gcp so that's your second option but that only really works if you're running in a compute instance inside gcp or something in gcp that has a service account the third option here is to create a service account and then a service account key which is a json file and then specify that using either the credentials argument inside the provider or one of three different environment variables that are out there and you can just set the path of the file or the actual file contents in that environment variable and terraform will find it and use it now generally speaking if you're running in a ci cd kind of context that's the option you're going to use you're going to store it in environment variable yes you could store it as a file but really you don't want sensitive data being stored as a file even on your runner machines you'd rather keep it an environment variable you can do that the easiest one to remember is google underscore credentials there are two others but that's the one i tend to use because it's easy to remember and i think it's also the shortest to type and i'm lazy so that's what i do so those are the different authentication types that are available when it comes to gcp the last thing that i want to mention is enabling apis and this is something that is fairly unique to gcp versus azure or aws when you create a new project it has no apis enabled by default apis are what allow you to access the different services that are available inside of gcp so if you want to use the compute engine you have to enable the compute engine api now because none of the apis are enabled by default when you're using terraform you have to either enable those apis via terraform or enable those apis in the project before you try to use terraform because if you try to use terraform the api is not enabled the terraform apply or the terraform plan will fail because it's going to go hey i'm trying to make these api calls and they're all getting rejected in the context of this project so that's another thing to really keep in mind when it comes to using gcp and using the providers you have to enable those apis remember that because otherwise that will come back to haunt you okay i think i've covered everything i want to initially about the provider so let's get into two examples in our first example we are simply going to spin up a compute engine instance inside of gcp but before we do that we have to take a few other steps to create a project enable billing on that project and then enable the api so that we can actually create the compute instance so let's go over to visual studio code now and we will walk through that process okay here we are in visual studio code and i have the directory getting started gcp open in the left if you're looking for these files you can always find them on my github repository called terraform tuesdays the link should appear or be on its way out right now there we go okay and we're first going to look through the readme doc and i put together a pretty good readme doc for this one let's scroll down to the first example and let's see there we go example one okay so we're going to start out using a really simple example if you want to follow along you need to have a gcp account created and you have to have a billing source enabled a billing account enabled and you need to have the google cloud sdk running locally or or you can launch the cloud shell from the console in gcp either one is fine i'm going to be running this locally so let's walk through that we'll start by opening up the terminal and we're going to run this first command here gcloud init now i've already run this but i'm going to run it a second time just so you can see what's going to happen and what it's going to do is it's going to tell me what i'm currently using for my configuration if i already have a configuration which i do and it's going to ask me if i want to update that if i didn't already have a configuration then it would pop open a browser window i could select which google account i want to use and then it will ask me if i want to allow the google cloud sdk permissions to use my account now i don't want to change anything here so i'm just going to cancel out of this but that gives you an idea of what's going on when you do gcloud in it now let's scroll down a little bit more and the first thing that we want to do here is create a project that we're going to use in gcp so i'm going to create a project id which will be taco net dash a random id that gets generated and then i'll run the command gcloud projects create the project id and i'm going to do dash dash set as default so that will become my default project for any future commands that i run so let's go ahead and run those two commands first copy them and paste them down here and it could take a moment for this project to create it's usually pretty fast the next command we're going to run is what caches my credentials and my project information locally so that terraform can discover it so a gcloud auth application default login is what does this so i'm going to go ahead and run that command now so that terraform can find those credentials and use them all right so i'll go ahead and paste that down here and it is going to pop up a url because i'm using wsl if you were running this in a native console it would just pop open the browser window for you i'm going to go ahead and click on that link and it's going to open up in a separate browser which i have open now in a separate monitor so i'll go ahead and select the proper account that i want to use and i'm going to allow the google auth library to use my google account i'll click on allow there it's going to give me a code that i paste in so i'll go ahead and paste that code in now and now i am using that project and it has my credentials cached so that terraform can use it so we're good on that end all right now that the project is created and we've got our credentials cached now remember the next thing we need to do is enable the api that we want to use and we also have to associate billing with this project so first let me go over to the browser there we go and from within billing i'm going to click on my projects and now i should see the taconet project there it is we can see billing is currently disabled for that project we need to enable that before we enable the compute api and try to create something so i'll go ahead and click on change billing here and i will choose my billing account for net in the cloud and click on set account now billing has been enabled for that account let's go back to visual studio code and now we're going to enable the compute engine api by running gcloud services enable compute.googleapis.com so i'll go ahead and grab that copy and paste and paste down below there we go okay that could take a moment for the api to be enabled i found that usually the first api you enable takes a few minutes or a few seconds and then additional apis actually tend to take a lot less time so let's go ahead and jump to when this api has finished enabling itself and we'll get down into running our terraform configuration okay our api has successfully been enabled so let's go ahead and hide the terminal and the next thing we're going to do is take a look at the terraform configuration we're using and then we'll instantiate that terraform configuration against the google cloud project that we've created so the configuration we're looking at is an example one and it's this main.tf file so let's take a look at what's going on here we have our standard terraform block and our required providers i'm sourcing just the google provider here and i'm saying version 3.x so this will keep me in the version three major version of the provider and then i have a few variables two variables defined here the region where i'm going to create resources and then the variable gcp project which is the project where i'm going to create these resources and then within my provider block i defined the region and the project remember the credentials is not necessary here because i have my credentials cached locally next i'm going to query a data source of google compute zones in the region that i'm currently using and i don't have to tell it that because it will inherit that region from the provider and from those available zones i will use one of the zones to create my compute instance scrolling down a little bit more i need a public ip address that i'm going to use for my instance so i'm going to source that it's google compute address type and then scrolling down some more we have our google compute instance i'm going to call it apache because i'm going to install the apache web server on it for zone i'm going to query the google compute zones data source for the first entry in that list and use that as my zone i'm going to add a network tag here called allow http which i'll use in tandem with a firewall rule in a moment to allow traffic on port 80. the machine type is e2 dash micro which is the smallest e2 type which i believe should be free so hopefully it'll be free or basically no cost if you're following along for the boot disk i'm selecting the image ubuntu os cloud and i'm using the 1804 image that they have available scrolling down some more we get into the network interface the network i'm using is the default network that comes with that region and i'm not going to get into gcp networking right now because it's a very complex topic but just be aware this is kind of like the default vpc and we'll leave it at that under access config i am saying that i want to nat its private ip to a public ip address and here's that address that i created with the previous resource google compute address and then lastly i have a metadata startup script so this is basically like user data or custom data it's a script that's going to run when it starts up and i'm passing it what's in the file startup script.sh this is a very basic script that just installs apache replaces the default web page that's pretty much it scrolling down a little bit more here's that firewall rule that i was talking about we're going to create a google compute firewall rule to allow http under allow we're going to say port 80 is allowed and the protocol is tcp target tags is if the traffic is on its way the destination has a tag of allow http which we added to our compute instance then allow that traffic if it doesn't match that target tag then don't allow it and because we're not specifying a source by default the source is anywhere okay and then lastly priority 1000 just gives it higher priority than the default rules that are in the firewall lastly for the output we're going to have the public ip address we're going to output that so that we can connect to the apache server and just test to make sure that it's all running okay so with that out of the way i'll bring the terminal back up we're already in example one so i'm going to go through the standard terraform process and we'll start by doing teraform in it and we're using the local back end here so it's not going to have to initialize a remote back end or anything like that there we go it successfully downloaded the providers and it's initialized you can run terraform validate if you've made any changes and you just want to check and then lastly we're going to run terraform plan in the variables we're going to pass it that project id that we of the project that we just created and we'll output that to a tf plan file so go ahead and run that command now and that should take a moment to generate the plan that we want to run once it's done generating that plan we'll go ahead and run terraform apply looks like it's all good so let's go ahead and grab that and copy it down here and we'll run terraform apply and now it's going to go out and create those resources in the compute engine and the firewall and the address so it's going through its process this is not exciting we've all seen this before so go ahead and jump to when the apply is complete and then we can try using that public ip address to validate that the apache server is running in our project okay the apply process has completed we've got our public ip address here so i'm going to go ahead and go ahead and copy that public iep address and we'll go over to a browser i'm going to go ahead and open up a new tab here we'll go to http con slash and there we go no surprise it's a bunch of tacos so as you can see it has created that compute instance installed apache and put a custom web page in there it's super simple nothing too exciting here let's go ahead and close that and we'll go back to visual studio code and walk through the next example now we just did a lot of work setting up a project before we could even use terraform and maybe you want to avoid all of that work one of the ways that we can do that is by creating a terraform service account that has permissions to create projects and enable apis on that project wouldn't that be nice well that is what example two is going to do we're going to walk through the process of creating a service account for our service account in gcp and then that service account will have permissions to create new projects in our organization so you're going to need a gcp organization to do this which is a little bit of a process to create an organization that's a little bit more than just creating a project but it's definitely worth it if you do want to follow along and this is something you would do in a larger organization in addition to creating that service account and the service account key we're going to use terraform cloud as the back end to run this process because i like to include terraform cloud and i'll show you how you can get that service key in an environment variable that terraform cloud can use so let's go back over to visual studio code and start walking through the process all right now we're going to start walking through the process of creating a service account and what we're going to call an admin project that's going to have the capability of creating additional projects in the same organization and we're going to enable those projects for billing so we don't have to do that either it's pretty slick now this is all based off of a blog post on gcp which i have linked in the readme so definitely take a look at that tutorial if any of this seems confusing i definitely borrowed heavily from that to show you this that being said first thing we're going to do is get our organization id and our billing account id because we're going to use that information later on in our configuration so let's go ahead and run these two commands here org id equals gcloud organizations list i'll output that in json and i'll query the first name and then of course they don't give you just the id it's organization slash id so you have to take the id out of that string so that's what that whole command is doing if you have multiple organizations this is just grabbing the first so you might have to tweak this command to grab the organization that you want to use so i'll go ahead and copy that and bring the terminal back up and i'll run that command that runs pretty quick and now i'm going to get my billing account it's basically the same process and i'm going to run that now and we'll store it in billing accounts like the organization id i'm grabbing the first billing account so if you have multiple billing accounts you might have to tweak what number you're using in this jq statement to grab the correct billing account all right so we have that information now the next thing we're going to do is create an administrative project which will house our service account because service accounts need a project to live in so we're going to create terraform dash admin random number as a project id we'll go ahead and do that first copy there we go and paste i'm good at this i use keyboards and then we're going to create a project with that project id and we're going to create it in the organization we want to use so we have to specify that org id and we'll set it as default so go ahead and grab those three now and run it down below there we go it is creating the project and then the last thing we're going to do is link the billing account to that project id so now that project is enabled for that billing account so i'll go ahead and grab that now there we go and go ahead and paste that down here alright so now that project is enabled for billing ooh it did not like that what didn't it like oh cloud billing quota exceeded i don't know what that is all right add a little snafu there it turned out i had maxed out my quota on billing accounts from doing all these demos so i had to remove it from a few other projects so that i could assign the billing account to this project so a little little bump in the road there but hey you know that's what happens when you're doing demos so anyway we've got the billing account linked to our main project our admin project now we're going to create the service account that terraform cloud will use for its credentials when it wants to interact with gcp so the first thing we're going to do is run gcloud im service account create terraform and call it our terraform cloud account so we'll go ahead and run that first do that down here and that should go very quickly all right that's done and now we are going to create the service account key which will be stored in the project id key dot json file in your home directory and it's going to reference the im account that's in the current project that we're working in so we'll go ahead and run that now and go ahead and paste that down here there we go now we've created that file that's good we have a few policies that we want to bind to our service account the first one is roles viewer this lets it view all existing projects the next one is resource manager project creator this allows the service account to create new projects and then the last one is billing dot user this allows the service account to assign billing to projects hey look what we did there we gave it all the permissions it needs so let's go ahead and grab these three permissions commands and go ahead and paste them down here and once those have completed now our service account has all the permissions it needs now we need to enable some apis for our project so that it can interact with these other projects and also interact with billing and i am so i'm going to go ahead and run these four commands here to enable the necessary apis go ahead and do that and hit enter and now once those have completed we'll be ready to start configuring terraform cloud so while we're waiting for those apis to enable let's go over to terraform cloud and create a workspace for us to use and we're going to call that workspace gcp getting started so i'll go ahead and copy that and switch over to our browser all right here we are in the terraform cloud i'm going to create a new workspace here the type is cli driven workflow because i'm going to be driving it from the cli and i'm going to go ahead and create that gcp getting started workspace all right that has been created the next thing we'll do is create some variables so i'll click on variables and i'm going to create the first variable here let's go back to visual studio code the first variable i'm going to create is called billing account so go ahead and grab billing account here copy it and we'll go back to the browser and i'll paste that in as the key and then for the value let's go back to visual studio code and i've got two echo commands here to echo the current billing account and org id so paste those down here so i can get each of those all right so there's the billing account i'm going to copy the billing account go back and paste that in and i will mark it as sensitive and save that variable the second variable we're going to create is org id so i'll go ahead and copy that we'll add that variable paste it in grab my org id go ahead and paste that in and set it as sensitive now i don't know if either of these values do need to be sensitive but you know it doesn't hurt to set them as sensitive so that's what i've done those are the two variables we're going to need and then lastly for authentication purposes we're going to set an environment variable so i'll go ahead and add the variable here we'll go back to visual studio code and that environment variable we're going to set is called google credentials so i'll go ahead and copy that value and paste it in here all right so that's the key is google credentials terraform will look for that environment variable and use what it finds in there so now we need to specify the contents of our json file now within terraform cloud the environment variables cannot be multi-lined so we have to remove any line breaks from our json document and we can do that by using jq and then we're going to get all of the contents of the json file and we'll do dash c to compact it and remove all of the line breaks so go ahead and run that now copy and paste there we go all right here's the full value and i probably need a little more room to work with it so we'll move this up and i'll go ahead and copy the entire contents there we go copy back here we'll paste it in the browser we're going to mark it as sensitive because this is quite sensitive information and we'll go ahead and save the variable and there we go now we're all set we've got our terraform variables and our environment variables now you might be wondering what's in the terraform configuration let's go check that out all right let me shrink back down the terminal and i'll go ahead and hide it for now because we don't need it and we'll take a look at our terraform configuration that's in the example 2 directory so i'll go ahead and open up the two files that comprise it and i'll close example one not not to confuse you so in our terraform.tf i just defined the required providers that's the same and the remote back end that's using my organization and the workspace that i created okay we're good on that let's go over to main and see what we're doing here we have our two variables that we specified the billing account and the org id and then we also have a variable called region and one called prefix the region is where we're going to create the projects and the other one is the prefix naming for that project we specify our provider is google and specified the region but not the project because we don't need to specify that right now and then we're going to get into a random id for naming and we're going to add the prefix variable to that random id so we can use it to create a project next we're going to create a google project for the compute engine and we're going to do that by creating a google project with the name from the random id with that prefix project id is going to be the same for the billing account we'll give it the billing account uh value and for the org id we'll give it the org id value see pretty simple here nothing complex happening and then because we want to make things easier for ourselves we'll also pre-enable some apis that we'll probably use for this project and the way that we're going to do that is a for each loop inside this resource so we're going to actually create four instances of this resource using a for each loop and then we just have each api we want to enable and that's going to be what goes into the service argument we have each dot key so that's each key that's in our set the project we want to enable it on and whether disable whether or not to disable on destroy that's set to false right now so that's the configuration for the resource to enable all those apis for our project and then lastly we will output the project id because that would be pretty useful as an output that somebody else could query as a data source when they're setting up their terraform configuration leveraging gcp pretty cool huh okay let's go ahead and get this deployed all right so we are now going to deploy this second example using terraform cloud if you aren't already logged into terraform cloud you're going to have to run terraform login from the command line and accept all the things i'm already logged into terraform cloud so i don't need to do that so i'll go ahead and open up the terminal again let's clear out what we've got going on here so we can actually see what we're doing and i'll go ahead and ahead and run terraform in it here go ahead and paste that in initialize our configuration with the remote back end there we go that's done and now i'm simply going to run terraform apply dash auto approve so let me copy that and paste it down here now the way that terraform cloud works is it's going to run a plan first and because i've done auto apply then it will run or auto approve it's going to automatically apply that plan once the plan has been calculated so it's spinning up a dedicated instance for us in terraform cloud to do this plan and apply and then it will spin down that instance when it's done isn't that kind of nice and it manages state for you not bad all right so it's done with its plan it knows what it's going to do now it's going to go through the process of actually creating the project and enabling all those apis now as usual this isn't the most exciting thing to watch so i'll go ahead and skip to the part where this has completed its apply right our application has completed successfully we can see it created a project called taconet 610 and then some other characters so let's go over to the cloud console in gcp and verify that project is there all right here we are in the browser and i'm looking at manage resources and we can see under the projects in my net in the cloud organization there's the terraform admin project where the terraform service account lives and then there's the taconet 610c 5fd which is the project it just created and i also got that as an output so now i can use that in a separate configuration to spin up resources i've created a project creating engine that's kind of nice right all right in those two examples we did two different things we used the cli authentication to authenticate to gcp and spin up a compute engine instance using terraform and then in the second example we used a service account and terraform cloud to spin up a new project with apis already enabled and billing enabled so that's pretty useful if you're going to be spinning out a bunch of projects though do you mind that you have to increase the quota on your billing account if you're going to go over something like 10 associated projects because otherwise you'll get that quota error that i got so hopefully that helps you get started a little bit working with gcp and how you work with some of the resources in it for the next video in the gcp series i'm going to delve into deploying resources in gcp using github actions because that was a request from a viewer so thank you for that request and that will be the next episode in the series all right that's gonna do it for me for today as always thank you so much for watching i really do appreciate it just a couple quick reminders i want to plug my patreon if you've been enjoying these videos and you want to help support all the time that i spend making these videos and demos i'd really appreciate that you can join the fine other patrons that are displayed right now and help me out a little bit i really do appreciate it if you can spare a dime if you can't i also totally understand that too for some people times are tough so if you can't and you just want to subscribe to the channel that's awesome very much appreciated if you want to give this video a like i like that and i like you we're friends yay and then lastly i just want to shout out a fellow youtuber a fellow youtube channel i did a great podcast recording with michael lavanne today and i just wanted to highlight his youtube channel so i will definitely throw a link in the description as well as a pop-up here somewhere to link to his youtube channel he's got a bunch of great content around using cdks to deploy infrastructure's code so definitely worth checking out that'll do it for me for today as always thank you for watching until next time stay healthy stay safe out there bye for now [Music] let me tell you i had some big plans to go to conferences and interview people and do awesome podcast episodes i even went and bought this zoom h6 recorder that can take up to six different inputs so i could go to these conferences and i bought mics and i was all prepped and then the pandemic happened and i didn't get to go to conferences and record awesome podcasts but you know what conferences are happening again and i'm excited i'm going to be out in the field with that recorder if you happen to be at a conference and you want to record an episode with me hey i'll be there let's do it alright bye now

Channel: Ned in the Cloud

Views: 1,886

Keywords: Google Cloud Platform, Terraform Tutorial, Using Terraform with GCP, GCP Tutorial

Id: JQYgFSYFi-o

Length: 36min 58sec (2218 seconds)

Published: Tue Jul 20 2021