Deploying HCP Vault and Consul

console and vault are both ga on hashicorp cloud platform and that is what we're going to investigate today now you may have noticed that i didn't mention terraform in any of that and that's because we're going to be using terraform and terraform cloud to deploy console and vault on hashicorp cloud platform so if you want to know more about that tune in this is going to be a good terraform tuesday [Music] what's up everybody it's ned bellevants ned1313 on twitter and welcome to terraform tuesday we are going to be terraforming it up hopefully you've got your tacos all ready to go today's topic is all about deploying hashicorp cloud platform console and vault using terraform and terraform cloud now i'm throwing a lot of jargon in there so i'm going to cover what those are in a moment before i dig into that just a quick reminder if you want to support this show and get these episodes a week early as well as my weekly newsletter you can do that on patreon so that's how you can do that so check out the link it's down in the description if that's not your thing hey quick reminder i also do a daily podcast or at least during the week it's a daily podcast called the daily check-in and it's like about 10 minutes of just my thoughts so i'm not taking up an hour i'm not taking up 30 minutes this is 10 minutes you know if you're walking the dog or doing some random task you can listen to it and get some interesting tech advice some career advice or just general thoughts on what's going on with me so if that's of interest to you that link is also down in the show notes so definitely check it out give it give it a listen let me know what you think i want to hear it anyway let's talk about what's going on in the world of hashicorp cloud platform okay so what is that hcp what is that let's start with that thing and then we'll work into the other topics right so hcp is basically a managed version of the software that hashicorp provides now the two obvious candidates for that are vault and console because those are two things that you need to set up a cluster of servers to run it's not like terraform you're not setting up a cluster of servers to run terraform generally you're not and even with nomad you might not necessarily be setting up a cluster of servers to run nomad but but vault well you need to set up one or more vault servers to you know offer the vault service same thing with consoles so the idea here is what if instead of setting all that stuff up for yourself what if you could just get it as a service click a button boom you've got vault or boom you've got console and then you can make that service available to all of your applications that you want to integrate good idea that's what hashicorp cloud platform is and right now it is only available on aws now how do you connect to this thing good question the way that you do it is when you go create an instance of hcp you create an instance of a hashicorp virtual network now the implementation detail here is that on aws it's basically a vpc and then they spin up the resources inside that vpc and they manage things like upgrading the version of vault that's running they provision the whole environment for you and you get some options when it comes to sizing how big do you want your vault or console environment do you want to use a development instance or a standard instance and that comes with some caveats as well but the nice thing is that at least on the vault side and i assume this is true of console as well you're getting the enterprise version of the software so those enterprise bells and whistles you might not have with the open source version well you get them so that's that's a nice kicker so you have this hvn hashicorp virtual network that's where your vault and console service resides and then you peer that vpc with other vpcs in your environment or you can wire it into the transit gateway so you have a couple options for getting things to connect you have control over the cider block that's used inside that hvn so you can make sure to pick a cider block that doesn't conflict with any of the other services you might be offering and i have it on good authority that the smallest block you can choose is a slash 25 so you could choose a slash 24 and that gives you a pretty good range of different address spaces you could plumb and i'm sure you could find a slash 24 that you have available in your ipam software for this service so hopefully that works for you if you're in other clouds if you're in azure or gcp and you want to consume this thing you can also enable a public endpoint for either vault or console and make it publicly accessible now the downside to that is there's no firewall controls at the moment so you can't say make it publicly accessible but only from these source public ip addresses if you make it publicly accessible it's publicly accessible that's just the way it is right now there may be a future enhancement where you can control some of those firewall rules i got to imagine at least on the aws side that would be really easy you would simply just enable network acls and security groups for that vpc that the hvn exists inside of and you'd be good to go there's also and this is a little confusing there's a separate azure offering to do console as a service on azure that you can get directly through the azure marketplace that is also managed by hashicorp but you pay for it through azure whereas hcp you pay for it directly to hashicorp it's not part of the amazon marketplace or anything like that so just some points of distinction there it's only available in aws right now but it will be coming to the other clouds as they build out the service so just bear that in mind so how are we going to deploy this thing how are we going to dig into it well there are some pre-requisites that you have to have in place and let me go over those briefly all right those prerequisites that you have to have in place if you wanted to follow along with the demo i'm going to be doing in a few minutes well first of all you need a hashicorp cloud platform account you can sign up for one no problem there'll be a link down in the description on where you go to sign up for that it's free to sign up you can use your github account it's very easy so no big deal there the second thing is you're going to need an aws account obviously since we're deploying this thing in aws you need an aws account so you can provision a vpc and like an ec2 instance to interact with this thing if you don't want to do that you could manually provision a vault or console instance make it publicly accessible and then you don't need aws stuff at all but for my demo i am deploying stuff in aws so you need that account you also need a key pair in the for ec2 instances in the aws region that you plan to deploy in i'm using us east one and i already have a key pair if you don't have that you're going to need a key pair and lastly because this thing is going to use terraform cloud for remote state and execution you're going to need a terraform cloud account which those are also free and when you create an organization in terraform cloud you get up to five free users including yourself and you can sign up for it with github so most of this is free basically is what i'm saying so let's actually go over to the interfaces for hcp and terraform cloud and see what those things look like okay starting with hcp this is the hcp portal that i have logged into right now i don't have anything provisioned so it just takes me to the overview and i can see i can get started with console or vault when you sign up at least right now as of when i'm recording this you get a 50 credit which is plenty if you're only going to be spinning up the development instances i don't remember the exact pricing but it's something like three cents an hour for the development instance of vault and two or three cents an hour for a console so you could run both of those for an entire month and i think the 50 would still cover it and that's assuming you run them for the whole month so that 50 credit is going to go a decent ways down on the left you can see there's the hashicorp virtual network so that's an hvn you can have one or more hvns provisioned and then below that you have the resources of console and vault and then lastly under terraform that will take you to terraform cloud where you can provision all this stuff before we get into that the last thing i want to call out is under identity and access control it has the access control right here and under there is service principles now in order to do this whole thing you're going to need create a service principle here that is the credentials that terraform cloud is going to use to talk to this via the provider and provision the hvn and the console and vault instances so you're going to need to create need to create this service principle ahead of time and then copy down the client id and client secret that it gives you when you create a service principle all right so that's hcp we got that covered on my other tab i've got the hashicorp terraform cloud login going and i am in a workspace that i have helpfully named terraform tuesday hcp and within this i set it up to run from the cli because you can have terraform cloud point at a cli remote execution or you could have it point at some sort of version source control for when it kicks off a new run i'm just doing it from the cli i did have to add some variables in here so if we go under variables here are the variables that i have outlined i've got the key name which i have set to hcp because that's the name of the keys that i created in us east one for aws i also have the client id and client secret again those are the credentials that terraform is going to use to talk to the hcp provider and then below that i've got my aws variables as environment variables and that's the access key and secret key for the region and user that i want to use to create resources on aws so that's the two portals those are the settings that you're going to have to configure in the portals before you go ahead and run through the demo now i've already done all of this so we can go over to visual studio code and start walking through the configuration now just as a brief explanation before we dive into the configuration i have broken things out into modules as much as i can because i have a feeling i'm going to reuse some of this stuff so i did want to get it into modules so i have roughly four modules that are being used and they're all controlled through a main.tf a main route module so we're going to dive into what those are i don't know what level of detail i should really get into as usual all the stuff is available on github in my terraform tuesday repository so i don't need to go into every nuance in detail you can pick it apart if you want but let's go over visual studio code and get an idea of what's going on there okay here we are in visual studio code and the directory that these files are in is 20 21 04 27 that's when i'm publishing this particular video for my patrons and then you'll get it a week later so if you're watching it a week later you know that's why the date is what it is and within there like i said i've got four modules one is the hcp network so that's to create the hvn or the hashicorp virtual network the next one is the hvn aws peering and like i said we are going to pair this hvn with one of our vpcs in aws so that module handles the peering of the vpc with the hvn and then i've got a module to create the vault instance and another one to create the console instance so that's the overall structure here and then i'm also going to be creating a vpc dynamically for the peering so i'm going to be using the vpc module that's available on the public registry so with that in mind there's also a really good readme here so if you want to run through the readme it kind of breaks things down a little bit on what prerequisites you're going to need to follow along and all that jazz let's open up the main.tf and just kind of see what's going on in here okay so like i said we're going to create a vpc and i actually set this up that you could create multiple vpcs because you might want to pair multiple or peer multiple vpcs and consume these services so i have it using a count and it's looking at a variable called vpcs which is a custom object that i'm creating within the variable and we could jump over to that in a moment but let's just look at the structure of the whole thing before we dive into the details so we'll start by creating that vp one or more vpcs using the vpc module from the public terraform registry then we are going to create the hvn the hashicorp virtual network and for that the only inputs that you really need are the region you want to create this in so that's going to be u.s east one the cider block you want to use to create this hbn and remember you don't want this to conflict with any vpcs you want to peer with this hvn and then finally a naming prefix what what do you want to name this hbn so you know what it is and it's basically going to take that prefix value and then add a dash and then a random string of characters so that it's unique within your deployments you can do this a number of times below that we create the peering connection between any vpcs that have been created via that previous module that was further up and the hvn that we created so it's going to create one peering connection for each and as part of that peering connection it's also going to set up the routing from the vpcs to the hvn so on the hvn side you have no insight into that vpc you have to tell it what the the cider blocks are for the vpc you're appearing and it adds it into the route table for the hvn you have to go add those re to the route table on the vpcs to say any traffic bound for the citer block that the hvn is using send it down the pairing connection and so that is handled by this peering module adding those routing connections we also have to add the vpc id the hvn cider block and the hvn id so we need all that information for this peering module once we have the peering relationship set relationship set up we will create an instance of vault and for that all we need is the hvn id where is this vault instance being created do we want it to have a public endpoint and that's set to false by default but you can update that variable to be true and it will give you a public endpoint and then finally a naming prefix it will use the same naming prefix you're using for everything else for consistency console is basically the same you need that hvn id you need to set whether or not you want a public endpoint and you need a naming prefix once you have all that created you actually want to access console and vault and the way that we do that is by spinning up an ec2 instance on one of the public subnets and then making it accessible via ssh so we can connect into it and test out our connectivity with our new vault and console instances so the way that we'll do that is we'll get the latest version of amazon linux and then we will create a security group that allows ssh from anywhere and i know best practice is not to allow ssh from anywhere so please if you're using this for production don't do this you can borrow some of this stuff in the modules but don't do this you know you want to lock production down a little bit more than that and then i'm going to create an ec2 instance and as part of that ec2 instance i am going to you add some user data in here that it will run as a script as the ec2 instance comes up so maybe it makes sense to take a look at what's in that template file so here's the template file that it runs when it starts up what we're going to do is first install all the hashicorp tools using yum once we've done that we're going to install vault console and then make sure jq and curl are installed because those are common tools you're going to use to interact with vault or with console then we're going to bootstrap our console and vault connections so as part of the outputs from the modules it generates an admin vault token for us to use i'm going to write that token out to vault token the file so that we know so that we have a token we can use to connect to vault with and then i'm going to write out the private vault address to a file as well we're also going to generate an admin token for console and so i'm going to write that out to console token now we have ways to connect to all these various things now we're also going to run the console agent on this ec2 instance so we need to create basically some files that we'll use to configure the console agent and get it up and running and those are going to go into the directory slash opt console so we'll make that directory the things we need to run the vault not the fault the console agent we need the certificate authority for the console servers or server that's running in the hbn and we can get that from the outputs of when we create the console instance so we have that we're going to write that out to a file called ca.pem and then we can also get the client config from the outputs of creating a console instance so we'll write that out to clientconfig.json and then we have to replace one value in there changing the path for the ca cert to where it's actually going to be so just a quick command to do that and then lastly how is the agent going to authenticate we need to give it an a token a console token to talk to the console server and so we'll write that out to clientacl.json and we'll add in the console token there and then move all those files from the home directory into slash opt console and we are all set so all that's going to happen when the ec2 instance comes up and then once we ssh into that ec2 instance we'll be able to run some test commands to verify that vault is working and that the console agent runs successfully okay so that's the whole setup why don't we go through the actual deployment okay so let me go ahead and bring up the terminal here and as always we'll go ahead and run tara from terraform in it first i should also mention that i'm using terraform 0.15 you definitely want to be on 0.15 or newer when you do this otherwise i'm not certain that everything that i've done inside these modules is going to work properly the other thing that i want to point out is that the hashicorp hcp provider is 0.5 right now so it's still relatively new they're still adding some functionality to it so i was a little constrained on what i could put into these modules because the provider itself is still being developed to a certain degree now that i've done this i have it set up using a remote backend so it probably makes sense to look at that now under the terraform.tf file in my root module we can see the back end is set as remote i have the organization in there and the workspace it should be using and i ran terraform login to get myself connected with credentials to terraform cloud to do these remote runs okay so that's another thing that you have to do ahead of time since we've initialized successfully and we're all ready with our remote back end we can go ahead and run terraform plan and what's interesting about this if you've never used the terraform cloud platform before it actually runs this remotely so it's sending this these instructions to terraform cloud and then it's going to mirror the output back to the cli here so you can see it's using linux amd64 that's all happening on some remote container or runner or worker in terraform cloud and then it's just mirroring the output back to me on what would happen if it ran this apply after it did the plan so this is purely just a plan it's not actually applying anything right now let's go ahead and wait for it to finish and and there we go it sometimes it pauses at the end before it actually gives you the cli control back but we can see that it's going to add 21 new resources so our plan completed successfully we know it's actually going to deploy some resources so now the next thing that we can do is we'll do apply and we'll just add an auto approve on there so we don't have to wait and we'll go ahead and kick that off now if we go over to the terraform cloud console we can see this remote plan and apply happening in real time it actually gives us a link to do that if i open it there we go it's loading now so this is the remote plan that we just kicked off if you do an apply it's going to run a plan first that's the way it works when it's done the plan if you use the dash auto approve switch it will automatically jump to apply so the plan is currently running and then it's going to get into the apply and this mirrors essentially what we would see in our cli as well so you can really monitor it from either if this had been done through version control instead we would only see what's happening through the console as opposed to at the cli because the change would have been kicked off by some sort of commit or merge on the version control side all right so this is now running the apply the thing you need to know about spinning up these various services is spinning up the vault and console services can take up to 10 minutes and i don't think you want to watch me stare at a screen for the next 10 minutes so through the magic of editing we'll fast forward to that time but before we do that i think it makes sense to just jump back to the hcp console and if we go to the virtual network we can see there's now a virtual network there so it has provisioned that virtual network if we go to vault we can see it is in the process of creating our vault cluster and if we go to console it is in the process of initializing this cluster for whatever reason it takes you right into that cluster view but that cluster is also initializing so those are spinning up like i said you don't want to wait the 10 minutes so we'll fast forward this to the point where this has successfully applied all right our apply finished if we hit the drop down here it gives us some outputs but we don't really need those with the exception of the ec2 public dns if we had created multiple ec2 instances there would be a whole list here which is dependent on how many vpcs you created since we only created one vpc we've got one public ip address to connect to our ec2 instance from so let's go back to visual studio code and actually make the connection from there all right we're back in visual studio code and you can see that the remote run from terraform cloud still gave me the output locally so i can go ahead and use that information so we're going to go ahead and do an ssh and the name of the user is ec2 user so we'll go ahead and put that in and now we need to copy that public ip address go ahead and paste that in here and then we're going to need to refer to the key pair that i used so i'll go ahead and put that in as well when i called that hcp there we go and yes i would like to connect there we go we're connected and if everything ran successfully if we do an ls there's our console token our vault address and our vault token if we cut out say the vault address there we go that's the private address to get to that instance of the vault cluster that's running so important to know that this uses the development instance of both console and vault by default because i want it to be as cheap as possible which also means there's only one actual server behind each one so don't use this in production because it's called development for a reason anyway so what do we do now that we're here well if we look in the readme and scroll down to the bottom i have a few commands that you can run to try out some stuff and we can start by exporting the vault address and vault token values into environment variables so that we can then check on the status of our vault server so i'll go ahead and copy those paste them down here there we go and now if we do vault status it's going to reach out and say hey looks good let me actually move that up a little bit so you can see all the information you can see that it's initialized it's unsealed and that we are connected and it's version 1.7 enterprise so you have all the enterprise features available to you here by default it creates a namespace called admin i believe but you can create additional name spaces so if you've ever wanted to play with vault namespaces here's your opportunity now for console we can run the console agent and remember the configuration is in the directory opt console and that is also what we're going to use for the data directory so we'll go ahead and run sudo console agent and point it at the configuration and also use the same data directory and we'll just put the little ampersand there so it runs in the background i'll go ahead and run that first there we go all right so this will join up to the existing console cluster as an agent you can see adding server and synced node info so we're good it has successfully connected to the console cluster if we want to see the members of our console cluster we can run this command console members and we have to give it a token to authenticate that command with and we can pull that from the console token file so go ahead and highlight that now and we'll go ahead and run that command and we can see we have two members we've got one that is the server and it's running build 195 enterprise so again you get access to all the enterprise features in console and then we have our local console agent that we just started up all part of this console deployment so that's pretty much it that is how you can use terraform to deploy hcp vault and hcp console and get it peered to an aws vpc now as you saw in the demonstration i created vpcs as part of the demo but you probably already have some vpcs you can take the modules that i've already created as part of this and reuse those modules for your own existing vpcs if you do if you like don't want to create one on the fly and this allows you to not only play around with atp but also play around with the enterprise features of vault and console something that i've been wanting to do and especially getting a copy of those enterprise licenses that's pretty pricey now it's just available you can just play with them that's kind of awesome so that's gonna do it for me today thank you so much for watching as always i really appreciate everyone subscribing and liking the videos and giving me all the great feedback the comments are fantastic and just so you know i do read all the comments and even if it takes me a little while i try to respond to them as well so keep the feedback coming keep the requests coming i make this every week and i've got a long list of things to do but if you have a great suggestion please put it in the comments i will try to get it get to it as i do these different terraform tuesdays until next time stay healthy stay safe out there and bye for now [Music] hey wanna know a secret when i do recordings for pluralsight i have this this is a clicker every time i screw up when i'm trying to say something i click and then i try to say it again and every time i click i can see it in the audio waveform and that tells me that i messed something up and that i should delete where i see the click and keep the take that happened after the click so little fun fact for you there bye

Channel: Ned in the Cloud

Views: 455

Keywords: HashiCorp Terraform, Terraform tutorials, HashiCorp Vault, HashiCorp Consul, HashiCorp Cloud Platform, HCP Vault, HCP Consul, Terraform Cloud

Id: tgDT3ltdrek

Length: 29min 19sec (1759 seconds)

Published: Tue May 04 2021