Configuring a KinD Cluster with NGINX Ingress Using Terraform and Helm

hey there nick trinitakis here in this video we're going to go over setting up a local kubernetes cluster using the kind tool which stands for kubernetes and docker it's a great tool to spin up a kubernetes local cluster because it just all runs within docker that means you can run one command and you're going to have a multi-node or a single node cluster up and running in about a minute it also works on native linux wsl2 within linux as well as mac os so in addition to running this kind cluster we're actually going to be doing it through terraform using the kind provider we're going to also set up an nginx ingress controller using helm all managed within terraform so we're going to have a single terraform config file well technically it's broken out into a couple config files but we'll just be able to run a terraform apply to have your cluster up and running ready to go so now you can run whatever services that you'd like on your cluster and have them all working through localhost so that's the agenda for this video i have all the config files here ready to go they will be up on my blog by the time that you watch this video so i'll leave link to that in the description if you want to if you ever want to take a look here at the code so before we get rolling here let me just run this demo script which is basically just going to be running terraform apply and then just spinning up like a little dummy service so we can check things out but this is from ground zero so you know the script doesn't run a terraform init but we can see here that uh if i scroll up here there is no terraform related files uh you know based on state or just installing specific plugins that we need here we're providers so if i run a tera4 minute here we're going to install all the provider plugins that we need here and we're going to go over all this code in a second here i just want to get this out of the way here and also just start running this demo script here and let's take a look here at this demo script while it runs it's really nothing fancy at all all it runs is a terraform reply here with letter approved just so we don't need to type in yes then it's going to go to kinds documentation here they have an example echo web server service basically so we're going to apply that into our cluster and then just curl uh basically the foo path here on localhost just to get a free response because it's basically an echo service you know basically this script is just a one-shot script that we can run to make sure everything is up and running without having to manually install like a test servers or something like that but we can see here this is all working and yeah now it's just running terraform apply we can see we are using the helm resource here to install the nginx ingress controller and we're going to go over all this configuration we can also see here too there's the kind cluster provider this resource is up and running it's uh yeah spinning up a local cluster named demo local this is all good to go you can see it's taking a little bit here this is normal it takes a little bit over a minute here eventually once this completes then it's going to install the nginx ingress controller using helm but yeah let's not just wait for this thing here and let's look at some of the tools that i just spoke about here so this is kind's documentation kubernetes and docker you know you don't need to run this through terraform they have a command line tool that you could install and use instead but my philosophy here is if i'm going to be using terraform in production and maybe like a staging environment on whatever cloud provider i decide to use right this could be amazon with the eks it could be on google cloud maybe digitalocean or linux basically pick your poison right well these cloud providers are really good and a lot of them do have minutes kubernetes but uh you know kind in this case is just meant for me to run a local kubernetes cluster but still you know if i'm going to be using terraform to manage let's say eks and amazon you know i might as well also use terraform to manage kind as well in development and now even when it comes to something like the nginx ingress controller yeah chances are if using a cloud provider you may end up using your cloud provider's uh ingress controller instead like for example on aws there's a load balancer controller which you would use instead of the nginx ingress controller but for locally this combo of kind and nginx ingress controller which is over here the official one from kubernetes really really good combo you can also use this one in production as well but you know if you're on aws at least the aws load balancer controller offers a little bit more perks but maybe that's a video for another day and yeah so terraform again we're not going to go over super basics about terraform here but you know it doesn't have a built-in provider for kind but luckily this author here created one and that's what we're using here author is also super responsive because i noticed in some of the documentation it was a little bit skimpy so there were some things that i just didn't know what to do because it wasn't documented but eventually i figured it out i opened up an issue and then even ended up opening up a pull request just to add a little bit more to their documentation here uh what did i add yeah just basically how to convert some stuff from kinds yama config because again kind does have a command line tool that you can run you can also pass in a config file using yaml actually i think i did a video about using the env substitute command in the past which i'll link to in a card where i actually did spin up a kind cluster using eksctl instead well actually no ekrctl was related to spinning up a cluster on aws never mind but i think there was some video i don't know maybe maybe one over kind in a different video yeah so many videos so many videos but in any case yeah this should all be up and running now cool so yeah this went through ran a terraform reply and eventually got to the point where that cluster was created here and then it is going to install the nginx ingress controller using helm and then i also set up a null resource here to basically just wait for the ingress to be ready and we'll go over all this code in a second here but yeah eventually after that you get to the point where we just add that echo service running into our cluster and then we just curl localhost through here basically this code that just ran over here and then we actually get the response there and it all works so if i actually curl localhost foo here we can just see it's echoing back what we have and if i run a cube ctl get all on everything we can see that the kubernetes cluster is up and running it actually runs two different servers here two services here this one from kind here foo and bar i didn't go to the bar one but that's what this is running here and we can see you know the most important part right is the kubernetes cluster is up and running the nginx ingress controller is running in ingress nginx as the namespace and yeah we are all ready to go so now if you had your custom workload or whatever services that you want with one to run within a local cluster you can now do that but yeah let's start looking at some of this code here for terraform and i'm just going to open up my sidebar just so we can easier look at the stuff so yeah number one we have a couple of different providers that we needed to install so there's that kind provider that we just took a look here on github right we need to install this thing then we want to install the kubernetes official one from hashicorp which is the company who makes terraform they have a provider for kubernetes as well as helm so we're going to use both of those and i really like locking down my versions to the exact version instead of using something like you know the tilde operator like this or whatever i don't know i've just gotten bit so many times in the past where i used like a rough estimation for uh a version number and even just bumping up the patch release to the next one resulting in something breaking so i really like explicitly going all the way down to the exact patch version yeah it's a little bit more work needing to update these things a little more frequently but that is work i'm worth doing because i mean stability especially in a production environment is uh number one priority or at least a very high priority i don't care that it takes me an extra couple minutes a week or something just to like update patch releases just by having to go into a file and like update a version number but yeah so there's also the null resource from hashicorp as well we'll go over that in a bit technically you know i think everything is going to work with an older version of terraform but terraform was one owed not too long ago so you might as well just um make sure that you're running 1.0 or above but i think technically 0.15 and above maybe even 14 should work here but yeah let's start diving into some of this other config file right so we just went into versions there let's maybe take a look here at some variables like certain things that we can configure very easily so in this case we have something like the cluster name so i decided to name it demo local over here then also we can configure where we want our cube config file to be written to so by default this kind cluster provider that we're just looking at over here this one it will write out your cube config file to the current directory of wherever you run terraform and actually if you use the uh kind cli tool it ends up writing it out to your cube config file path here which i kind of prefer because i just don't want to have this file lingering in this current directory you'd have to like get ignore it then because there's like sensitive information there about accessing your cluster uh it basically comes down to preference so in this case and again this is not like an open source module that i created or anything this is just like a one-off thing where yeah this position i'm working at now we're setting up all sorts of different terraform infrastructure including kubernetes and you know part of that for me starts with getting things running locally in development so i wanted to set up a kind cluster but yes so we can configure where this config file gets located totally up to you we can also choose what in ingress nginx version that we'd like to use based on what helm version is available so in this case 4.05 at the time i make in this video is the latest version you can always drop that down to an earlier version you can always go and check out nginx's repo on github here and see some of the releases so if you go to releases here then they have a list ah look at that see that four days ago 4.06 came out but this code was written uh sometime last week so 4.05 was the latest version and actually i think that's a great segue into just like why use helm to manage all this and why not just like apply it directly with cube ctl into your cluster like you totally could now you know this is a development cluster so the need for uptime isn't super critical right like i can just decide to update this to 4.06 which i am actually going to do now and then all i have to do here is terraform apply this and i'm not going to bother doing auto approve just so we can see what's going to change in the cluster so we're basically just doing a live update now of the nginx ingress controller using helm and we can see here terraform is going to let us know by the way we're going to be doing this one change here from 4.05 to 4.06 you know this is going to change cool well the null resources is triggered to always change but i'll get to that one in a bit here but yeah this is just a live update now to nginx ingress controller of 4.06 which right now by default this thing is only configured to have one replica so this is going to have a downtime event where the server is just not going to be responsive where any of your you know the nginx ingress is not going to be responsive for about i don't know maybe like 5 or 10 seconds but yeah in development this isn't a big deal also becomes a non-issue if you decide to just run multiple replicas which you probably would do in production even if you decide to use the ingress nginx controller as well so we can see right now we're just waiting for to be up and running it's all good now and actually if we go back and run a cube ctl get all a you know it is now up and running too for 27 seconds here this is version 4.06 so cool like that made it super easy right just to update a new version of nginx there nice so yeah the last thing here is if you wanted to customize what namespace the nginx ingress controller is going to be in this will actually create it for you because helm has that capability to basically create a namespace if it doesn't exist in my opinion it's a really good idea to dump this into a custom namespace instead of using the actual default kubernetes namespace but uh yeah that's up to you here just a very simple configuration thing here now with all that out of the way let's take a look here at the kind cluster config file first and again you know if you're new to terraform or not a super pro with terraform because i'm definitely not you know i have these things all separated out into different files but you could totally just have all of this in a main.tf file but i prefer to have them broken out because it's a little bit easier to reason about but we can see here we're configuring the kubernetes provider as well as the kind cluster provider this resource here to use that custom config path that we defined inside of our variables and since we are using tilde here if i jump back to the versions one actually no variables one we can see we're using tilde here to be a shortcut to your home directory i actually don't want to cut this file out because it does have some some sort of information in there because i do have other clusters being managed but um yeah going back to here this actual config file we do need to run path expand on that otherwise it's going to like literally write it out to a tilde directory within the current working directory and uh funny enough well not funny enough but again with terraform you just can't run things like path expand inside of the variables so that's why i'm not running path expand here and then just referencing it here uh yeah you need to call them out and there's actual individual config files but takeaway here is this is how you set up a coin cluster using terraform so you know we can pop in the name whatever the cluster name that we like right ours was demo local we can customize optionally where this path is going to be located if you actually didn't have this property in here then it's just going to write it out by default to the current working directory let's wait until everything is running before terraform reports back if it works or not and then we can also customize the kind config file so going back to the documentation for kind here there's all sorts of different configuration that you can add here and this could be done on the command line if you happen to use kind their cli tool here right you can do like kind create cluster and then give it a name but you know infrastructure is called code as is a nice thing so it's nice to have all that's in a terraform config file instead of just like ad hoc running a command in the command line right but you know kind also does have a config file itself like there is a yaml file that you can configure somewhere down here which i did use in the past before i started using terraform but we can see here yeah um this is an example config file that you could set up all sorts of different things that you can do and then um yeah then you can just pass in this config file from the command line but again we're replacing all of that with terraform so instead of writing a yaml config file for kind we're going to have all that configuration in here so we have things like the api version and we can even configure specific things about the control plane because if we go back to the documentation here for ingress to get all this to work yeah you need to do some stuff around setting extra port mapping and basically this configuration that we see here so for the control plane yeah we need to patch like cube adm config patches you need to do this stuff and then extra port maps for like you know if you're going to have your service running over port 80 as well as 443 which i happen to do you know because you just want to be able to go to localhost directly over http or https without customizing a port then yeah you just need to add this configuration in and this configuration literally that we're looking at right here is this configuration here but in terraform like hcl config instead of yml so we can see we have the extra port mappings for 80 and 443 as well as just popping in this custom cube adm config patches here it looks a little bit messy here yeah it needs to be basically inline yaml right now this provider resource doesn't have a way to yeah configure this in a different way i guess you could technically make this like a multi-line string to make it a little bit cleaner to get rid of needing to like you know put everything on one line but in any case that's basically the same config file right and then for the other uh worker node we don't need to configure anything special and um yeah this gets you going completely with kind uh capable of running nginx ingress controller being capable of accessing these uh ports here so we can just go to localhost nice so now let's take a look here at the ingress configuration itself so this one also we are configuring helm kubernetes here to use the same kubernetes config path just so we can access our cluster you know from the cube config file location that we took a look at before and the variables sorry my brain just paused there for a second but anyways yeah this is the official helm release from hashicorp here right so we can just manage our helm stuff here instead of having to run an ad hoc helm command and again this just goes back to it's kind of nice to have everything in terraform configuration so that you can just run a terraform apply instead of having to run something like a kind command to create the cluster then another helm command to install and configure the nginx ingress controller and then maybe another command to this another command to that before you know it you're writing like you know all these shell scripts here or wrappers around tools where you're just running like four different command line tools like but if you have it all defined here in terraform configuration you can just terraform apply and you're good to go the only reason i actually have this demo script over here is because like yeah this is totally separate like custom workload stuff like the only reason this literally exists here is just to demonstrate that the ingress is actually working so like normally this code wouldn't be here at all normally you'd have like your own web service or whatever you're trying to run any cluster right but yeah let's go back to that helm set up here and that is over here so in this case yep we're just gonna grab the official nginx ingress controller just using all the documentation that's applied that is supplied on helm's uh provider so we have this resource here where we have the name and the repository and the chart all this stuff is like just you know standard stuff you'd run on uh the command line using helm's tool but in terraform format and then we can also pop in whatever version that we want right 4.06 in this case we are also going to make sure that it runs in the namespace based on our variable and if that namespace doesn't exist then this is going to create it by default for us which is really nice because if this didn't if this wasn't here then you know this name says namespace would have to exist beforehand then we need to do a little bit of configuration when it comes to configuring the nginx ingress controller to work with kind and that's actually going to be in this yaml file here which we will take a look at in a second but lastly here like depends on so yeah this provider or resource has no idea about the cluster being a dependency of this so like we can't install the nginx ingress controller into a cluster until the cluster actually exists so in this case we're just saying depends on then we have client cluster default here so if i jump back to over here that is this guy over here so basically you know the helm setup just won't happen until this is completely ready to go so that's why we have that depends on there but if we didn't have that then yeah the nginx controller is going to be tried it's going to be installed before the cluster is ready and then we're just going to get continuous errors there so this fixes that entirely which is one of those things like kind of why i'm making this video right it's like um these little like i don't want to say edge cases but things to think about right like even going back to here like this wasn't super obvious because it wasn't even documented at the time of when i did it but now it is but cool so yeah let's go to this values config file here again this is not even super important to go over it but it's very important that you do these things here otherwise kind is just not going to be i don't know i don't know compatible with the nginx ingress controller so if you just helm install the nginx ingress controller without defining these custom yammer properties then the ingress controller is just never going to be up and it's never going to work and it's interesting because if you go to here their actual documentation for the ingress and you scroll down somewhere it actually says to uh if you want to install the nginx ingress controller without using helm they do have this example over here so if we go to this example here and and again you could just like keep ctl apply this one but if you do this yeah there's all sorts of different configuration options here for the nginx ingress i'm not going to go through all of them but it's somewhere in here i'm trying to just if i can just one shot find it really quickly here but yeah there's basically just a whole bunch of different custom properties that we pass into the nginx ingress controller and those custom properties from this yaml file are what we're porting back over here and this was a son of a gun to to get working fortunately i didn't have to like go to that config file and map every custom value over um i did find an issue somewhere in kinds repo where there was a closed issue about it getting working with like helm when you installed the nginx ingress controller and yeah someone basically had this ready to go to copy paste which is amazing because something like this would have taken probably multiple hours just to like you know make sure that every single one of these is set correctly and like which ones do you actually need versus like not needing but it turns out like all of these are necessary and even this one here which is like a little bit funky so when you're dealing with the ml right you can have a boolean defined as true and false and typically you know just as a good habit it's a good idea to quote your strings but in this case this actually needs to be a true value that is quoted so very important there if you don't do this one your nginx ingress controller is not going to come up correctly and things are just not going to work but yeah once we have all this configured and ready to go then uh yeah it is up and running and all nice now there is one last step here to get all this going and that is basically waiting for the ingress controller to be ready so if we go back to our output here and scroll up before we do the upgrade then it actually does take a little bit of time usually i want to say in maybe 20 like 30 seconds or so for this ingress controller to be ready so what i ended up doing there was we're just using local exec inside of this provisioner here for a null resource so basically to my understanding again not a terraform expert yet with no resources you can basically just run some arbitrary code that you like that's not associated to a specific resource here in this case we're just running a local exec yeah just running cube ctl wait and it's going to wait 90 seconds until the nginx controller is ready and in this case this actually would only ever run once like when you first create things well actually let me rewind if you took this provider and you dropped it inside of here which technically is valid syntax then it's only going to run once when you first set up the cluster but we just saw now like we decided to upgrade to helm uh nginx integration controller like 4.06 if we did that then this is actually not going to run because that's like part of doing a helm upgrade not like a helm install or crate so this null resource lets us basically apply this all the time whether or not we're creating something or updating something and we're just triggering it on a unique uuid so you can also trigger this on like you know like a date time or something like that but this is guarantees that this is going to run basically every single time we run a terraform apply and um in this case like i'll just run it again now like it is going to finish very very fast because in this case there's nothing to wait for because you know this time around we're actually not making a change right we're not going to be updating to a new version of the ingress controller here but we can see though that this is going to be triggered and we are going to run that so in this case though you can see it's going to finish like immediately right within like one second and all it's doing yeah cube ctl wait for this specific pod to be ready which is the nginx ingress controller it'll eventually time out after 90 seconds i ran this thing so many different times so many different environments it really never takes longer than 30 seconds and in fact like i didn't just come up with this magically like off the top of my head um it is like a general concept but if we go back to here this is actually literally this command here where we're just going to wait for it to come up and yeah i just dropped it into local exec here this is the command that we're going to run will require you to have cube ctl installed locally which you probably do if you're working with kubernetes anyways but yeah that's that's all uh good to go here and this one like uh you know uh this helm release here depends on the cluster being up well this one depends on the helm release engine x controller to be ready as well so yeah just make sure the ordering of all this stuff happens in the correct order because we want this to happen last after this one finishes up although this one doesn't wait until the ingress controller is actually ready you know there's some web hook that needs to fire or whatever like i don't know the implementation details but i just know that we need to wait until this happens and then we're good to go so like technically right like you can totally have this resource not here like let's remove this from the config file right but then you know we're back to like well now you're gonna need like technically maybe another wrapper script around this one and run your cube ctl like weight command here instead and like why if you can have it all set up in one command where you can just terraform apply it without having to run additional things then that's really handy and it's really important too because if you try to install your service into the cluster before the nginx ingress controller is ready then that is actually going to fail so like this line would fail here if that controller wasn't ready and yeah that's not good right you want to have this stuff to be basically as bulletproof as possible so i think terryform really helps there a lot and um really happy with the setup now you know because prior to this i was just using all these different command line tools in unison and it totally worked but now it's like this is cool this is something i can commit up to github all in one like infrastructure repo or terraform repo or something like that and someone else can clone it down and just up it and run it you know as long as they have like cube ctl installed or whatever and they're good to go so that's going to be it for this video getting all this up and running is yeah a nice little exercise let me know in the comments below or if you're going to do this also let me know if you're using a different tool for local kubernetes development um i did try a couple of them i just figured where i just found that kind was the best one for me at least because just waiting one minute for an entire cluster every up and running was pretty nice and yeah it didn't get any weird networking issues and stuff like that uh yeah with that said is there anything else to cover i don't think so that's everything right here's how you can install cube ctl by the way i'll leave a link to this one in the description if you need it but chances are if you're watching this video you already have it installed yeah thanks a lot for watching if you like the video please give a thumbs up because it helps a lot and on that note i will see you in the next video

Channel: Nick Janetakis

Views: 5,418

Keywords: Kubernetes, kind, terraform, helm, nginx, ingress controller, cluster, docker

Id: Nm2c9xvGMpU

Length: 24min 19sec (1459 seconds)

Published: Tue Oct 19 2021