Kustomize - How to Simplify Kubernetes Configuration Management

we cannot do that with pure kubernetes manifest application running and customized with customers go go go go go go i'm starting to like it more assuming that you are already using kubernetes you almost certainly wrote a bunch of yammer files you're applying them with cube cuttle apply or running similar commands you know how that works let's assume that that's the case and if you do you almost certainly discover that you cannot manage your system based only on kubernetes yellow files the problem is that if you use only kubernetes xml files you must create a huge number of copies of the same things because you have variations whatever you're deploying to let's say staging is not exactly the same as in production and if you're using third-party software you're almost certainly going to customize how that software behaves for example you might have ingress host being different in one environment compared to the other or to begin with to have english host because it is not defined as a base manifest or set of manifests in the third party application similarly you might have a different number of replicas of a deployment in let's say staging then production or you might want to add horizontal for the autoscaler in one environment but not in another because it's small and so on and so forth the list can go on the point here is that you will almost certainly have some variations between environments and your software will not run in exactly the same way as it is designed by some third party vendor so you need to start applying changes and i will assume that you don't want to repeat yourself you want to apply the dry don't repeat yourself logic right otherwise you will end up with copying and pasting files over and over again and not being able to manage them effectively so we need a way to customize to introduce variations between different environments and from the base third-party software manifest to whatever you're doing and so on and so forth so we need a way to apply variations between different environments and also to customize how third-party applications behave when running in our system as i said before we cannot do that with pure kubernetes manifest or we can but then we need to make infinite number of copies and that's an answer that's madness we're not going to do that so how can we apply kubernetes manifest but also be able to customize them to template them or to do whatever is appropriate way to introduce variations between environments so helm solved the problem that i'm describing by introducing templates but we are not going to use him we are not going to use templates because we might not need templates and actually in some cases templates might not be the best choice or maybe in all cases anyways this is not the comparison now between helm and customize uh if you want me to do that i definitely can just let me know in the comment section this is about applying customize which allows us to accomplish similar results as with helm but without templates by going purim pure kubernetes definitions no templates same results or maybe better but you will have to wait until the end of the video to figure out whether that's better or not here's a hint it is better in some cases maybe not all so what is customize as i said before it allows us to customize to modify kubernetes symbol definitions to suit our specific needs in specific environments it works directly with cube cattle so you can use it without installing anything additionally it's part of cube cattle or you can install customized cli and use it as a separate binary in today's demo i will use a binary customize binary but remember you can do everything without that binary or almost everything but today i'm using the separate customize cli binary and the most important thing the most important takeaway for all this i will be using only and exclusively yaml nothing else i might introduce some json there just for the sake of variety but it's going to be pure kubernetes yaml or json no templates nothing special nothing extra so what did i do before i started recording this video before i start running the demo i created the kubernetes cluster in my case it is mini cube you can use any kubernetes cluster and i installed the nginx ingress you don't have to use nginx to use customize that's absolutely not a requirement but the demo the manifest that i will be deploying today use ingress nginx and genex ingress so if you want to follow along a kubernetes cluster and nginx ingress are the requirements and i already installed actually i had it from before customized cli the link to the installation instructions is in the description section so go and check it out i'll start by cloning a repository that contains all the manifests some of those that we will use and some that are not really relevant uh for this demo so the first step clone the repository with the manifest and then we are going to explore customize starting from very basic point and then progressing slowly towards something really really awesome so git clone https by the way the gist with all the commands is in the comment section so don't try to pause the video and then figure out what i'm typing if you're following along just copy and paste from the gist it's in the description of this video so what did i say github.com and then we farsig that's my repo and argo cd production don't be confused that it says argo cd production we are not using argo cd today uh we are using something else but this is just the repo where i happen to have some manifest i want to show you so cloning the repo and then i'm going to go inside of that repository and now let's see what do we have there right ls-1 for example so there are some directories with applications this is mostly about targo cd but again this is about customers just repo contains samara city manifests we have argo cd events workflows i have a couple of applications that i recently installed defined here i'm going to use argo workflows as a demonstration of application that we might want to apply to our cluster using customize so let's see what's inside that's ls-1 argo workflows now inside of that we have bayes and overlays for now imagine that overlays doesn't exist i will explain what overlays is uh for now just remember the directory structure can be basically whatever we want but there is certain naming convention that many people apply and that the convention is that base contains the base manifest the manifest that are always applied and overlays are different environments or different clusters we're going to get to overlays later for now let's focus on base so let me go inside of that directory that i showed you argo workflows and let's take a look at what's inside bayes famous directory and it has most of those things are typical kubernetes not typical this is all pure kubernetes except the only exception is customization jumble uh i will explain what customization yaml is in a second but here we have some config some ingress uh some namespace r back you know i will not even show you what's inside of those yaml files because this is not a demonstration of how kubernetes works this is just typical kubernetes yama files there's nothing special about them so let's take a look at what's inside base customization.yaml it's pure yaml right from kubernetes perspective this is yet another kubernetes uh kind right so you have deployments and services and what's or not this is kind customization so it's yet another kubernetes resource and what it specified is the list of definitions that we want to apply to our cluster if we choose to apply this specific customization so we are here referencing namespace yaml and english yama those are easy to explain they are manifest that are in my repository but the second third and fourth are a bit stranger let's say so let me fill you in with the background story this is argo workflows application that is fully defined in uh repositories so fargo itself uh so i don't want to replicate all that i don't want to copy and paste all the definitions of a third-party application in this case argo workflows so what i'm doing actually and my goal here is to use whatever is defined in as official manifests of that application so those are those three and then customize it to suit my specific needs and in this case on top of the base definitions the project community led and maintain manifests i am using a few of my own which is namespace yaml and ingress yamo in other words i'm combining something manifests stored in github.com argo project argo repository uh and combining them with my own and this is very important customize allows us that allows us to combine not only manifests in the same repository but manifests gathered from different repositories in summary this is a combination of the manifest defined in argo workflows project plus my own those that i'm using to customize the behavior the default behavior of argo workflows and then i have patches strategic merge this is important one and i'm going to show you what that file is for now please note that this will patch an existing resource defined in somewhere in resources and we're going to see how patching works and finally we have the namespace i want to deploy this in a namespace called argo now let me open this git repository argo repository and see what's inside remember for example here we are referencing uh argo repository and then manifests and then directory base let's see how is that defined in in the repo itself so here's my browser i'm going to open that project and we're going to go to the directory what was it manifest yes here we are manifests and then from customers we saw that one of the references is to the base and then there is customization here as well so each directory referenced by customers must have customization it can we can reference a specific file and then it's just kubernetes manifest or a directory and then in that directory there must be customization jumble that defines what are the resources used from that directory and how are they used and here you can see that this customization further references other directories with customization yaml and so on and so forth so you can define and branch into as many directions as you want now the important one here that i want to show you is workflow controller so let's go there and see what what's inside and why is that one important so workflow controller inside of that workflow controller there is customization as always but the one i'm interested now is this one workflow controller copy map and i will explain in a second why so here is the definition this is just a coughing map it's basically an empty config map and this is where i had a problem i mean imaginary problem it's not a really issue and that is that i want that config map to have certain entries before i apply to the cluster so i do not want to use this resource in its default shape i want to customize it so remember this coughing map called workflow controller config map it will become important in a second so now if i go back to my customization yaml you can see that i have dispatches strategic merge this is how i'm customizing that config map that i showed you the original one and it's referencing configurable so let's take a look at what that config yaml is so cut what was it base config right and this is how it looks like so here i'm not defining the whole copy map even though actually the the original is not bigger than this what i'm doing is the time telling customers hey i want you to find the config map that is called workflow controller configma and then whatever is there keep it keep it as is i don't want you to replace that coffin map this is not the full definition what i want you to do is to find data and then add config entry with those in this case with those values so if there is already config i want you to overwrite it so we are telling customers hey this is the kind of the resource this is the name and then patch it with whatever is defined the rest of this manifest whatever is in the rest of this manifest i want you to apply it as a patch to that resource keep things as they are just patch that specific part so you can think of custom ways as a patching mechanism while helm would be a templating mechanism let's build this customization see what we're going to get so customize and then we want to build and we just need to specify the directory uh that contains the the top of the tree let's say that we want to build and this is the base directory that i already explored it might take a couple of moments it tends to be slow when it needs to evaluate many different files from different github repositories but here we are we got the result right this is a combination of all the manifests some are remote in some other git repository some are local and but let's take a look what i'm really interested here is that config map that that i'm patching here let's find it where is it where is it here we are right so this is the coffin map that is defined in original form side of fargo project and this is the patched version of it right so it took the config map it kept whatever is there already actually there wasn't much but imagine that there was much and it patched it by adding this entry to the data section right i want to use kubernetes api because by default argo workflows doesn't use and by the way hey this is important i'm showcasing here argo workflows that's not the main subject but if you would like me to explore argo workflows and i think you should know about it because it's absolutely awesome let me know in the comments section and i will create a video about argo workflows but this is now i'm using it only as an example right not i'm not preaching it in this video okay so we built it and when we executed customized build base in this case or any part we just get all the manifests with their patches applied and combined and all the stuff prepared for us and output on the screen now customers doesn't have a mechanism to apply that to kubernetes and it shouldn't have because that's not its job its job is just to output combined patched version of all the manifest referenced in all the customization yaml files so it was output on the screen for us and nothing more what we need to do is pass that output to kubernetes and we do that with cubecuttle so we're going to do something similar but pipe now the output the result into cubecut will apply so let's do that so the command is customize build base and then we want to pipe that to cube cattle apply and then file name is dash dash means standard input instead of the real file or the directory we're using standard input in other words output of custom is built and let's see and now we are outputting with customers customized version of our manifests gathered from different places and piping it into kubernetes uh with cube cattle and that's about it we got it we have application running and customized with customize but we might want to go further so let's take a look at how we can improve on what we figured out so far you might have noticed that one of the manifests the ones that i define not from the base repo is english yaml so let's double check what was what did that create cube cattle dash namespace argo that's where it was applied deployed get ingresses right we got argo server here that is accessible through acme.com now let's say that this is production setup and let's say that i don't want to use acme.com in production maybe that's okay in staging maybe that's the case somewhere else but what i really want to do now is say hey for production and production only i want to have a different host i don't want to affect anybody else in any other cluster or environment but for this specific cluster for this let's say production environment i want to use a different host how would they do that right as i mentioned before i could now copy and paste that but we're not going to do that we're going to use what we call overlays in customize to make slight modifications that are applied only to production i already defined everything so we just need to take a look let's take a look at what we have in the directory overlays and then we have over there production directory now what i'm doing right now i'm making customization that is specific to production without affecting everybody else everybody else might be using base or extending base but this is now customization only for this specific environment and what we have there is another customization yaml and ingress patch json so you already saw how you can patch using yaml when i patch the config file and that patch is applied to all the instances all the places where argo workflows is running but i want to patch ingress only for production and i will be using json instead of yaml mostly to show you that you can use json as well you don't have to be limited with jumble if that's not what you want so let's take a look at that customization and that is in overlays production and then customization.jumbo so what this one is doing is saying you can ignore the workflows that's not really relevant for this demo but this very much is this is telling customers hey apply everything that is defined in the base directory which is sub sub directory of this one and then do some additional stuff so basically do everything we already do did but a bit more for this specific environment and that a bit more is to apply a set of patches i have only one patch here it could be more patches and this is now the typical kubernetes way to patch something actually you will see in the json later how it is typical but there is a path to file that provides a patch and then the target target is the definition or reference to the resource that we want to patch and in this case that's networking kubernetes i o it is version one and we want to patch ingress called argos server so this target is defining what we want to patch and this part will be applied to that target so let's take a look at that thing respect what what is it what's inside right uh ingress patch right and this is this is the standard kubernetes patch json you could execute this with the cube cattle patch command so we're just telling customers to apply a patch that you can do without customize but this time we're going to store it in git and what's or not so we're going to replace whatever is the value of the specification rules the first host and use argo workflows.talkme.com so instead of acme.com which is the host applied every time we in all the environments where we want to run our workflows for this specific one for production we are going to change it to be argo workloads acme.com so let's build and apply this new one this patch door overlay let's say so it's customize customize build just as before but this time the path is not based we are going to say hey build something from overlays production uh customers no we don't need to specify actually the file from all release production and then pipe that to cube cattle apply that file name dash right same as before but we are now applying uh an overlay that will extend the base definitions and it will take a couple of moments come on work faster go go go go go go go there we are you can see that almost everything is the same right uh most of the things are unchanged and the one that matters is here ingress networking kubernetes io argo server was configured so now that we applied this overlay specific to production it modified english i mean kubernetes cube cattle uh send it to kubernetes kubernetes figured out that it's slightly different and modified it so let's let's confirm that that's really the case cube cattle dash namespace argo and then we want to get ingresses and you can see now the host before it was acme.com now it's argo workflows.takmy.com so we applied uh an overlay we applied the patch that is specific to production it will not affect anybody else uh using base or some other overlay only those applying the production overlay now let's say that we want to upgrade this uh application let's say that we want to change the tag of of one of the images used for our argo workflows maybe upgrade to a newer version or something like that in other words let's say that we want to change the tag of one of the images used in one of the containers of this application how would we do that so let me see what is the currently used image and the tag specifically for one of the deployments so cubecuttle the namespace is argo i think and i want to get deployment argo server i just happen to remember that that's one of the deployments used and let's output it as yaml uh the call whatever is running currently in the cluster and what i'm really looking for is what is the image that is currently used here we are right you can see that currently it is using the latest image now let's say that first of all maybe we want to upgrade we cannot upgrade easily when it's latest but uh we will actually let's make it like this we want to in this demo want to upgrade it to or maybe downgrade nobody knows because it's latest to a specific tag and at the same time apply the good idea of using all these specific tags in any way anyways i want to change the latest to a specific tag so think of it as upgrading from something to something else how would we do that there are a couple of ways but in any case i want to go to overlays production i want to apply this change only to production and keep everything else every other place environment name space where the rag or workflows is running intact right so i'm going to do that and try to apply yet another patch to change the image but this time i'm going to i could i could do something like this i could output customization yaml and write my yaml here i could definitely do that but i'm not i'm going to use a shorthand command and that's customize edit i want to set image to something else and the image that i want to change is argo project and then argo cli if i remember correctly that's what is currently used and that image should be the same argo project and argo cli but the tag should be 2.12.4 i checked before this is the latest tag that is currently available at the time of this recording so i'm going to change uh whatever is the current tag of the image i don't care now what it is but this is the image and i want that image to stay the same but instead of being latest which is right now i want to apply this tag and only to production remember without affecting anybody else who might be using those definitions to run argo workflows in different environments now that's the output of the command actually there's not much to output let's take a look at customization yammo um cut customize customization actually and the customer is added this definition to the yellow right so if you remember before this didn't exist so this was added the images section and it says hey if there is an image called argo project slash argo cli i want you to use this as the new name which happens to be the same but i could even change the image itself in this case i'm not doing that and i want to apply this new tag to that image so that is yet another thing that we can do we can apply customization to our specific syntax to change the tag or even the full image of something so now let's go back uh where we were twice and uh let's customize build just as before actually i'm going to repeat the command let me find the command where was it that that executed before there is no reason for me to here we are right customers build overlays production and pipe it to cube cattle apply dash file name dash right i'm executing the same command that i executed before but this time i have this new entry to customization yaml inside all release production and let's see i'm waiting come on come on it needs to go through all the this is a bit problematic it needs to go through all the git repositories that tends to be slow but now it's finished so we are fine now you can see that everything is unchanged unchanged hopefully if i didn't make a mistake there should be a change to the deployment somewhere uh i don't know where it is it doesn't really matter uh here we are configured i found it there's the guy you can see that it reconfigured argos server now let's confirm that it did the right thing that it really applied the new tag and we can do that with cube cattle uh what is it namespace argo and what do we do get deployment what was it argos server and we want to output as yaml and then we want to hunt the image where is the image defined here we are right it applied this specific tag we could do the same thing without images section we could apply just a patch but this is a convenient way simply because this is something you will be doing fairly often right we change images we upgrade stuff we deploy new releases all the time and we accomplish that with customize edit set image we could do the same thing with set name prefix name space name suffix and probably something else and number of replicas so commonly done operations can be done with the customize edit set and then image or not and everything else goes through patches and that's about it now i could create i have right now overlays production i could have overlays staging overlays this overlays that i could create as many variations of the base definition of the manifests which are my version of the manifest in this case defined in a github repository of the argo workflows project and so on and so forth so the the options are limitless basically and really depend on how many environments how many variations we have uh compared to the original manifests and compared to different uh environments that we might be using so that was customized it is a very convenient way to customize to apply patches to create variations of our kubernetes manifest that will be deployed in different environments and also allows us to easily upgrade our releases but through convenient ways to set the image and a few other commands it's an awesome project and it took me a while to get used to it because i come from the helm land where everything is held as time passes and as i'm using more and more customers i'm starting to like it more i'm on a verge of saying that it is a better tool than helm it's not doing the same thing there are quite a few differences the base idea how we should accomplish some objectives is very different but the the end goal from the user perspective is very similar and that is to provide means to apply different variations of our manifests of our application in different environments and to easily upgrade our applications and do a few other operations now is it better or worse or the same as hell that's a hard question that i'm not going to answer now for a few reasons because first of all it depends on use case second because this is a video about customized not the comparison but if you want me to compare it with home just let me know in the comment section i think that would be an awesome video what do you think so let me know in the comment section if you want me to compare it with helm for now try it out play around with it and let me know what you think hey hey stop don't go yet uh subscribe to the channel click the like button uh hit the bell icon and all that stuff jazz uh make sure that you get notifications when i publish the next video and tell the world that you liked it because you did right you did like it correct okay now that we're on the same page click the like button subscribe and all this stuff and see you see you next time see you in the next video

Channel: DevOps Toolkit

Views: 6,093

Rating: 4.9103141 out of 5

Keywords: kustomize kubernetes tutorial, kubernetes kustomize tutorial, Viktor Farcic, manifests, kustomize kubernetes, kustomize tutorial, kubernetes manifest tutorial, kubernetes explained, kustomize k8s, kustomize, k8s, Kubernetes

Id: Twtbg6LFnAg

Length: 32min 41sec (1961 seconds)

Published: Thu Jan 21 2021

Reddit Comments

[u/so0k]

Too bad the integration into kubectl seems abandoned at a very old version Also operators like istio adopt overlays (strategic merge patching) but it’s slightly different to Kustomize.. the landscape is messy but it’s a great last mile modification tool without needing to fork upstream “bundles”

[u/ozeta86]

I used kustomize on our microservices cluster recently. The architect asked for a canary release and various multitenant customizations. So, as Helm was overkill for our kind of deployment, I went for kustomize. I created a base deployment with 4 different overlays , I used templating with env vars pretty much everywhere. So as a first step I run a envsubst to every file inside the manifests folder, then kustomize everything. It works like a charm. It's all really readable, every single piece is at it's place, and as a result I got a template that can be used for multiple services.

As the matter of fact, the base kustomization is common with every microservice, and some of them have only some piece of kustomization. I ended using it also with our Canary on the Istio manifest files. Like the mservices, everything is plain and simple to read.

[u/mightyroger]

Victors videos are fun to watch and always more interesting than reading the docs :)