Helm vs Kustomize - The Fight Between Templating and Patching in Kubernetes

helm is a much better option a match a competition of five customized is a much better option the competition the fight so who is the winner [Applause] today we are going to have a fight it is going to be a competition it is going to be a huge match fight to death between helm on one hand and customize on the other who do you think will win the fight the competition will follow certain guidelines there will be four rounds four rounds in which those two will be competing first we are going to see how do they behave when we want to define our applications so how do they fare when defining applications then in the second round we are going to see how do they behave when we want to define environment specific configurations or applications when we want to specify how should an application behave in a specific environment let's say production or staging or something like that then they will be faced with unplanned changes with unplanned customizations what happens when somebody needs something that was not taught true taught off in advance and finally we're going to see what happens when we want to use third-party applications with helm and customize to make the competition the fight fair there will be a couple of rules first we will assume that whomever is using homework customize is applying github's principles that means that everything should be defined and stored in a git repository no random arguments no random things happening everything based on definitions stored in git we will compare those on the grounds of defining and applying application or kubernetes resource manifests those might have other features and if they do those features are out of the competitions so whichever other features they have they are out of the scope we care only about ways to define applications to define manifests or templated manifests or customizable whatever manifests and how to apply them to our kubernetes clusters finally we will not go into details of what helm is and how it works in all its glory nor the same thing for customers we will not go into details i will assume that you have certain level of familiarity with those if you don't then look for my past material uh i use helm almost everywhere or i used we'll see about the future and for customized there is a link here somewhere above my head that you can follow to see the video about customize and customize only but in this video we're not going into details of what each of those tools is how does it work and all those things we are just going to witness a match a competition a fight so before we start the competition let's talk about preparations we did for the competition to take place i created a cluster i'm using minicube it could be any kubernetes cluster doesn't really matter i installed nginx ingress because some of the demo manifests the manifests require it and i cloned the repository v farsig which is me slash helm vs customize github repository the address to that repo and all the commands are in a gist and the address to the list the url is in the description so go to the description of this video if you want to do the same as what i'm doing if you want to replicate the steps if you want to reproduce the competition the fight and i have two terminals here right in this side in this terminal you will see hem commands and in this terminal you will see customized commands you will see equivalent commands for both more or less as equivalent as they can get and you will be able to compare both of them by seeing them both in parallel so let's start the first round which is about defining applications how do we define applications in one or the other so when defining applications with helm we use templates and the templates look something similar to this helm templates english jumbo right now they are a mixture of yaml which is a typical kubernetes definition manifest yaml plus some templates or snippets of templates like in this case release name or more importantly in the examples that follow this one values ingress host so it's a combination it's a templated yaml so it's not pure real which might be good or bad thing depending on how you look at things now if you go to customize the equivalent of this definition in customize would be this one customize base ingressiamo right now the difference is that this one is pure pure yaml there is no template there is nothing uh except simply a typical kubernetes definition for an english controller nginx in this case now there is another thing uh important in both cases we need to be able to customize hey i'm using different customers look at me to customize how will certain manifest behave to find unit for specific use cases and in both cases we want to be able to change potentially the host for example of of an application of ingress host right so in case of helm we would do that customization through variables so we have something like this helm values.yaml that define hey ingress is enabled and the host is this one right so this is how we are going to define the default host using helm in case of customize we don't have any equivalent of variables of values the closest we have in customize would be something like that the base customization yaml now this one doesn't define anything but the resources that will be deployed deployment service and ingress in case of helm everything in the templates directory will be deployed and in case of customize only the resources that are specified will be deployed which can be good and a bad thing depending on how you look at it customize forces us to be explicit to to set hey i want these and this and those and but which are those are the resources i want while helm uses everything that is in a template directory now that is convenient because you can just push things over there but on the other hand if you want to exclude some files you need to start writing some nasty uh if statements which are in my opinion horrifying but hey i'm judge i'm not a jury in this case but in this specific case there is no equivalent of values in case of customize for a simple reason because uh we are now presenting the default behavior how do we define the default behavior in helmand customers with the potential to change it but later so we have no values simply because our equivalent of values in customer is because by default it should be pure yaml we are going to see how it how we change that later now let's see how we can apply those manifests in helm and in customize so in helm it would be helm upgrade dash dash install which both combined are equivalent to cubecut apply devops toolkit and we're going to call it the directory actually where the chart is defined is helm and the namespace will become dev and finally we want him to create the namespace so that's how we do it in helm in case of customize we are going to create the namespace first i will show you later how we can actually let customize create namespace but for now we'll create it cube copy create namespace customize dev and we are going to build the manifest now this is a huge difference between potential huge difference between helm and customize helm can be used to apply manifest to the kubernetes cluster with customize no custom you there is no customized equivalent of cube cattle apply and that can be a bad thing but can be a good thing because with customers assumes that we are using cube cattle it doesn't try to replace cube cattle in any form of way its job is only one it has only one job and that is to generate manifests that can be piped into cube cattle to apply them to the cluster so we would do something like customize build and then the path customize which is base and we want to pipe that to cube cattle and we want to specify the namespace which is customize dev and we want to apply whatever is defined in the input file name dash means standard input this is how we apply manifest with helm on one hand and customize on the other and you already saw how to define manifests what the what are the differences between templating and pure kubernetes jumbo and how we can really haven't seen how we can customize yet but we are now talking about defining the application with its default behavior so in this round who is the winner uh i would say that customize has a slight edge uh when we're talking about default behavior of applications because helm requires upfront planning you need to figure out in advance which parts of manifests will be templated and we don't need to do that with customize because it uses overlays which we are going to explore in the next round so i would say that for defining applications customize is a winner simply because it's easier it is pure kubernetes now you can use pure kubernetes symbol files in helm as well but that if you do that you will it would be a tie but then custom helm would be a huge loser later when we get into customization so we definitely need to plan in advance with helm now let's go into the second round clear clear second round is about defining environment specific applications or manifests or customizations of our applications let's say that we want to deploy to production all right how would we define what production is and how does it differ from the default uh manifests uh using home and customize so in case of helm uh we would have a separate file with values so the values that we used before are the default values and we would have values that overwrite those defaults in usually in a separate file so we would have something like values.production.yaml and this would be the values that we want to overwrite for production ignore the parts that are commented you will see later think of them as non-existent they will become important later in the fight but for now and this is how we tell with helm hey in production i want my host to be different instead of helm.stoolkidsseries.com it should be hem dash production and then whatever is the address right so we overwrite the default values or we customize the default specification using variables or values and that is pretty straightforward anybody can do it no matter their knowledge about helm and even kubernetes itself you don't need to know anything about almost anything to change your values now in case of customize it's a bit slightly different a lot different actually so let's take a look at customized overlays production this is the overlays that defines production and the customization jumbo file defines what will be different uh in production and we can see here that we are specifying the base so the it will use whatever is defined in the base directory as the default setup default manifests and then we have a couple of additional entries here again just as with the helm ignore the commented parts they will become important later for now imagine that they don't exist so we're specifying the namespace where application should be running when being used in production uh we cannot do that easily with help that's a bit of a bugger and we are specifying a patch we are saying hey for us to change how what will be the host english host in production we are going to patch something based on the specification this file and that something is the ingress called devops toolkit so as you can see this is a bit more complex we are staying with pure yaml or json and applying patches instead of specifying different values uh through values.tml or equivalent in helm now let's take a quick look at that patch so cut customize overlays production and then english patch this is a standard let's say kubernetes patching mechanism so we are still sticking with kubernetes we are saying hey we want to replace uh something specified as spec and then rules and the first rule or zero index rule and the host and the new value of that path should be this customize.production. so in case of uh customize we are using patching we are patching things in the same way we would be patching it with kubernetes more or less but uh define this code defined as yaml storable in git now if you want to apply those things if you want to deploy our application to production in case of helm that would be let me clear both screens in case of helm that would be helm upgrade just as before dash install and in the application is devops toolkit and the path to the chart is helm and then we want to specify the namespace which is helm production and we want to create that namespace just in case it doesn't already exist and here comes the new thing the important part which is we want to specify additional values which are stored in helm and then values dash production so we are basically running the same chart but overwriting values with some other values which in this case are stored in values production yaml file and in case of customers customize build and then the pipe pot is customized overlays production and then we want to pipe that to cube cattle apply as the file name dash so in case of customize we are specifying a different directory which is an overlay directory that will use all the base manifests plus the patches the end result is the same but the way how we customized our application to be specific for production or slightly different in production varies greatly and i might say that hyung is a winner here a clear winner it is much easier to change especially for the end user right maybe not for the operator but for the end user it is much easier to change a value uh of a variable than to create the patch you know with a replace this with this and that and what's or not it is definitely harder with customize for the end users it and it is much easier to change values in helm so a big winner of this round this helm it is much more user-friendly especially for those who are not how do i how to say how to call people who are not kubernetes ninjas right who do not understand everything about kubernetes they should understand they just want to deploy their applications somewhere winner is helm now let's clear this up now let's clear this up and start the new round and the new round is what happens if we have unplanned customizations what would happen if we need to add something that nobody thought we might need to our application and that's something to be added only to a specific environment and as example we're going to use resources let's say that for whatever strange reason which couldn't be justified in real life we do not want to specify resource requests and resource limits in all the environments we want to have it only in production and nobody nobody until now talked about including it in either customized or hemp charts how would we deal with that as an unplanned change specific to an environment so in case of helm we would open helm values the channel and then we would start adding additional values so we would add something like resources i'm not commenting here instead of writing so resources enabled uh false and we would add data to be empty now this is very important in case of helm when we do post changes we need to make sure that those changes do not affect anyone who is already using a chart and doesn't want to apply those changes at least when that's what we want so with enabled false we are saying hey later on when we change the template whomever is currently using our chart and doesn't want to have additional uh information about the resources should keep it to enable false and that should be the default value so we are trying to avoid messing with existing users because they might not want to define resources and the next thing apart from changing values we need to change the template itself now i'm going to open this snippets deployment resources only so that i can copy and paste because i'm too i'm not i'm too old to memorize things so uh if we would need to change the templates and to change the templates we would need to modify uh one of the templates and that's the helm templates deployment for example might be where we want to add resources and then we would need to modify the existing template in this case deployment by adding something like this and this is a bit ugly uh what this really means is that hey if the resources enabled is false then do nothing and if it is set to true then do something and that way we are maintaining backwards compatibility you will see a lot of if statement in helm charts in home templates uh as a way among other things not the only thing but among other things to maintain backwards compatibility so it translates to apply those changes only to those who opt in for those changes and the changes are to add the resources section and uh the values of the resources section you know the memory and cpu requests and limits are whatever is in the resources data value variable and then indent it by 12 meaning move it to 12 characters to the right it is far from pretty or trivial but again depends whether you like templates or not now let's see how we would accomplish the same thing with customize actually let me keep this on screen so that you can see uh see the hem part so with customize uh we would create a patch instead of changing existing templates or what on what's or not we would create a patch that is specific to production and this is a huge difference everything we do with helm applies to everybody unless we say otherwise and in case of customize we create patches that are specific for some overlays or some environments or namespaces clusters whatever we choose to differentiate it so cut customize overlays production deployment patch now imagine that we just defined we just wrote this file and this file is basically saying hey when you take the base deployment the standard deployment definition called devops toolkit i want you to go to spec template spec and then containers to get that to the that part of the existing definition find the container called devops toolkit and then add or modify the resources so you can think of this part as all this as being a pot that to something that should be changed and the latter part starting from resources is what we want to patch it with and then we would need to make a change to customize overlays actually let me clear it so vm customize overlays production customization and i will uncomment here you would normally normally write it we would say patchy strategic merge and this is the file we want to use to patch something right so it stays pure yaml we are just defining a patch that will be applied only for this overlay or only for this environment in this case for production now let's clear this up there is one more thing i forgot almost hey hey that was bad for me so helm i will need to modify the values that we use to overwrite default values of helm those that are specific to production so it's helm values production and then we would be writing and in this case i'm commenting this section hey by default resources is set to be enabled false and now we are going to change it to be enabled through so we're going to enable the resources section of the template only for this environment and we're going to specify data uh limits uh cpu memory request cpu and memory right and one more so we're specifying values that are specific to this this environment production and there we are so applying is the same as before now in this case i will proclaim customer is a winner when things are unplanned then helm is a bit more complex at least from the perspective of the person doing it because it requires changes that could potentially affect everybody so we need to ensure that the default values are set in a way that that they do not apply to anybody except to those who want to uh opt-in for that new addition to for that new change uh it's nasty if statements and the multiple value definitions default ones and all written ones and so on and so forth customize is much more friendly when we want to make unplanned changes because patches are at least in my opinion as a judge of this competition more easier not more it's easier now let's go to the next section and for next section we're not going to use screen so i'm going to go full screen again so how do we deal with third party applications with applications that we do not control or at least do not control fully or maybe we could control them but we don't want to because they are third party apps if you use a third-party app that has a help chart that does everything and i repeat everything that you need helm is a clear winner there is absolutely no doubt that you should use helm with third party applications if the chart that they provide does everything you need however if you have extra requirements then helm is going to be very problematic when it comes from third parties and the reason is very simple nobody and i repeat nobody can define a hemp chart that contains all the permutations that everybody or somebody or anybody might need and the reason is very simple kubernetes is huge is vast and we are nowhere close to have some kind of standard than the things that are adopted by everybody let's take networking as an example most health charts assume that you're either not exposing that application to the outside world through some form of ingress or that you're using nginx ingress but that is not always the case and actually with increased adoption of service meshes that is less and less the case it might not be english it might be istio gateway or it might be english but not engine x but something else there are many permutations of how you might expose your applications to the outside world and then there are many other things how do we do internal networking are we using again service mesh or kubernetes services how do we deal with persistent volumes and storage and so on and so forth how do we deal with secrets almost all helm charts assume that you're using kubernetes secrets but that is less and less the case and if it is less and less the case then hey which secret management are you using it could be something like sealed secrets or it could be vault or it could be many many different things it is impossible to predict all the permutations that everybody using a third party up or sometimes commercial sometimes open source anyways third party app because there are too many of those permutations so if helm chart of a third party app does everything you need use it but if it doesn't then customize is a better option and i would argue that for most of us actually help charts of third-party apps do not do exactly what we need them to do and there are some modifications that we need to do and that is possible with helm but it is not easy with customize i can say hey use the official kubernetes manifests for this app let's say argo workflows and then patch them with those changes replace secrets with this or do this or do that and i can even select which manifests i want to include unlike charts that are basically including always everything except there is an if statement that can be used to exclude something so who is the winner of this round depends i hate when others say depends because i think it translates to you don't have an opinion but in this case it really depends if a hem chart third party help chart does everything you need to do use helm film is a better option but if there is any change you want to apply to manifest all those third party apps that is not already thought through and included in account chart customize is so much easier so much better option and on top of that let's not forget about one more important thing not all third-party apps have helm charts but absolutely every third-party application that is meant to run in kubernetes has kubernetes manifests and customize adopts kubernetes manifests naturally for customized kubernetes manifests are what actually it needs so who won nobody that's cool one neither of the solutions give us what we truly need and which one you're going to use depends on what is your target audience for end users for people who are not let's say c submits or operators who do not understand kubernetes in depth helm is a much better option because i can say hey i'm a javascript front-end developer i want to deploy this application i don't know much about kubernetes i'm just going to take a look at those values change a couple of them a few tweaks to the values not to the manifest i cannot change the manifest but i can change those values uh i understand what those values are and off i go my application is up and running i know nothing about kubernetes that is user friendly that is great and knowing the details about technology becomes um a bonus not a requirement and with helm understanding of how kubernetes works of its manifest resources and what's or not is a bonus it's not a requirement unless you're the person defining help charts then you definitely need to know ins and outs of absolutely everything but for for the end user helm is so much easier now for the person who does define manifests and defines what should be the resources deployed and applied and what's not in kubernetes customize is a much better option because it is more flexible it is closer to kubernetes doesn't require us to think in terms of templating but in terms of defining kubernetes resources like pure yaml manifests and applying patches differences changes to those manifest depending on the environment or in customized terms overlays so who is the winner depends who is the audience or who are the judges if the judges are developers the winner is helm if the judges are operators or cs admins people who understand kubernetes in depth the winner is customized so there is no winner because one is end user friendly and the other one is uh sysadmin operator friendly now there is a gray area between it's not like hey this is only for end users and this is only for operators of course there is a gray area in between but if i would need to qualify it i would say like that no winner for developers helm for operators customize thank you for watching and remember the most important thing you need to do apart from choosing how more customize your boat is to subscribe to this channel and then to click the like button and if you want to support this channel then you might want to get some courses or the books they are all in the description of this video it means a lot to us if you subscribe it means even more if you like the video because you did like it right if you want to buy us a coffee or a pizza or a redbull then get one of the courses or the books the links are in the description thank you so much for watching see you next time cheers

Channel: DevOps Toolkit by Viktor Farcic

Views: 7,014

Rating: 4.9605265 out of 5

Keywords: kustomize vs helm, kustomize, helm, kubernetes, k8s, templating, patching, template, patch, helm 3, kustomize vs helm 3, kustomize vs helm 2021, kustomize kubernetes, kustomize k8s, kubernetes helm, helm kubernetes, helm chart, helm charts kubernetes, helm chart tutorial, helm charts beginners, helm charts example, kustomize kubernetes tutorial, helm kubernetes tutorial, helm kubernetes example, helm kubernetes explained, kubernetes helm 3, helm 3 kubernetes, Viktor Farcic

Id: ZMFYSm0ldQ0

Length: 34min 4sec (2044 seconds)

Published: Tue Jan 26 2021

Reddit Comments

[u/minimalniemand]

we currently use a mix and I'm seriously considerung only using kustomize in the future

[u/skaven81]

Come on, people...It's not a fight. Kustomize and Helm don't have the same roles, and so they can't "fight" with each other.

Helm's primary purpose is in packaging apps -- its templating features are secondary. Kustomize is a templating utility -- its packaging features (if you even consider it having any) are secondary.

If you are deploying persistent apps into Kubernetes, you need to be using Helm, for its packaging capabilities. If you only use Kustomize (or just helm template to generate manifests), the end result is just an un-versioned pile of YAML that you can kubectl apply to get running in your cluster. But that's not a "package". What if something gets removed from your application? The next time you do a Kustomize run and then kubectl apply, it's not going to remove those deleted things from the running app, because Kustomize doesn't grok what is currently running in the cluster: it's just a templating tool.

Helm, on the other hand, does grok what is running in the cluster. It assigns a single label and version number to the package that is running in the cluster, and you can upgrade and downgrade between those versions. Resources that were added or removed between versions are properly added or removed from the cluster as expected. Kustomize can't do that, nor should it.

The big caveat to all this is if you are deploying in an "immutable infrastructure" sort of way, where every app deployment is brand-new. So instead of "upgrading" the existing app you have running, you create a new namespace and deploy the "new" app, then use blue-green or whatever to migrate the traffic to the new version. In that context, the value that Helm brings around packaging is substantially less, and so in that context you very well might get away with just using Kustomize to template your deployments.

[u/iputfuinfun]

I think this can be bested summed up by this quote by some of the maintainers of ArgoCD.

"But in the worst case, where existing chart parameters cannot support your desired manifest changes, you go through a whole song and dance of bundling a new a Helm chart, bumping its semantic version, publishing it to a chart repository, and redeploying with a helm upgrade. In the Linux world, this is analogous to building a new RPM, publishing the RPM to a yum repository, then running yum install, all so you can get your shiny new CLI into /usr/bin. While this model works great for packaging and distribution, in the case of bespoke SaaS applications, it’s an unnecessarily complex and roundabout way to deploy your applications. For this reason, many people choose to run helm template and pipe the output to kubectl apply, but at that point, you are better off using some other tool that is specifically designed for this purpose."

- GitOps and Kubernetes

[u/MacNugget]

Interesting topic, but it's difficult to cover this subject adequately in video. I'd much rather have read this than watched it.

[u/shared_ptr]

There is a middle ground between the two, which is a different templating solution such as Jsonnet.

We've open sourced some of our internal documentation, and one of the pieces is called "Why Jsonnet", which covers exactly this:

https://github.com/gocardless/utopia-getting-started/wiki/Utopia:-Explanations:-Why-Jsonnet

Kustomize allows you to patch, but very inflexibly, and isn't good at code reuse or large projects. Helm can be flexible through (god awful, imo, but still) templating, but as soon as you walk outside of your chart you need to change the upstream package.

Jsonnet can provide the flexible templating with full code reuse, and enable patching from the outside in. Cuelang is another alternative for this. Both are much better suited to managing kubernetes resources than either jsonnet or kustomize, going by my experience.

[u/mikesplain]

IMHO the best thing to happen to helm (besides getting rid of tiller) was deprecating the default charts repo. Forcing tool developers to take on the lifecycle management of charts has rapidly sped up so many charts from being constantly outdated and insufficient to being automatically updated with each tool release.

Neither tool is perfect but at this point they both have their use cases in my tool box.

Nice video and great discussion!

[u/austerul]

What fight? I use them together and many people I know do that. They cover different use cases (though there's some overlap) so it's not one or the other.

Cheap clickbait.

[u/stemid85]

We have a service oriented architecture with Golang, Java spring boot and .Net. I started setting up the staging environment for this using Helm but thankfully discovered kustomize before it was done and made the switch.

I'm happy with kustomize. I can't find anything it's lacking in our context.

We keep the kustomizations in the same repo as the source partly because it's simple and it works and also because it keeps CI setup simple. Devs push and trigger deploy and no need for any other repos to be involved.

At first it surprised me how little we needed Helm templating. I just setup a base kustomization and then use one overlay per branch that needs to be deployed. So staging and prod essentially.

[u/tommy3511]

We use TypeScript currently. It's easier to edit and reuse. The editor integration is also perfect.

https://kosko.dev/