How To Create, Provision, And Operate Kubernetes With Cluster API (CAPI)

most of the efforts in infrastructures code type of tooling has been around hiding the api most of the tools around the infrastructures code allow us to define infrastructure typically in a declarative format with the exception of maybe pollumi and to interact with different providers using the cli it's almost as if we forgot the benefits of using the apis to do almost everything and part of that everything is to manage different types of resources but that trend changed now we see an increased number of tools that are allowing us to manage whatever we need to manage using an api and there is no better api today than kubernetes api and that is especially important since we recognized that kubernetes and kubernetes api and scheduler and everything else is not only about running containers we've been using kubernetes api not necessarily the whole package but definitely the api to manage virtual machines instead of containers bare metal servers networking storage and so on and so forth and that of course includes but is not limited to kubernetes servers themselves how about we try to use kubernetes to manage kubernetes and there is a project that tries just that that project comes from kubernetes special interest group and it is called cluster api so let's explore cluster api as an api for you to manage your kubernetes clusters no matter where they are [Music] creating and managing kubernetes clusters is complicated it's actually very very complicated from the early days we got cube adm or cubadmin that allows us to bootstrap kubernetes clusters but kubernetes is much more than that we need hardware we need software we need services we need networking we need storage there are many things that we need the complexities of managing kubernetes clusters differ from one vendor to another if you use managed kubernetes offerings like gke eks aks and so on and so forth then the process is relatively simple and straightforward but if you want to manage kubernetes clusters ourselves if we want self-managed solution instead of something that is managed by somebody else then things can get complicated very very fast so cluster api project tries to simplify management of kubernetes clusters especially at scale and it has specific goals that it is trying to accomplish to begin with it is focused on managing the life cycle which is creating updating deleting and whatever else we need to do with our kubernetes clusters it tries to provide the same declarative api no matter where our clusters are running it can be on-prem or in public cloud or anywhere else and finally it is trying to be invisible we are likely going to see other tools popping up that will use cluster api behind the scenes what i want to explore today is whether cluster api is going to be the thing that all of us are going to use whether it is ready for you to adopt it and whether it makes sense to use it in the first place today i'm not in the mood of talking too much without touching my keyboard so let me stop this introduction right away and jump straight into a demo and show you how it works and later on we'll discuss whether it makes sense using it and if it does under which conditions the biggest difficulty for this demo is to choose where i'm going to run it because cluster api supports quite a few providers almost everything is supported and that's great news and i will go through the supported providers later for now i just need to choose something i do not want to use on-prem simply because there are no servers floating around this office so i will skip on-prem i will go for cloud but i don't want to be biased and show you something in aws or gcp or azure because then you would swarm me with comments like hey you're biased you like aws more than azure or you like google instead of the other two and so on and so forth so i'll choose a smaller player and do my stuff in i don't know you know digital ocean why not let me use cluster api to create and manage and do different operations with kubernetes clusters in digital ocean now don't freak out and say i do not care about digitalocean because whatever i'm showing you should work more or less the same no matter where your infrastructure is because that's one of the goals of cluster api it is trying to provide the unified api that we can use to manage our clusters no matter where they're running and that's a good thing so digital ocean it is let's see how this works whether it works in the first place and so on and so forth now before i start let me tell you that i have a gist with all the commands and the link to the list is in the description of this video so if you want to reproduce what i'm doing well you can i did only two things in advance two things that i will not show you two things that i did as part of preparation for this video the first one was to install cluster api cli so i have the cli already running in my machine i will not show you how to install it but the link with instructions is in the gist so check it out and the second thing i did was to create a few environment variables and that's all i did in advance speaking of variables what matters from the perspective of cluster api is that all the vendor-specific things are provided either as a config file or variables and then the manifest themselves are more or less the same no matter where we are running and that simplifies the things a lot because i could just change those environment variables and keep the same manifests and then start managing my cluster somewhere else that's a nice thing because it provides a unified interface that we can use both to define what we want to have to define the desired state and also to interact with the api in the same way no matter where the clusters are running very cool what matters for what i'm going to do next is to say that i have not used manage kubernetes cluster i will not use gke or eks or aks or any of those even though i could or maybe i couldn't but i will get to that later i want to manage it completely myself and that means among other things that i need a virtual machine image that will be used to create the virtual machines on top of which my cluster will be running so the first task the first goal that i have in front of me is to create virtual machine image that will be used for all the nodes of my cluster theoretically i could create those images any way i like as long as those images contain all the tools that are required to run kubernetes i should be fine but i did not want to make my life more complicated than it already is so i will use yet another kubernetes sig project which is called image builder and the image builder happens to have already everything i need it uses a combination of packer and it creates images with background and then it configures those images with whatever is needed using ansible and it has also all the configuration files we need and all the scripts we need and so on and so forth let me skip through all that and clone the repo with the files that i need next i will go into the directory of the newly cloned repo and then to images and copy which is short for cluster api and i will show you the make file it's a huge one huge huge huge make file that contains all the targets that i might need including those for building images in digitalocean the targets that matter in this context are those that start with build dash build node ova and gc and the audio is what i want is short for digital ocean and i can see here that i can build the ubuntu 1804 and ubuntu 2004 and send to s7 or i can build images of all of those the three of those i will go with ubuntu i prefer ubuntu myself and why not take the newer one instead of the older one my target of choice is going to be ubuntu 2004 so let's build the image by executing make build the o ubuntu 2004 and let's see what we'll get in your case it might work or it might fail if it fails it is likely going to fail because it doesn't have some of the dependencies that is likely going to be ansible or packer the script itself tries to install what we need in my case it worked with packer it installed packer but it did not work with ansible so i had to install ansible myself in any case what i want to say is that if the script fails in your case then you are likely going to have to install something manually and that is likely going to be ansible if you do not have it already and now the process is running it is creating an image this will take a while it will take actually a lot of time so i will use this opportunity to tell you that you should subscribe to the channel and you should like this video and you should join the channel as well because that helps a lot keeping the lights on so please do those things or at least some of those things and now i do not know how to entertain you while those things are running while the image is being created so we'll fast forward to the end of the process and then we can continue it took like 15 minutes that's a lot of time especially for digital ocean which happens to be faster than most of the other providers but the good news is that it finished it created the virtual machine then it configured everything that needs to be configured in that virtual machine and then it converted that virtual machine into an image and then it shut down that virtual machine that's the simplified version of the process that was just executed then took like 15 or more than 15 minutes there's one more thing left and that's for me to figure out what is the id of the newly created image because i will need that id very very soon actually you can see the id from the output you see that last line it says id 96680 etc etc but i will ignore that for now because what i really want to do is confirm that the snapshot was really created and i can do that by listing all the images in my digitalocean account and i will do that with your cuddle compute image list and here we go those are all the images that i have which is only one because i created only one image so far and now i can get that same idea that i could have fetched from the output but okay why make it easy if you can make it complicated the last thing that i need to do in relation to building images is to put that id into specific environment variables the digital ocean provider will expect me to have do control plane machine image and the unknown machine image variables with the ids of the images that i want to use i will use the same image for both but in theory you could have different images and probably you should have different images but that would be the subject of a different video and to finish it all up i'm going to go out of the repository that i cloned because i do not need to build the images anymore i might need to build an image later on if i want to upgrade the cluster but we'll get there for now i'm going out of the repo and back to where i was cluster api delegates the task of creating and managing and destroying clusters to specific providers so we need to install all the providers of the infrastructure that we are using in this case i'm using only digital ocean but you need to trust me that you could install provider for almost anything and installation or to be more precise initialization is through the command cluster cattle in it and then dash dash infrastructure digital ocean so digital ocean is going to be our infrastructure of choice today mostly because i wanted to use the cheapest option i can imagine and digital ocean is right somewhere around the top of the cheapness leaderboard and now i can create the cluster or to be more precise i will not create the cluster i will create the manifest that will be used later on to create the cluster and i will do that because i'm lazy i could just open a blank yaml file and write everything myself but that would be a waste of my time in your time so i'm going to use the cli to create all the manifest that i need and the command is cluster cattle generate cluster the name of the cluster is going to be devops toolkit the infrastructure where i want to run the cluster is digitalocean the target namespace is infra that's where kubernetes resources that will ultimately create the cluster will be running and kubernetes version that i want to use is 120.10 and the control plane machine count is three i want three machines to act as a control plane and i want three worker machines and i will store the output of that command into cluster yaml file most important argument in that command is kubernetes version which is set to 1 2010 and that version needs to be the same as the version of the kubernetes components that were installed in the image that i built earlier i will show you later how to configure kubernetes version that will be baked into vm images but for now you need to trust me that the image i built is based on kubernetes 1.20.10 and that's the version that i want to create so i have to specify that argument here as well it's a bit of a duplication of things and you will see later that the cluster api is not shy from using duplication which is not necessarily the thing i like the most but hey not everything is perfect the output that is stored in cluster yaml shows us what are the different resources that will be created i will not bore you with explanation of each of those resources at least not just yet for now what matters is that all the different resources that describe different kubernetes controllers are in the file in cluster yaml and we can apply those resources and they will be created in the management cluster and then cluster api will use those definitions to create the actual infrastructure the actual virtual machines and whatever else is needed since i said that the target namespace is infra i'm going to create that namespace you know the drill cube catalog create namespace infra and i will apply the resources that i have in the cluster yaml file and you can see that i got the cluster and dl cluster and cube adm control plan and dl machine template and machine deployment and so on and so forth i got the resources that i need and i will explain them later maybe i will maybe i will not depends on how bored i am for now what matters is that we can use kubernetes api to see what's going on we can for example list all the clusters and see what we'll get and we can see that there is only one cluster because that's how many clusters i created so far or i wanted to create so far and the phase is provisioning so right now cluster api is doing whatever needs to be done for that cluster to be up and running to be operational we can also list machine deployments which is a resource that creates the vms themselves the cluster resource will try to tie them up later but for now what really really really matters is that machine deployments are working to create the vms that i need that we need and we can see that there are three replicas three of them were updated just now and all three of them are unavailable it will take time until all the vms are created and configured and so on and so forth so let me see what other resource i might want to show you yes yes i know cube adm control plane is probably what you care about the most so let's take a look at the control plane what's the status of it as you can see there isn't much going on all the columns are empty simply because there is no control plane we cannot have a control plane until at least one vm is created so i'm going to get back to this command later to be more precise i will wait for a while and then repeat the command and if it's still empty i will wait for a while longer and repeat the command until we get something and since that part is boring i will fast forward until i do get something here we are we can see that this time there is one replica running that one replica exists but it is still not available it will become available later or to be more precise it will never become available unless we do something and i will show you later what we need to do hint we need networking and cluster api does not install networking by default which is kind of silly but hey many things are silly about cluster api but many things are great as well so it's a mixed bag so let me fast forward again and see whether we'll see some change over time let's say five to ten minutes and there we are now we have two replicas that are still unavailable and they will continue being unavailable i will show you how to fix that but what matters for now is that it says initialize true so the control plane is initialized even though it's unavailable and that's just what we need to make it available and we will make it available by installing networking and we will use calico as our networking solution and to install calico we need cube config so i need to retrieve cube config of the newly created almost working but not really working type of cluster so let me do just that by executing cluster cattle namespace infrared that's where resources were created and i want to get cube config of the cluster called devops toolkit and i will store the output of that command in cubeconfig.cml one thing you should not do is pause this video and try to get this key and try to access the cluster that i just created because after i'm finished recording this video i will delete this cluster so do not try to steal cube config from me because that cluster will not exist for long i'm just trying to save you from futile attempts trying to be wannabe hacker or something like that now that i have cube conflict which i can access the newly created cluster which is still not working but i can access it somehow i can use that cube config to install calico and i will do just that by executing cube cattle cubeconfigurescubeconfig.cml and i want to apply the file that is stored in project kali called documentation so let's just go through it there we go bubba so let me hit the enter key and there we go i installed calico as my network now if you ask me this should have been part of cluster api or something like that but it's not it's a bit annoying that i need to set up networking myself but hey as i said before cluster api is a mixed bag of great things and uh half baked things so that's what we have and to confirm that my cluster is really operational now i will retrieve all the nodes of the newly created cluster the status of all the nodes is not ready so it will probably take a couple of more moments until calico kicks in and connects all those nodes so let me fast forward a bit more a couple of minutes later or one copy later the cluster is operational the nodes the status of the nodes is ready we have two control planes and three worker nodes but hey wait stop stop stop stop stop i'm supposed to have three control plane nodes not two one control plane node is missing i said that i want three i got two something tells me that i will never get the third node because something is not working well in the digitalocean provider i might be wrong though so we'll see later for now i will skip waiting for the third control plane node because two is enough if i want three and i got two then i have quorum i have the majority my cluster should be operational even though one of the nodes is missing and i don't have a good explanation why but that doesn't matter because the cluster is operational actually it does matter but not for the demo the ability to create clusters would not be very useful without the ability to upgrade clusters or update clusters or delete clusters do other types of operations with clusters so let's see how that works i will what should i do let me upgrade kubernetes version to something new just as before the first step is to create a new virtual machine image a image that will have an upgraded version of kubernetes components so we'll go back to the directory where we were before but this time i cannot just build a new image i need to change a file that is stored in the directory packer config and the file is kubernetes.json so over here i will change the major and minor or only major depending on the case in any case i will change kubernetes versions all over this file there is one and there's a second one and the third one and how many more do we have four then fifty i think that there are five entries all together so let me save that file and now i can execute justice before make and build digital ocean ubuntu 2004. now as you already know it takes a lot of time to build an image i think it's around 15 minutes maybe even a bit more so i will fast forward to the end of that process and see what we should do next now let me list the images to confirm that it was indeed created so the command is your catal compute image list and yes it's over there i can see by the name that it is kubernetes 121.6 on ubuntu 2004 and i will memorize or copy the id of that new image because i will need it soon now let me go back to where i was and modify cluster gamble that's the file that cluster api created for us that's the one that we use that i used to create the cluster in the first place but this time since i want to upgrade my cluster i need to modify that file and this is where complications start if you're used to manage kubernetes clusters this will sound ridiculous but the process is as follows more or less we need to create a second machine template that will be based on the newer image so we need to create a second group of nodes and once those nodes are created we need to tell the control plane to use that group of nodes and to shut down the previous group of nodes and to make things more complicated we should do that first for the control plane and then for the worker node so long story short we need to create new nodes and then we need to tell the control plane to use those new nodes and shut down the old nodes and then we need to create another group of new nodes for the worker nodes and tell control plane to use those and shut down the old nodes and so on and so forth it's a complex operation and there is no way around it we need new nodes and we need to shut down the old nodes and we need to tell control plane what those new nodes are and so on and so forth and the workloads should be moving from one set of nodes to another we should do rolling updates there are quite a few steps involved in upgrading our kubernetes cluster using cluster api those same steps are executed more or less in the same way no matter which kubernetes cluster we use but if you're using gke or eks or aks or any of the managed clusters all that complexity is hidden from us because it's not our job it's somebody's else's job to do that or to be more precise the tools know how to do those things but cluster api doesn't know how to do those things so we need to perform quite a few actions ourselves and change the definitions quite a few times first time for control plane nodes and the second time for worker nodes and a couple of other things it's madness i'm so used to public cloud to manage kubernetes clusters that this is something that i'm trying to forget but if you're running on-prem servers then you have to do that more or less unless you have a tool that will do it more intelligently but that tool is not cluster api at least not today now i will not go through that process in this video because it takes a lot of time i want to keep this as brief as possible you just need to trust me that the process could be better and i'm saying that while ignoring potential problems potential errors and there are quite a few issues related to upgrades of the clusters with cluster api they will be probably fixed because the project is in early stages but right now it's very complicated and shaky and tedious and so you need to trust me that the process is far from great and i'm going to skip showing it to you you go through the documentation if you're interested in using cluster api and check it yourself instead i want to talk about cluster api i want to discuss whether you should use it and if you should use it when you should use it and stuff like that so let's switch into pros and cons and conclusion and you know the things that i do at the end of my videos let's start with pros because that is going to be a shorter list at least right now while the project is still in very early phases the biggest advantage i see of cluster api is that it is based on kubernetes api it's about time that we use apis to manage everything and kubernetes clusters should not be an exception we should move away from the idea that apis are hidden from us we should move away from the idea that everything is cli only using apis to manage stuff is a great idea it's something that we should all be doing with all types of resources and kubernetes clusters are not an exception and speaking of api i love the idea that the api itself is more or less not exactly but more or less the same no matter which provider we're using so we can install all the providers we need and yet the api the interface the definitions will be more or less the same no matter whether we want to create and manage clustering in aws or azure or google or on-prem and so on and so forth that's absolutely awesome i'm very happy about cluster api's decision to use kubernetes api to be api based to have control plane and to be more or less universal or uniform across providers because that verifies some other assumptions that i've been making for a while i'm just not sure whether the implementation of cluster api is what we should be going for but i'm 100 convinced that using kubernetes api and all the other goodies of kubernetes to manage stuff no matter what the stuff is is the direction we should be going for i will get back to that in a second for now i want to jump into coins into the things that are not so great to begin with cluster api is definitely better than using cube adm only and exclusively there is no doubt at least in my mind that this is better than cube adm or at least that it will be better in the future when the project matures a bit more however it is still too complex and i know that some of you are going to start throwing stones at me and saying you don't understand the complexity complexity is good but it's not good i like the idea that things are easy things should be easy and this is far from easy we need to build images instead of providers giving us images already pre-built and thus building images only if we want to do that not because we have to do that the upgrade process is horrible it takes me back to where we were maybe five years ago i do not like it it's what dks was at the very beginning and even eks which in my opinion is not a really good kubernetes service moved away from such type of upgrades no matter which tool you choose today or which service you choose upgrades will be hey i want new version of kubernetes i want this specific version let me execute this command or let me start this process and that's about it there are no steps involved like hey let me change this execute that and change this and execute that simply put it is not user friendly another negative thing with cluster api is that it is only for kubernetes clusters that by itself should not be necessarily a very bad thing because there's nothing wrong saying hey we use this tool for this and that's tool for that it's okay to say this tool is specifically designed only for kubernetes and infrastructure around kubernetes and you need some other tool for something else but we already have tools that do the more or less the same things that cluster api is doing except that those tools are having much wider scope we have tools that are managing not only kubernetes clusters and infrastructure related with kubernetes but everything any type of infrastructure any type of services and any type of provider and that tool happens to be cross plane but i will get back to that later another thing i don't like right now is that eks is the only supported managed kubernetes cluster everything else is self-managed so if you want to use cluster api to manage a kubernetes cluster in google you will not get gke you will get self-managed clusters you will get nodes in google cloud but you will not get kubernetes as a service you will be managing it fully completely yourself the same thing goes for other providers for azure for alibaba and so on and so forth self-managed is not necessarily bad it's good for on-prem maybe sometimes at least not always and there are situations where you do want to have complete control over your kubernetes cluster even when you're running it in public cloud but for vast majority of people at least those running in cloud managed kubernetes clusters are the way to go you want to use zks and aks and gke and even digital ocean that i used has managed kubernetes service i think it's called the digital ocean cover and something like that anyways this is not giving you many kubernetes except for eks so we are limited mostly with exception of eks to self-managed kubernetes clusters not managed kubernetes clusters not kubernetes as a service we cannot create those with cluster api at least not today at least not in november 2021 and that makes me think that cluster api is not really a good solution for anything but on-prem clusters if you want to go cloud there are better tools to do that some tools that exist already for many years some new tools i will comment on those later but there are better tools to do the same thing more or less another important thing is that in november 2021 the project is very green it's just graduated from alpha to beta but providers are very shaky even though cluster api itself might be working very well and might be production ready providers are not you saw me using digital ocean i wanted control plane with three nodes i got two nobody knows why but that's what i got upgrades are sometimes working sometimes now there isn't enough information in logs events are non-existent more or less and so on and so forth so it is still very very green but it has a bright future or maybe doesn't it might have a bright future we'll see about that and the last thing is the chicken and egg problem that all the tools of that kind are having you need the kubernetes cluster to run cluster api just as you need kubernetes cluster to use argo cd and you need kubernetes cluster for other tools kubernetes is becoming the defacto standard and there is an increasing number of tools that require kubernetes cluster just to start with but since kubernetes is more or less commodity that shouldn't be a big deal so here are a couple of big thoughts to begin with cluster api might be a good solution if you're running on-prem i would not really use it for cloud at least not in the state it is right now it has a potential for greatness or actually it would have potential for greatness if it was created earlier the ideas behind cluster apis are great but we already have tools that do the same thing as cluster api except with bigger scope and more mature and if you watch my videos you probably know which tool i'm talking about that's crossflame crossplane is doing the same thing that cluster api is doing plus many many other things the scope is much wider it's not limited only to kubernetes and it is based on similar principles it is based on the idea that we have control plane to manage our resources that control plane is sitting in kubernetes we are using cube api to do stuff so the principles are the same except that cross plane is more mature it has additional features that we might need and the scope is much wider because let's face it most of us do not need only kubernetes there are always other things that we need to manage other things related to infrastructure or to services and this one will not help us with this this is very similar to what tks cattle is doing except that it is based on cube api which is a huge bonus and that it can work across multiple providers but it is still limited only and exclusively to kubernetes which would be great if we don't have tools that already do that with by the scope and more mature which again is cross plane there are a few others but cross plane is the one that is closest to being that type of software widely accepted type of application now comes important note i'm very heavily involved with crossplanes you might consider me biased i'm trying not to be but you never know so you might say victory is saying bad things about cluster api because he's promoting cross plane and i am promoting cross plane but i'm doing that because i believe that it is indeed a better solution now whether you trust me with that statement or no is a different question right now before i leave i need to tell you a bit about my feelings i feel really really good after using cluster api for a while because it made me appreciate cloud even more i got so used to gke and google cloud and eks in aws and aks in azure and civo and lino then digitalocean and i used managed kubernetes services in all of those that i forgot how painful it is to create and manage kubernetes clusters yourself from scratch which is more or less what cluster api is doing it is a better choice than cube adm but not necessarily a great one at least not today now we'll see where the project will go and where it will be half a year here but right now i'm not convinced you

Channel: DevOps Toolkit

Views: 6,950

Keywords: devops, devops toolkit, review, tutorial, viktor farcic, cluster api, cluster api k8s, cluster api tutorial, cluster api 1.0, cluster api vs crossplane, kubernetes, k8s, cluster api kubernetes, crossplane, cloud native, open source, devops toolkit youtube, devops toolkit by viktor farcic, devops toolkit series, viktor farcic youtube, viktor farcic devops toolkit

Id: 8yUDUhZ6ako

Length: 33min 54sec (2034 seconds)

Published: Mon Dec 13 2021

Reddit Comments

[u/kepper]

I'm using Cluster-API with OpenStack and so far I've been very impressed. We're wrapping our own K8s-aaS service around it. It's a pleasure to work with.