How To Create Virtual Kubernetes Clusters With vcluster By loft

i'm back into multi-tenancy game now you might say hey this is boring because you just explored a couple of videos ago capsule which sounds like an amazing solution to have multi-tenancy within a cluster and capsule does indeed sound like a better solution than creating a separate cluster for each tenant or does it how about doing something in the middle how about giving each tenant a cluster but not a physical one because managing physical clusters is complicated managing different kubernetes clusters is far from easy especially if you need to have them on demand and if there can be many of them and so on and so forth however if we could create virtual clusters inside the cluster if we could do that we would probably get the best of both worlds we could give each tenant a separate cluster but if that cluster is virtual then operational cost wouldn't be so hard we would still need to manage only one cluster even though each tenant gets a separate one with full permissions for its full glory now let me backtrack a bit and explain potential issues with having multiple tenants in a single cluster like what i showed you with capsule and if you haven't seen capsule please go and check it out the link is somewhere above my head so if you have multiple terrains within the same cluster one way or another they end up having their own namespace or namespaces each term can have multiple namespaces here are the problems with that each of those tenants cannot have cluster wide permissions to do stuff such tenants cannot create custom resource definitions they cannot install any of the third party applications that have any type of cluster by resources which is like half of third-party applications tenants cannot even list name spaces they cannot even see which namespaces they own within a cluster so that inability to create cluster-wide resources or even to have read permissions for cluster-wide resources at least among those that belong to a specific tenant is potentially a huge issue because in those cases tenants are restricted they cannot do anything they want and even if you say hey i do not want my tenants to do anything they want you still want your tenants to feel like it's their own cluster and not being able to do something as simple as cubecut will get namespaces to see what namespaces belong to me is an issue and the real problem is that multiple tenants share the same cluster and we are going to try to overcome that problem we are going to try to figure out how to give each tenant their own cluster but those clusters will be virtual clusters inside of a single kubernetes cluster and those tenants should feel like they own the whole cluster they might never even find out that their cluster is virtual cluster one of the many clusters inside of a single kubernetes cluster and we are going to try to do that with v cluster i go through many tools in this channel and i almost always find a thought in every single one of them in almost every single tool that i explore i end up saying it's okay or it's good it's horrible and there is always that but but you shouldn't use it because of this and that there are very few that i consider absolutely amazing and this one might be one of those but let's reserve that opinion for the end of this video for now let's jump into the practical part of this session let's see we cluster in action i already created the cluster and i already installed v-cluster cli that's the only thing i did if you follow the channel you already know the drill all the instructions all the commands that i will be executing are energized and the gist is in the description the link to the gist actually is in the description so if you want to follow along you know what to do so let me show you how to create a virtual cluster for a tenant so the first thing i will do is create a namespace now you might say hey are you going to give your tenants a namespace and the answer is both yes and no i will not really give tenants a namespace but i will need the namespace where that virtual cluster will be running so i will create a namespace for a team called 18 that will be my first tenant in this cluster so cubecattle create namespace a theme and there is one more step that is not necessarily mandatory but it's very welcome and that is to create a service inside of that namespace which later on will be used to access that virtual cluster i would love if we cluster would provide us with means how to create that service easily it doesn't so i created it myself it's not really a big deal i will create a service that is load balancer type and that's about it it's just a simple service and it is to have specific selectors you will see later why basically we cluster with gear to service with those selectors anyways i will create load balancer service so let's just get going and do cube cuddle uh dash dash namespace a theme and then apply this file name service yaml right and there is one more thing the last thing missing before i actually start creating virtual clusters and that thing missing is that i need to know the ip of the load balancer service it could be not port by the way anyways i need to know the ip because i will need it by the time i start installing a virtual cluster and i can get that ip by executing cattle namespace 18 and i will get the service called 18 cluster that's the name of the service that i created here we are here's the definition right so and i will output uh output it to yaml and what i really need here we are my load balancer was not created yet so i will need to wait for a couple of moments let me fast forward to the end of the process actually there we are the load balancer was created and this is the ip i will copy the type p and there is the this is the last step i promise i will modify him values file that i have here and i will replace this ip which is from me practicing that thing and i will put the ip of the newly created service right this was a bit annoying and painful and i wish that we cluster made this simpler but it didn't still now we are finally ready to install a virtual cluster inside of this real kubernetes cluster and we can do that by clearing the screen first and the command is v cluster in the namespace a theme and we want to create a virtual cluster called a theme it can be any name and extra uh values are held values.yaml there we are virtual cluster is being created it will take a couple of seconds i will count two five one two three okay there we are that's it we have a cluster running inside of a cluster and our tenants should be able to use it right they should be able to access that cluster without knowing that it is a virtual cluster for our tenants for our users it will feel like it is a real cluster even though it's virtual now before i proceed there is one thing one grievance i have with the big cluster i would love if this command would have let's say dash dash output yaml or something like that i would love if it would output yam also attack and store that yaml in a git repository now if you go to the official documentation you will find instructions how to generate copy and paste and modify yaml but it would be great if the cli does that it doesn't now while i'm still admin of this cluster let's say what we got cube cattle get all in the namespace a theme and this is what we got right those are the pods that constitute the virtual cluster there is a pod a team o which is basically based on a stateful set so virtual cluster creates a stateful set that contains the pod here's the pod that pod has two containers and those two containers are the whole cluster they're extremely lightweight they're as lightweight as it can get i will explain the architecture and how it works later for now just think of this as being a pod that is a whole cluster excluding coordinates coordinates is running still on the parent cluster or the base cluster right now what i need to do next is to enable other people let's say that team a or a team to become a tenant in this virtual cluster or to be more precise to become a tenant in my cluster and owner of that virtual cluster i want my tenants to feel like they own the virtual cluster but not the patent one not the real cluster so what they need to do is create a cube config for those tenants and i will do that with yet another command which is v cluster in the name space 18 and i want to connect to the virtual cluster called 18 i could have multiple virtual clusters in the same namespace and i will specify the server which is https and then whatever is the ip of that load balancer virtual service that i created initially there we are i have clip config and let me show you that your config this is the cube config doesn't matter it's just the cube config that now i can ship i can send to my tenants and tell them hey i'm giving you a whole cluster you don't need to know that it's virtual or it's not virtual i'm giving you access to a cluster full admin access even though that's virtual cluster right so now imagine that i'm changing my role i'm not cluster admin anymore now i'm a tenant i'm a member of the a team i'm the admin of the a team it doesn't matter what i am what matters is that i'm not really a cluster admin anymore so i'm a person who got that cube config and i will export cubeconfig variable to be whatever is the current directory cubeconfig.yaml so keep this in mind i'm a tenant now i'm inside of that virtual cluster i do not have access anymore to the cluster itself to the real cluster to the base cluster so what can i do you gave me access somebody gave me access to the cluster and i can do cubecut will get namespaces let's see what you'll get there we are these are the namespaces of that cluster yeah i have cube system q public kubernetes and default right it's a whole cluster for me i can deploy my applications from now on i can say hey this is my application app yaml that has an ingress it has a service and it has a deployment just a silly application to show you how virtual clusters work and i can say hey how about me creating a new namespace cubecuttle create namespace production and how about deploying that application to production i can say cubecattle uh namespace production that's the namespace that i created as a tenant right now and i can say apply file name app.yaml right and now my application soon will be running in production but not production of the cluster itself but production namespace that i created inside of a virtual cluster that have been given access as an admin if at least the namespaces get namespaces we can see that the production namespace that i created is there if i list the resources let's say cube cattle dash namespace production and what should i do get deployments we can see that the application was deployed right to the deployment one of the resources of the application is there i can list the pods as well right here are the pods in production i can see the three pots because that deployment has three replicas i can see services services right there also there and what else ingress yes that application has ingress here we are and ingresses are there as well what else let's say no yes let's list the nodes cube cattle get nodes right and there are even nodes in this cluster me as a tenant in a virtual cluster i see the nodes of the physical cluster i cannot change them and i actually do not see all the nodes i see only the notes where my pods are running but since i have three pods they are distributed across three nodes and i can even see the nodes it completely feels like this is my cluster with no restrictions i can do anything i want within that virtual cluster without affecting the base cluster the real cluster the cluster in which that virtual cluster is running now that you saw how it looks like being a tenant in a virtual cluster actually there's not much to see because it feels for tenants as if they are in a real cluster no restrictions they have their own cluster and they do not even need to know that it is a virtual cluster but let me switch back let me become again the administrator of the cluster itself you know the real cluster because i want to show you what happened in a background while the tenant was working with the virtual cluster so i'm going to unset cube config right i'm going to remove the config or the virtual cluster and go back to the config of the real cluster so let's start by listing namespaces cubecattleget namespaces and the important thing in the output that will appear just now is that production namespace is not there while i was a tenant i created a namespace called production and that namespace does not exist in my cluster in the real cluster it exists only in the virtual cluster similarly if i do cube cattle uh dash names space 18 right that's the name space where virtual cluster is running i can do get deployments because the tenant the previous me created the deployment and that the deployment does not exist again that the deployment was created in the virtual cluster that virtual cluster is actually a real kubernetes cluster of sorts it is k3s cluster running in a pod so scheduler is there and the deployment that i created is there and so on and so forth all the resources i created were created inside of the pod detects as a virtual cluster now if you understand kubernetes you might probably say hey that's actually a bad idea because that means that the pods are running inside of a pod so it's containers inside of containers and that's a bad idea that's very bad idea but that is not the case it is true that the resources created inside of the virtual cluster are being created inside of the pod inside of the containers of the virtual cluster there are a couple of exceptions pods and services and ingresses are actually created in the real cluster so the scheduler inside of the virtual cluster is managing all the resources but when the time comes to create the pods and services and ingresses it delegates that to a scheduler on the parent cluster or the real cluster and that means that the performance the networking and storage and the what's or not is the same inside of the virtual cluster as inside of a real kubernetes cluster because the resources that matter which is spot services and ingresses are created on the base cluster and i can show you that even though the deployment is created inside of the virtual cluster and that deployment created a replica set inside of the virtual cluster and that replica set created pods those spots are actually running on the host cluster and you can see it here right when you're inside of the virtual cluster as a tenant you see the pods as well and you see them with their proper naming here inside of the base cluster those spots are using naming commercials so there is a guarantee that there will be no collision between the names and the three pots that you saw inside of the virtual cluster you can see them here as well they are actually running on the host and the same goes for services and the same goes for ingresses those three resource types are running on the base host and everything else is inside of virtual clusters while at the same time all those resources from the user perspective are managed inside of the virtual cluster finally as an admin i can say hey no more i do not want to give you virtual cluster anymore i don't like you i will revoke your privileges i will just remove the virtual cluster here's how i would do that big cluster namespace 18 delete 18 right i'm deleting the virtual cluster and the interesting part here is that i did not only delete the virtual cluster i deleted even the resources that the virtual cluster created on the base cluster on the on the real cluster and i can show you that if i do cube cattle dash name space a team get pods pots are not there because there is that parent child relationship and as soon as the virtual cluster was deleted all the resources created on the main cluster on the base cluster were deleted as well i rarely get so impressed with the tool as i got through the v-cluster and i got even more impressed when i saw the date of the first release the release 0-1-0 the oldest release i could find in the project was made in april 2021 and i'm recording this in june 2021 so i'm recording this only a few months after the first release of the project and it is already doing almost everything i would need it to do it is mature project that is only a couple of months old i cannot even imagine how far the project will get within a year it has a strong possibility to become the de facto standard for multi-tenancy in kubernetes i showed you capsule before which is also awesome it uses different approach to multi-tenancy but if you would have to place bets on one of those two projects i would say that v-cluster has slightly better chance of becoming the standard it has very low overhead because the virtual cluster itself is based on k3s and k3s hardly uses any resources it is the most resource efficient type of kubernetes cluster we have today so each virtual cluster will not consume any significant amount of resources and the workloads running in those clusters are not going to have any performance penalty because the workloads that matter are running on the base cluster on the parent cluster on the real cluster while everything else including the scheduler and deployments and stateful sets and whatever else you're creating is running is being managed by a virtual cluster and not by the parent cluster that's how the tenants have the full control of those clusters that they are given because everything is happening in those clusters except that the creation and management of pods is delegated to the main cluster to the main scheduler to the one running on the real cluster so cluster admins do not have an overhead of managing real clusters of managing many real clusters one for each tenant tenants on the other hand have a full control of what is happening inside of their virtual clusters and from their perspective they're not getting a virtual cluster they're getting a real cluster because they can do inside of those virtual clusters because they can do inside those virtual clusters all the same things that they would do in a real cluster there are no restrictions they can apply cluster wide resources they can manage their own namespaces they can do anything they want they could install for example argo cd or flux and manage their workloads in a github space they could have pipelines running in those clusters and so on and so forth tenants can do in their virtual clusters all the things that they would normally be able to do in real clusters without the overhead of managing many real clusters and without performance penalties and virtual clusters do not need to be created only for tenants they do not need to be owned only by teams i can easily imagine other use cases we could have a staging environment as a virtual cluster we could create virtual clusters for every pull request and deploy applications there and test them and then destroy virtual clusters without the fear that whatever we are deploying is going to clash with something else that we might be deploying we could combine virtual clusters with the policy management for example we could use kyberno or opa gateway or whatever else you are using essentially we could do in virtual clusters everything that we could do in normal in real clusters try it out and let me know what you think you asked me in one of the comments what do you think about the clusters and that's this video keep asking questions keep demanding what i should do next and i promise that i will do it sooner or later i don't guarantee you when see you next time cheers you

Channel: DevOps Toolkit by Viktor Farcic

Views: 2,253

Rating: 5 out of 5

Keywords: virtual clusters, virtual kubernetes cluster, vcluster, tenant, multi-tenant, multi-tenant kubernetes, devops, devops toolkit, review, tutorial, viktor farcic, cluster loft, cluster k8s, loft, loft vcluster

Id: JqBjpvp268Y

Length: 21min 59sec (1319 seconds)

Published: Mon Jun 28 2021

Reddit Comments

[u/clustersam]

Well if you want to give users root access to your kubernetes nodes... use vcluster: https://gist.github.com/protosam/a97d9c3db588d475fe686eb32280318a

Edit: Ire at the almighty youtube algorithm for the confusion here.