How to configure GitLab CI runner on your PC using Kubernetes executor

[Music] uh don't forget that thing on the [Music] right I need to know why moving our app to the cloud didn't automatically solve our problem you wouldn't let me re architect the app to be Cloud native just put it in containers you can't solve problems by saying techy things kubernetes if you're wondering what is the best and most cost-efficient way to run your gitlab pipelines Wonder no more I'll just tell you I'll tell you why kubernetes is the best option I'll explain kubernetes resource and limits and I explain how they're used don't worry you will not require a PhD in kubernetes I'll show you how to quickly configure micro Kates on an abun desktop and how to hook it up to your gitlab organization or to your gitlab project lastly we'll run a few test pipelines gitlab provides multiple options to run gitlab pipelines they call them executors most of them aren't a good idea for typical project so shell executor is the simplest one to configure but it does not have any kind of job isolation between jobs or doesn't have repetitive build environments the SSH executor is the least supported one according to gitlab documentation so why would anybody use that parallels or virtual box executors are useful when you have Windows machine and you need to run the Linux jobs for example Docker executor seems to be only useful when you have Windows jobs to run and finally Docker machine executor is very strange Docker company deprecated the docker machine so gitlab forked the docker machine and pledged they're going to support Docker machine forever I understand why they did it uh it's the reason for that is gitlab itself runs their Pipelines using the docker machine but I don't believe this is the best option before we move on let me explain kubernetes resource and limits and the best way to explain them is using magnets so with the typical Docker machine setup you would launch a VM of the same size so I have three here and each one has two cores and 4 gigs of RAM and these are my containers uh look they're all different sizes so let's say this is the rubocop uh just runs linter this is the unit test job and this is the system test job and yeah this works but a single job uses a single machine so if this machine is in use to run the linter it cannot be used for anything else this is not ideal you couldn't put any more containers on it you could tweak the sizes of these machines you could make them smaller but then this will work perfectly for smaller jobs but then the ass system spec job that requires 10 GB of RAM to run won't fit on these machines what do you do you're not using the resources the most efficient way possible on the other hand with the kubernetes executor you could spin up a large machine in your cloud or just configure a physical desktop computer with 16 cores and 64 gigs of RAM or however much you need and this is one large machine the beautiful part about this is that your jobs are all different sizes but you can configure your job memory requirements and CPU requirements directly in your gitlab C configuration so I can configure that this job requires 1 gig of RAM and one core to run so I could reserve this space on this large computer using the kubernetes resource requests and limits and I can say all right this is this one little job I'm going to put it here it kind of fits and then this RP job requires 2 gigs of RAM or 3 gigs of RAM and 1.2 CPUs because it needs to run the Ruby processor it needs to needs on the postgress postgress typically isn't the Bott MC in rails specs so you spin up a container a pod like this and then this job this job fix fits in it perfectly and then finally the system specs job requires even more system specs requires like I said 10 gigb of RAM and eight cores on so you reserve the space and you put it here and the way the kubernetes resource and limits work is that you request the space but then you could also specify the limit of how much any given process can grow in size see this rspec job doesn't occupy the entire space but it's possible it'll grow the memory consumption will grow and it will occupy the whole thing same with the system specs so as you can see this setup is kind of like a teteris game it allows you to use your Hardware more efficiently and even if you have jobs that require more cores and less Ram you could configure the kubernetes resource and limits uh for an odd size like horizontally let's say this is RAM and vertically this is CPU you can configure your jobs to do that and it works and using monitoring you could kind of watch how much memory your entire pipeline uses how much CPUs it requires and you can tweak the concurrency this setup requires some experimentation but going into it you would already have some idea of how much memory or how much CPU any given job in your pipeline requires so you could use uh monitoring tools like K9s which I'll show you uh to watch how much RAM the whole system how the whole pipeline uses and you could play with concurrency you could play with concurrency or parallelism of each individual job you could run let's say eight our spec jobs at the same time or 18 of them or run two system spec jobs or like four or seven of them and then you can kind of see uh how the whole system works the best part if you run out of space on this single machine you could just add another machine either of the same size or different size depending on what's uh sitting on your shelf collecting dust this is exactly what I did I purchased another machine with the 16 quartz and another 64 gigs of RAM and I configured these two machines to be the same cluster so essentially I can treat my two computers as one giant machine with 32 cores and 128 gigs of RAM with this setup I can run even more jobs in parallel by just adding more magnets all right this is the easy part let's install micro Kates let's install the gitlab kubernetes runner and let's run a few test pipelines this is the freshly installed obuntu this is a desktop obuntu Linux uh there is nothing installed on it I just simply installed the Ubuntu 2304 I installed the software updates on it and then I rebooted the computer that's all I did and I also Lo logged into my gitlab account so I could test we'll need a couple of things so first of all let's fire up the terminal and let's make it a little bigger couple of things we'll need to install is let's do install them because we'll probably be editing the configuration files on it let's install micro Kates micro Kates I'll just go to the website micro kat. and I'll accept all cookies I'll get started pseudo snap install micro Kates just copy and paste the command as is it's downloading stuff I'll kind of arrange the browser so I could kind of Click into it whenever I need to this is the first time I'm using the snap that's why it's taking a little longer all right so now um the stuff is spinning up I could just micro Kate status wait ready and it tells me that I have insufficient permissions so I'm just going to run these two commands and it says after this reload the user groups either via a reboot or by running new group group micro Kates let's do new group micro Kates and let's run this say same command again looks like it's micro Kates is running High availability node there's no second node there's just this node let's enable a few things micro case dashboard micro case DNS we W need any of this stuff uh for this uh experiment but if you in case you decide to use this cluster for something else later then you could enable a bunch of things then we could start using kubernetes that's a it's literally a single command install I just installed the entire kubernetes cluster on my desktop computer and I could start using it immediately I could run cctl get all in all namespaces and it shows me all of the stuff that's currently running some stuff is still container creating which means it's probably downloading images and it's now done um and to run the cctl command by default I'll have to run the micro Kates cctl all the time but I can make it easier if I just edit my bash RC and towards the end of it I'll just add an alas so I'll just let's copy an existing alas let's name it K and let's define it as micro Kates Cube CTL if I save it and then open a new tab now I can run K get pods and in the default name space there's nothing running currently which is which is what I was expecting so the next important part in order to install the helm charts we'll need to get the micro Kates config and this is the config for my cluster to use Helm the config file must exist in the cube directory so what I'm going to do I'm going to run this command and and I'm going to pipe it to home directory. Cube uh config just a file name config then I'm going to need to install Helm Pudo snap install Helm so that's done now Helm list that gives me the list of installed packages but it's currently nothing's currently installed so and also gives me a warning that the file the cube config that I just created is group readable and it's World readable so what I'm going to do is I'm going to chod 600 for this file and run Helm list now no warnings Next Step we're going to register a runner in gitlab in gitlab user interface either under organization or under project you can go to cicd and under there you can create the runner there is a little trick when you're trying to define the runner under the order level there's currently I believe there's no a no UI for it unfortunately you're going to have to use Postman or insomnia to call the rest API to to register the runner there but I'm just going to register the runner under this project not under the organization but under this just this project and this Runner will only run the workloads for this project so under cicd I can expand the runners and let me delete the old runner that I used when I was testing this and let let's create the new project Runner Linux Runner and I don't need to really specify anything other than the runner description so I'll just do team programmer Runner and optional configuration there's I really don't need any specify anything at all and now the runner is created and now most important thing that I need is I need this token I will need this token to start my kubernetes executor so next step let's install the kubernetes executor using helm so there's a documentation just simple Google I'll just add the gitlab repo didn't copy so gitlab has been added to your repositories then I'm going to update it just in case copy and paste now I downloaded all the charts and let's search the repo and these are the versions of the gitlab runner are that are available to me so there's this chart version and then the app version and note there is two versions 16.5.0 and the reason being is that uh when I was first testing this I installed this this version because this wasn't available yet uh and I couldn't get this one to work because they had a bug they had a regression which they since fixed in the latest chart version 0.5 5 58.1 that also has the same version of the runner but just different scripts and kubernetes so that part's working now I need to create the values file so um with Helm charts usually you could either run them or you could just create uh the file with a config file with initial values or with values that will be then inserted into running resources so let's get this values file configuring gitlab Runner using Helm chart section to create it so first of all I need to create my values file and to do that I'm going to use the default so default configuration can always be found in the value yl in the chart repository so which is exactly what I'm going to do just going to click the default and so this is the file let's download this I'll just download this as is it's now in my downloads directory so if I go to downloads here's the values file let's edit the values file and that's why I need a VI so I install the vi so so I get the nice syntax highlighting if you look at it it has a lot of stuff in it but don't let yourself be intimidated most of the stuff you can just leave as is uh there is some things that you need to change is for example this Runner token so um this is it so this is the value I'm just going to uncomment it and I'm going to go back to my gitlab and this is my Runner token copy and in insert paste okay it's there let's just look at the file from top to bottom um and we'll notice a few things along the way so look uh this is one of the important settings it's called concurrent it's how many jobs this Runner will run in parallel uh any jobs Beyond 10 will be just queed up until until Runners available uh there is check interval check interval is important option it's how often this Runner will check gitlab.com for for the new jobs and then there is a server URL I think it's not def faulted gitlab URL so that's the value that we need to specify and I made that mistake too it needs to be https if you're using gitlab itself and not self-hosted gitlab it needs to be https so let's uh do this so this is gitlab.com that's all you need https column gitlab.com for Slash and really that's it as far as I could tell this is this is enough configuration to start running jobs this is another important section in the configuration this is the runner section and in this Runner section you could specify the image that will be used by default to run all of your jobs and you could use Alpine but it's pretty Bare Bones if you want to use snap or pseudo appg to install packages you could use Alpine package manager as well but uh I'm used to auntu so I'm just going to change it to Ubuntu 2204 and 2204 is a long-term support version so it'll be around for a while so I'm going to save that as well and now that I have the values file I can go back to my documentation and because I'm using the helm 3 I could just copy this command and paste it here and obviously I will change the file name the values file is just value use. emo and my Nam space I'm going to run the gitlab runner in a default so default and Helm install namespace Okay so that's running and if I do K get pods this shows me that the runner is starting up and the container is creating what that means is it's just downloading the docker image layers from the from the internet okay the gitlab runner is running let's let's check the logs okay log DF paste another way and look at this running in user mode get L Runner run in instantiating initiating executed providers builds currently zero Max builds zero perfect so let's look at this uh at the runner go to runnner page and look at this assigned project Runners team programmer Runner it's currently available if I go there uh it shows you all the parameters the version of the application I'm using the IP address is connecting from the the Linux platform architecture and everything else I need and now let's run some test pipelines so just go to application and I have a pretty basic application that only has two files the rmy and the gitlab CI yaml if you look at it it's pretty basic it's a it's just a boiler plate example of a build configuration it has two stages build and test build stage has one job that just uh Echoes a few things to the to the terminal and the test stage has two jobs the unit test job and the lint test job and it has some sleep in it so we could observe the way it's running and not much else and see I'm tailing the logs here for the runner itself but let's uh let's modify this file I I'll just edit a single file and I'll just change one character in the file I'm going to make a a change a commit oops update gitlab config into the main branch now that's committed and let's go to pipelines look at this uh checking for jobs received job ID add a job to processing list and it's doing it's doing things so if you go back to my gitlab now my pipeliner is running build job is running and it actually finished just finished looks like compiling the code cleaning compile complete if I go further uh lint test job so see this lint test job is running in parallel with the unit test job so I can run up to 10 jobs on my computer and uh you could use the system monitor to see how how the computer is using the resources uh it doesn't use much because it's mostly using network at this point because my jobs are really lightweight if I run the K get pods it shows me the Pod of the runner itself and then the Pod of the running job and it's been running for 62 seconds I have a few delays there so after 62 seconds roughly uh the job is gone and so kubernetes the way it works it just creates the pods to run your jobs up to concurrent up to concurrent value which is 10 at this point uh I could probably run more because this machine has a lot more power so this machine has 24 virtual CPUs and it has 67 Gaby of ram which is strange it should should be 64 um you could also monitor your jobs using the K9s okay let's download the K9s tool from GitHub let's scroll down and find the amd64 version so K9s amd64 it's downloading and now there is a K9s so let's go to downloads K9s you could install the application properly uh on your Linux but I just downloaded executable which is not a Linux way but uh it po pulls up this it shows you um everything that's running in your C cluster by default and I could narrow it down to just a default namespace by pressing one and currently I only have one pod running I could press L to look at the the logs and it it's going to tail the logs for me uh it shows me how much CPU and memory is currently utilized and it even shows me um how much CPU and memory it even tells me how much CPU and memory each pod is using it remember kubernetes resource and limits so this is exactly it so I'm going to edit my gitlab configuration to specify how much my jobs actually require to run and to do that I'm going to just edit this file again you could change these settings in the helm charts values file to specify the default because every job if you if you look at the job uh it has several containers in it it has a build container that's main container that's running the auntu image there's a helper container and any servers containers you could specify separately how much CPU and memory you reserve and limit for each individual item here you could specify how much you want to give to postgress service how much you want to give to reddis service and how much to the actual build service but let me show you how you can how you can configure that in in the build service let's Google and there we go uh so this is how you can actually do this and gitlab even gives makes it super easy you can just copy this whole piece of text and it specifies the CPU request and limit and then memory request and limit for the build container there is uh those limits for the for the helper container and for the service container so I'm just going to copy the whole thing and I'm going to paste them paste this in here in this file so if I edit the single file right above I'm just going to paste this these variables and let's make changes here so let's do CPU request 800 M which is 800 M CPUs 8 of 1 CPU let's set the limit to 1 Point actually 1500 M um memory limit or memory request 2 gbt and memory limit let's put it at 3 gbes um there is also Emeral storage request and limit which I haven't seen before this is the first time I'm seeing this CPU helper CPU limit let's make it let's say 500 M and 300 M and I'll leave I'll leave these the same roughly and the service CPU request let's request 200 M and let's uh set the CPU limit to 500 M and remember you could overwrite any of these variables in any of the jobs as you run your unit tests or run your lter you could change the values for that particular job type because you'll know how much resources that job requires and uh that's it so I just made some changes and let's fix the indentation and this is yaml you don't want to be yelled at because yaml is what yaml means is yelling at your computer in case you didn't know so let's do let's change this specify kubernetes resource requests and limits looks like the requests and limits are not shown because my kubernetes cluster doesn't have the metric server editor Alex here at this point I tried to configure the kubernetes metric server for micro case cluster but I spent about 5 minutes doing that and I couldn't get it to work quite right and I ran out of time so instead of doing this I'll just show you my K9s that's connected to my decisely work cluster that is made up of two machines two nodes and don't mind the names well I'll r them later and these are the pods that are running and um you could see that it shows you in semi- real time how much CPU each pod uses how much memory each pod uses what is the percentage of the reserved CPU and what is the percentage of limit CP CPU limit so let's say let's look at this job for example it currently uses 52% of the reserved CPU or 27% of the limit so the limit is quite High you could order all your parts by pressing the shift C or shift M to order everything by memory and you could look at at them like this so this is the most CPU intensive job that's currently running and it's the build job it builds the docker container this next one is no that's actually the system tests that's the system tests that are currently running and so this is this is how easy it is to configure the kubernetes executor for your gitlab instance and you can run your jobs you can specify the resource request and limits and you can use your Hardware efficiently so as soon as I got it the idea that kubernetes is the best option my first brilliant idea was let's run everything on a cloud with the cloud I can scale my worker nodes infinitely and during the night time when most developers are asleep there's no builds I can scale my cluster way down even to the one node and pay almost nothing for it and when the when the load is really high when all developers are pushing mer quests all the time I can scale my cluster way up and I can handle way my jobs this is going to be great right wrong it didn't work that way at all well it did work for a couple of months on digital ocean but uh we had some issues the build times were inconsistent it takes time to spin up the new nodes and when the droplet spins up it starts off as a blank sheet of paper it doesn't have anything installed on it so the kubernetes cluster has to install the kubernetes kubet engine on it bring this into the cluster and then it has to download all these Docker image layers to run everything it needs to run your job so it needs to download the postgress image my SQL image reddis image needs to start all these containers and that takes time in my experience it took about 2 minutes before the newly started droplet can run any tests so you really want to use the CPU optimized nodes where you don't share that your CPU processing capacity with any other tenants so you have to get the CPU optimized nodes and those aren't cheap uh so I ended up with the 8 core droplets at the time of this recording the cost of the 8 core CPU optimized droplet is about $170 a month and uh in this final state that cluster was running between one node one droplet and 15 droplets so the maximum possible bill would have been about $3,000 a month in reality we were spending about $500 to $600 a month to run our builds because most of the activity is during the day it's 8 hours work day roughly and uh it's not that all the time that we have to keep all of the nodes running the problem with digital ocean and Leno is that also they billing is hourly so if you spin up a droplet and use it for 10 minutes and then shut it down you're still paying for an entire hour another issue with the digital Ocean or the node you don't really have control over how your cluster scaling works it just uh Works automatically this whole thing was good but not great so it felt like we were paying too much for this our application build isn't that large didn't feel like it it's going to cost this much to be fair we ran I think 18 rspec jobs in parallel plus about 10 system test jobs in parallel and when everything was uh hot and loaded it took about 10 minutes to run the build I wasn't really happy with this so I was looking for other options and uh if you like me and you have a couple of young kids you probably also have a desktop gaming computer sitting there somewhere collecting dust and doing absolutely nothing so I had this uh Epiphany what if I try to configure kubernetes on this spare desktop computer and see how it'll F against digital ocean that desktop computer isn't a little wimpy computer it has the pretty beefy AMD CPU with 12 cores or 24 virtual threads it has 64 GB of RAM which is plenty for most of our builds so I looked at the documentation and turns out the canonical the makers of auntu they also have the project that's that they called micro Kates simple one of the easiest way to install kubernetes cluster so I formatted the hard drive and I installed Ubuntu on it not the Ubuntu Ser server nothing complicated I just installed the Ubuntu desktop and then I installed the micro Kates I configured the cctl to connect to that and it was surprisingly very very easy then I used the kuus runner Helm chart to install the gitlab C Runner it connected to the organization and with some tweaks it started running our builds and I was surprised that this single desktop was able to handle all of our needs needs of my entire team with developers pushing builds daily and it was enough to run everything so basically that spare desktop computer replaced the need for the hosted kubernetes cluster on digital ocean my next action was I shut down the digital ocean kubernetes cluster and I just started running all of our Builds on that desktop computer then a couple of weeks later I looked online I looked at the PC parts picker and I put together another desktop computer that's even more powerful with 16 cores and 32 threads 64 gigs of RAM and it turns out the cost for this machine was about $1,500 so I ended up purchasing two desktop computers for about $3,000 and I configured the kubernetes cluster on both of them using micro Kates and then I connected them to each other I can treat these two desktop computers as a single bigger machine with 64 threads and 128 gigs of RAM and a lot of storage now and I finally achieved my goal every build no matter the time of day that my team submits or every merge request takes under 10 minutes to build I'm a professional full stack software engineer and this channel is a way for me to give back to the community and to share the solutions to the problems that I came across uh at work please consider subscribing to this Channel and like this video or even buy me a coffee using the link below so my wife doesn't think it's a waste of time have a good day

Channel: Team Programmer

Views: 1,266

Keywords: kubernetes, gitlab, ci, containers, docker, cloud, digitalocean, linode, aws, gitlab ci, builds, tests, pc, server, build

Id: JwXEAWLxKWk

Length: 30min 49sec (1849 seconds)

Published: Wed Nov 01 2023