Let's learn GitHub Actions in a self-hosted Homelab!

as we start diving more and more into it automation tools I thought it's a good time to start learning GitHub actions I recently figured out how to use it to deploy applications on a kubernetes cluster in my home map so instead of spending all day sitting in front of a terminal and configuring my applications I could just lay back and wait while it's all done automatically by GitHub actions this is pretty amazing and can be used in so many different ways like building and testing code running Security checks or deploying infrastructure the possibilities are endless to get started with GitHub actions in your home lab I'm going to show you how to set up a new project install a self-hosted runner on a Linux VM and run some scripts that will deploy a simple web server on my local kubernetes cluster if you are interested in Homeland automation this will be so much valuable for you but before we start I quickly want you to take a look at today's supporter of the video this video is supported by the tree a great free and open source tool that helps you to prevent kubernetes misconfiguration you can test your kubernetes manifests against pretty policies such as security advices anything that should or should not be deployed on your clusters you can also integrate it in all Cube CDL commands using that tree's webhook feature or your CI CD pipelines so before you do anything in kubernetes always validate your manifest industry it's really amazing and if you want to check it out you will find a link to it in the description of this video okay so let's start and let me introduce you to GitHub actions a platform to automate developer workflows in GitHub repositories so whenever you are having a git repository hosted on GitHub where you're developing some code or defining infrastructure or application templates you can use GitHub actions to automate any kind of tasks that should happen if you're performing specific actions on this GitHub repo so that means things like push code changes trade pull requests or open any issues any of these events can trigger automatic actions these actions are defined in yaml files inside the GitHub repo the so-called workflows and these workflows can run one or more jobs essentially this is just a set of commands that are executed where you can do all sorts of things such as building and testing your code deploying applications from templates or performing any other types of automation that you need to make it a bit easier for you to grasp I quickly want you to show this example here so this is the official GitHub repository of the GitHub documentation pages and in this GitHub repository there are multiple workflows defined to perform automatic actions here you can see there are a lot of these workflows in here that all do different things but there's one specific workflow I want to show you this year the link check daily.yaml so this workflow will check the documentation against any broken links it's executed on two different types of events one is the workflow dispatch which is just basically a manual execution triggered in the UI and the other event is a crunch up so it will automatically run every day and if you scroll down you can see the whole code that is then executed on these two types of events I don't think I need to walk you through this whole script step by step you should get the idea of it what we will do together in this video is we will Define a new workflow for one of my GitHub repositories where I'm storing an application template for a web server so this is just a simple nginx web server that I want to deploy on my kubernetes demo cluster this is the GitHub repository you can see I've created three separate DML files in here there's one for my application deployment this just runs a simple nginx web server with a single replica somewhere on the cluster and another one that will add a service object and a third one that will add an Ingress object to deploy all of this on my kubernetes cluster I would now typically just clone this repository on my Mac and execute the CDL command on all three files and this is exactly what I'd like to automate in a GitHub action workflow so whenever I make changes to these manifests and the code the workflow should trigger a new Cube CDL apply command to make these changes happening on the cluster without having me to do it manually all the time that's roughly the idea of it I know it's just a very simple example but that's how you can make your life a lot easier and once you know how it works you can translate this knowledge to automate any kind of tasks that you want okay so what question you might now have is where do these GitHub actions actually run because if you have a shell script or you want to execute any deployment commands like the cube CDL command you will typically need a Computing instance with a specific environment to execute that code and that is exactly what GitHub is doing in the background but so for any workflow that is triggered in GitHub actions GitHub will spawn a new process on one of their cloud machines that will execute a single job at a time and these machines are called the GitHub Runners GitHub provides these Runners with different types of machines and operating systems installed with systems such as Ubuntu Linux Microsoft Windows and Mac OS this is how you can get started pretty easily you don't even need to set up a server for it or install anything you just need a GitHub repository and then you can start defining all your workflows in the ml files completely for free and they will be executed on one of the github's cloud Runners how however in a home lab situation we usually have our own protected Network where we are hosting our own services like I'm running my kubernetes cluster with k3s locally and if I now like to execute my workflows to apply any manifest on the local cluster I would somehow need to allow access from outside to my infrastructure at home so that the GitHub runner in the cloud would be able to connect to it to say you could technically do it this way but I actually don't like this idea to be honest because I prefer to run the actual commands and workflows inside my own controlled environment and this can also be done pretty easily by just using the exact same software that GitHub is running on their cloud and just run it on one of my virtual Linux machines I'm still using the GitHub platform to host my repositories and Trigger the events but the actual Runner that will execute the job at the commands will run as a self-hosted instance in my controlled environment GitHub allows you to set up these self-hosted Runners also completely for free and it's in my opinion a much better way to connect to any local environments especially if you need to control access to internal resources they are probably making it free because it's also saving GitHub Some Cloud resources because they don't need to compute the actual workloads so it's a win-win situation for GitHub and for you as well by the way if you're having trouble following this tutorial because you are not run running a kubernetes cluster in your home lab yet or you are not sure about how to set up a virtual Linux machine to install the GitHub Runner then check out my YouTube channel I'm always doing tutorials about home labs and I also have a series about kubernetes and running my own virtual Ubuntu machines on proxmox I'll put you some of these interesting videos in the description of this video so just check it out whenever you have questions what you need to follow this project in addition to a kubernetes cluster you also need a GitHub account of course but don't worry as I said it's completely free to get started so you can just sign up on GitHub create a new account and just use GitHub action and when you go into the settings menu you can just go to actions Runners and here you can create a new cell Foster Runner that you can deploy into your own home lab infrastructure you can install it on any type of operating system like Mac OS Linux or Windows I prefer using a virtual Linux server in my home lab that I have created on proxmox but technically you can of course just use any other infrastructure you like you can install it in a VM you can install it on bare metal or just run it in a darker container it's completely up to you and I quickly want to show you how you can install the runner software on a Linux machine it's actually pretty simple you just need to follow these instructions here below create a new repository folder for the GitHub Runner and enter it download the latest Runner package from the GitHub servers extract the archive and then run the configuration script with the URL of your organization account on GitHub and your secret token when you have configured the runner you can simply start it with the Run script but if you'd like to make this a background task that is managed by systemd you can also enable it as a service this can be easily done by just executing the SVC script with a installed parameter once you've started this service and it's active and running you should see yourself hostel Runner connected to the guitar platform you can see myself also Runner is now in the idle mode so that means it's connected and awaits further instructions so it's ready and waiting to execute our workflows one thing that I should probably mention here is depending on which types of workflows you'd like to execute in a runner this might require specific applications or libraries to be present on that operating system for example when you're creating a workflow automation that includes command line tools such as Docker Cube CDL ansible or maybe terraform you would need to install these things on the runner system first in our case the workflow that I'm creating in this video here uses the cube CDL command to deploy the web server onto the kubernetes cluster and what the runner does when the workflow is starting it tries to spawn a new Docker container that has the cube CDL tool built in in and executes the actual commands inside this small container when the workflow is completed it removes the container again so it's just a temporary container to be able to run the cube CDL command that means you don't need to install the cube CDL command itself on the runner's VM but you need to install the docker engine and steel I so that the runner can use these tools to create the container and the user account where the runner service is started with also needs the permissions to access the docker CLI basically you can simply install the docker engine on Ubuntu via apt current user to this Docker group the new group command on your user account and now we have everything ready right so now that we have installed the whole environment and yeah I'm sorry I know this video requires a bunch of different things to be set up and installed yeah we have set up a GitHub account with an organization we have configured the runner installed to the right tools on it now we can finally start creating our project and what I'd like to do with you is I've already cloned this repository here that I've shown you before on my Mac let's open it in vs code Now to create our GitHub actions workflow we first need to create a new DOT folder called GitHub so this is where you usually store all configuration files relevant for GitHub and in here we need to create a new folder that is called workflows this is where we just need to put our workflows in it needs to follow a specific format to be recognized by the GitHub actions platform that usually is a yaml file I'll just give it the name kubernetes deploy.yaml and in this file we will first need to give it a name I will just give it the same name as the file name and now we need to Define on which type of events this workflow is triggered remember we can trigger it based on all sorts of event happening in this git repository in the official documentation you could look up what are the possible types of events mostly you would use something like a pull request or a push a push for instance is always triggered if you change code inside a GitHub repository and upload it so that's exactly what we want and you can also specify to which branches this would apply so if you have a large repository with different branches like main staging development and so on this will be very useful to you I usually tend to keep it just simple and for this repository we only have the default branch which is called main so only a push to the main branch will execute this specific workflow now okay great so now we can Define the different drops in here I will start with only one job that should execute a couple of different steps and commands I'll call it deploy and here we need to Define on which Runner this job will be executed note that you can Define different jobs in those workflow to be executed on different Runners because we of course want to use our cell phones the runner that we have installed I will just use it here by adding self-hosted and now we can start telling the runner what it actually should do when the drop is executed let's start with a very simple step called the checkout code this will basically just clone the repository into the runner's temporary container file system so that all the other steps we will Define later all have access to the code from our repo and we actually could just run it and already test it even though this would not really do anything but I'd like to do a quick test now so let's just see what happens when we push this code to GitHub so let's add the workflow file to a new commit name it something like initial testing and then let's push it you can see what's actually happening when you open the repository on GitHub and then go to actions there you can see our workflow which is now running and if you click on it you can see the actual job containing all the steps that we have defined currently that's of course only the checkout command but you can already see all the input and output of the console everything that is happening inside the runner so you always have full visibility of all the commands that are executed in GitHub actions it's pretty useful for troubleshooting and logging some of your automatic actions I will show you later why this is important as you can see our runner has executed our checked out code command and if we go back everything was completed now and the workflow has ended successfully this is basically how you can get started with GitHub actions I would strongly recommend start playing around with it and test it in a repository like this maybe execute a few hello world commands to just see how it works but let's now do something actually useful with this as I said in this project I want to deploy these three manifests here on my kubernetes demo cluster so we need to add more steps to this drop in our workflow that will execute the cube CDL apply command on all of these files just like you would do it manually in a terminal on your PC now typically you can do this in GitHub actions pretty easily you can just Define regular shell commands in your workflow that your Runner would then execute this would absolutely be a way to do it however there is a much easier method to do this for cubectl and kubernetes in particular because in GitHub actions you cannot just run simple shell commands on your Runner you can also utilize predefined environments for GitHub actions that already come with various tools installed such as cube CDL or maybe you want to do something else with ansible or terraform there are so many of these environments around that were created for these specific tasks and you can even create custom environments for GitHub actions you can find them on get hub.com Marketplace and search for something specific in here make sure you will select the type actions first otherwise you would also see the GitHub apps in the search results and then you can start browsing here to find something useful just a side note as you can see this is mostly driven by the community or the developers of the specific tools so always take a look at who has created this environment and if that's really what you'd like to use for example GitHub maintains a repository called actions Hub so this provides GitHub actions environments for Docker Cube CDL and a few other ones and especially the cube CDL environment contains some useful additions to handle authentication properly so that's why I'm using it in this tutorial because one thing is pretty clever here when we want to authenticate to My kubernetes Demo cluster we would of course need to provide credentials to the self-hosted runner otherwise it has no permissions to execute the cube CDL apply command on my cluster and you can define those types of secrets in the GitHub repository and use them in all of your projects this is a very secure way to store sensitive credentials that the runners need to do that we just need to go into the repository settings and in here there is a menu called secrets and variables go to actions and here let's just add a new secret that I call cubeconfig and this is where you can put your cubeconfig file in that you use to connect to the kubernetes cluster note you can just copy this file and place it in here unfortunately you first need to convert it to base64 so I'm simply just using this quick command in my terminal this will take the output of my entire cubeconfig file from my Mac that includes all the credentials and the IP address and settings to connect to the right kubernetes cluster and pipe it to the base64 encoder then I can copy the entire output and just paste it into the secrets variable in the GitHub repository now that we have done this we can just use this environment variables stored in our repository in our workflow right before the checkout command is happening so this environment variable will be globally accessible on all the jobs inside this workflow and how we can finally start adding our Cube serial commands to deploy our manifest files I will just write three different steps here for each of these files using the environment from the actions hub and the tour Cube CDL the first one will just run the cube serial apply command at the deployment file this will just run and deploy the application the next command will apply the service object and the last one the Ingress object that should be all that we need so now that we have everything ready let's commit these changes to the GitHub repository and then we just need to push it again let's go into the GitHub actions to see if the runner has executed all of these commands as expected there you can see our job is currently running so let's have a look inside what's happening but as you can see the initial deployment workflow just exited with an error if we take a closer look at what's happening here inside the drop that was executed and the step that was failing you can see this is one of the things that I also wanted to show you because I'm running the kubernetes to the tree on my cluster which I've mentioned in the beginning of this video and this tool prevents GitHub actions to deploy anything on the cluster that violates its policy just a few words more to the tree this is an amazing tool to secure kubernetes and prevent misconfigurations when we'll take a closer look at the activity lock in that tree you can see why it failed the policy check on cluster demo 1 and that's because the container image that I'm using in my deployment manifest doesn't have a pinned tagged version so I'm not allowed to use the latest tag in the container images and that totally makes sense because when you are using the latest tag on your automation scripts you never know which version it will deploy and it could actually have some breaking changes in your versions that are not compatible with your configuration so it totally makes sense to avoid using the latest tag in your deployments and only do updates to newer versions manually I've by the way recently made a more detailed video about the tree and how to use it to protect all your kubernetes clusters with policy checks the tree gives you the opportunity to use their entire feature set on one of their kubernetes clusters completely for free so that's really amazing if you want to learn more about it I'll link you my first tutorial video in the description down below so check it out and just test it in your home lab okay so like I said the policy tells us to change the latest tag on the deployment container image to something more specific so let's just check what is the latest version of the nginx web server on the docker hub put this version tag into the Container image and now our GitHub action should run correctly we just need to commit this new change to the repository again and if we now go to this job you can see all the steps were executed correctly that is a good sign if we execute the cube CDL command to get our deployments and resources in the nginx demo 2 namespace that's what GitHub actions should have deployed you can see there is our new deployment that was created by the GitHub Runner and here is our web server Isn't that cool I haven't done anything manually I mean except all the configuration for the GitHub Runner and serve the workflow if you're not saying well that's a lot of work I would have done that much faster by just executing these three simple commands on my Mac yes you're right but automating your project is always a lot more work up front don't be worried you'll get the payoff later in the process because if we know need to make any changes to this application for example let's try to change the domain in the Ingress object to maybe second nginx Cube table one yeah just for testing it I now can just commit this new change to GitHub push it and after a few seconds this change was automatically applied by the GitHub Runner without having me to authenticate to the cluster and execute any comment and just like this you can automate a lot more stuff also you can fully utilize all the tools from GitHub like roll back any changes when something doesn't work it takes some time to get into all of this I agree but for managing larger projects I think this is such a useful tool to be honest I'm pretty excited about this because I can use it in so many different ways in my home lab I could combine this with terraform to deploy all my infrastructure or I could automate the deployment of all of my home lab applications and servers or run regular Security checks on them I will need some time to figure out how exactly I'm going to use it in my home map these are just some ideas that came up in my mind if you have any other ideas just put it in the comments below and if you're interested I'll show you a couple of projects I'm doing with it in a few months anyway I think that's enough for today I'm sure you know want to try it out yourself please let me know if you have any trouble or questions and as always thanks everybody for watching I will catch you in the next video take care bye bye

Channel: Christian Lempa

Views: 30,509

Keywords:

Id: tIWDpG7sNTU

Length: 23min 53sec (1433 seconds)

Published: Tue May 02 2023