Configure GitLab CI Runner with Docker executor using AWS EC2

is your pipeline failing because you have issues verifying your gitlab account or do you simply need more power and minutes and the three minutes from gitlab.com are not enough and may have a possible solution for you if you have a linux server running somewhere that could be a great starting point in this tutorial i'll be using aws ac2 to show you step by step how to configure your own gitlab runner and run jobs on your own infrastructure yes we'll have to type in quite a few commands and use scary tools like vim but don't worry you don't need to be a computer genius to get this to run i'll guide you through the process step by step until your runner is registered with gitlab and you can see your jobs running this sounds like the kind of tutorial you are looking for let's dive into this so before we begin here is an overview of what we will do in this tutorial we're going to create first of all a new virtual machine on aws we're going to use here the ac2 server and on that virtual machine we're going to install the gitlab runner and we're going to install any required dependencies on the virtual machine as well finally we're going to configure the gitlab runner to work with gitlab.com please bear in mind that this is a fully manual process that is ideal for beginners who want to get started with gitlab ci this is by no means a production rate is set up and you shouldn't use it for production we will do everything from a browser so you don't need to install any software on your computer the first step that you need to do is to create an aws account and if you're a new aws user customer wherever you put it they offer you an free tier so there are some services that you can use for free for the next 12 months but you still need to provide them with billing details so what i'm showing you in this tutorial should be absolutely free for you but just bear in mind in case you make some settings or use some other services you may see charges in your account so i cannot fully guarantee that the first thing that you want to do is to start with the ec2 service and here we're going to create a new instance so somewhere in your account you should see launch instance and what you're going to see here is a list of amazon images that we can use and we're going to use simply amazon linux 2 which is based on red hat and that is a great choice for what we're trying to do next is a step where depending on what you select you may encounter some additional costs by default you will see here that i have the d2 that micro instance selected is a very small instance this has only one virtual cpu and only one gig of memory but for basic jobs in getlab ci that should be enough however if you think you need more power i highly recommend that you go a bit higher for example t2 large or anything else in the beginning it's totally fine to start with something small to see exactly how much you need and later on you can also add additional instances so that is absolutely fine i'm gonna go ahead here review and launch and without modifying any other settings i'm gonna click here on launch what's important here is that you will get this window with creating a security key in order to access this instance so even if we're not gonna use the key itself we still need to generate it because you will need to log in so we're gonna create here a new key pair and feel free to name it as you wish simply gonna call it gitlab runner and i'm gonna go ahead and also to download this key pair once you did that you can click here on launch instances this will take a minute or two to complete so feel free to grab a cup of coffee and take a short break i'm going to check here what the instance is doing by clicking on view instances after just a bit you'll see here that instant state is running so everything seems to be working properly and what we can do next is to select this instance that we have and then you will see here this connect button and this connect button will allow us to connect directly to the instant itself without downloading any additional software or configuring any keys locally so that's a very handy way of connecting to it whatever default settings we have here they're absolutely fine so all you have to do is click on connect and gitlab has then connected to this instance and now we have a terminal window directly in our browser now bear in mind this is still being developed and it's not perfect so there are a few things that work a bit differently than you would have in a normal terminal but especially for beginners that is a great way because it really reduces a lot of the complexity that you normally have so here we'll be able to type commands and the first command that we want to type is to update dependencies so i'm going to use here sudo yum because this is the package manager that we're using we're providing here dash y to make sure that we're not getting any questions and we're gonna type here update we'll find all the commands that i'm typing here also in the video description so if it's easier for you just to paste them from there that's also fine okay so we don't have anything to update we're gonna start after this installing the first dependency so it's almost the same command as before but this time i'm going to write here install and the dependency is called git all right so let's make sure that git has been properly installed so we're going to use here git dash version and if we're getting the current git version means that we are on the right track clear this window and start from the top next we want to install docker now ideally we want to have a gitlab runner with a docker executor so of course if we don't have docker installed it's going to be very hard to use the docker executor so for this we're going to type another command going to start with sudo and this time we're going to use the amazon linux extras and from here we're going to install docker so it seems that the installation was successful i'm going to clear again the window and the next step is to add our user which is ec2 dash user to the docker group so for that we're going to use user mode and we're going to add the user to the docker group and the user is ec2 dash user okay we're not getting anything back from this but if we're typing the command groups and we are checking in which group is the ec2 user we should see that also is part of the docker group finally in terms of docker we will also want to make sure that the docker daemon is started automatically otherwise we're going to get some issues so here we're going to use the system control enable docker this will ensure that docker is started automatically and finally what we should do is to restart our instance so we're gonna reboot here and this will take a few minutes maybe to to restart but we don't need to do anything we're gonna lose connection for a while but we can re-establish connection by simply refreshing the browser window in a few minutes so just take another short break and come back hit that refresh button in your browser and then you should be able to reconnect to your instance all right so let's go back click that refresh button and try to re-establish the connection and we're getting back the prompt so it means that the restart has been efficient and we're back in business so the last thing that we did was to install docker and to make sure that it's starting automatically so if it has been started automatically then we should be able to run a basic docker command so we're gonna use here docker run and the most basic docker image that we can use is hello world and this will ensure that our docker installation is successful so we're not been able to find this image it's being pulled but then finally we're getting this message here hello from a docker so it means that this point docker has been successfully installed here now we have installed a lot of dependencies but we still need the main thing and that is the gitlab runner finding the right address to download the gitlab runner is not so easy so you will find this in the video description and i'm going to simply go ahead and paste it here so it is a few hundred megabytes inside so it will take a few seconds to download this but afterwards we should be up and running right so we can see here if we list all the files that are available we're gonna see here that we have the gitlab runner so in order to actually start the installation process what we need to do is to use sudo rpm dash i and then i'm gonna simply right here get lab runner and with a tab i will automatically complete this and hit the enter and we're going to start the installation process while the installation process is running we're going to go in gitlab and we're going to prepare a few things here in order to use a runner with gitlab you need to register it with your account you can do that on a project level but what i recommend you do is to do it on a group level so if you haven't used any group so far probably now is the best time to go ahead and create a new group and we can call the group whatever you wish so i'm gonna simply call it the gitlab runner group but probably this is not the smartest name so don't don't use absolutely the same i'm going to also make sure that this is a private group and the rest of the information here is not so important so just going to go ahead and create this group and inside this group you can create new projects or you can move existing projects into this group so just keep that in mind the part where i'm getting is the settings so a project but also group will have this part with settings if we go in settings ci cd we're gonna see here something with runners so we can expand this part and we're gonna see a few things here so first of all if this is a validated account we have access to shared runners but we don't want to use the share to run us from gitlab in this project so i'm going to go ahead and disable this and what we actually want to do is to create a group runner and we already did a few steps so like starting to install the runner and then we need to also register the run so this is just the page that we need to see here at this point just to make sure everything is working properly let's go back here and what we can see well essentially we don't see any errors so that's a good thing it means that this step has been successful so in this regard we're gonna start registering our runner so the runner has been installed on our virtual machine it's currently not doing anything so what we want to do is to tell gitlab hey we have here a runner that we want to register with this group and essentially you can send any jobs to this runner it's available here so in order to do that we need to go through this registration process i'm going to type here the command sudo git lab dash runner register and this is practically something that we need to do step by step so we're going to get different questions so first of all we need the gitlab instance url because i'm using gitlab.com i'm going to get this information from here and you will see here the address so i'm going to simply click here on copy url go back here and paste this and hit enter then we need the registration token this is also being displayed here so i'm going to go ahead and copy this token as well we can enter a description for this runner runner on aws a c2 but of course the description is optional but it can help you if you're managing multiple runners understand like where they're running and why did you create them you can also assign text to your runner we're gonna skip this because we can also manage this directly from gitlab so there's no need to do this right away so if to skip just hit enter you can also enter an optional maintenance note so we're gonna hit enter again to skip this and now comes the most important part and we need to specify the executor and there are different executors and just in case you're a bit confused about what is with all these executors i'll link you the video description a blog post that explains all these different executors now in most cases especially if you want to use gitlab as you have it on gitlab.com what you want to use is a docker executor the next executor that you may want to use is shell but in this case i'm going to only specify the docker executor because i want to specify a docker image in the pipeline configuration i don't want to rely on any dependencies that are on the virtual machines so this decouples our jobs and pipelines from the infrastructure where we're running our jobs so i'm going to simply type here docker and now we have the possibility of specifying the default docker image generally on gitlab.com ruby is the default image but i just like to specify something small so i'm going to use here alpine and what we're getting back is the runner register successfully so that's already a very good thing let's clear this and check if our runner is running properly we can use here again the command gitlab runner and we can check the status so it's telling us service is running giving us some information about the platform but at this point we can go back here in gitlab and we can refresh this page and what we want to see in runners is available runners you see here the description that we have we have the version of the runner where the runner is working we have the possibility of editing some settings just in case we want to change something about this runner so i told you before you can still add some tags but this is what we want to do at this point in terms of tags just in case you have specified tags it's also important to have this enabled run untagged jobs just in case you have jobs that don't have any tags in their configuration you still want to run those as well just to keep that in mind in case your jobs aren't running so so far so good i think we're making great progress and by the way if you are happy with this tutorial and you think you're learning something new you can support me create more tutorials like this one by giving this video a like and subscribing to my channel it really helps a lot now let's continue with the rest of the tutorial where we're going to go ahead and actually create a job and make sure that this properly runs we're going to go through different scenarios there are a few things that we really want to make sure that they work properly i'm going to go back here inside the group itself and i'm going to create a new project and it's going to be a blank project so i'm essentially testing the runner just to make sure that everything is working properly i'm not going to initialize here with the readme i'm simply going to go ahead and create this project let's click here on new file and this will open up the web ide and what you want to do is to define the gitlab.ci configuration for a simple job so let's imagine here that we are trying to build something so i'm gonna call here the job build and we're gonna use image alpine so this will test if we have the right executor and here inside the script we can have a simple script so for example we can put some text into a file so this will be echo hello and we can put it in a file called let's say index.html doesn't really matter so much and what we can also test if we define for example artifacts if they're properly saved we're gonna simply define this file as an artifact right so we can go ahead commit these changes i'm going to commit directly in the main branch and this should already start the pipeline let's go inside the project here and we're gonna see here job testing the runner and the pipeline has been started so we can click on it to see what's going on and our build is already done so that already looks pretty good so let's try to understand what has happened here just to make sure that we are right on track so we're getting here in the beginning information about the runner this is the runner version that we have it's telling us on which runner this is running so that's important information we can see that we're not using the shared runners from gitlab second of all we can see here that we're using a docker executor with the image that we have specified this image is being pulled the environment is being prepared and so on git is working properly so this is why we have installed git because we're working with the git repository we have executed this command created a new file and then also the artifacts have been uploaded so that's pretty important step right there so we have job artifacts we can take a look at them we see that we have this file so from that perspective everything seems to be great now the problem usually occurs when we're trying to work with docker and when we're trying to build a docker image so let me show you what i mean by that i'm gonna expand here a bit the example that we have and i'm gonna simply go ahead and paste an existing configuration that i have i'm gonna find it here in the video description as well i'm just going to go ahead and select everything remove it now it's going to have here two stages build and package so in the build stage that's pretty similar to what we did before but then when we're building the docker image this is when things get a bit tricky because we're using here docker and docker this requires some special permissions and additionally we're going to require here also this variable we are overriding some configuration that will ensure things are working properly and what we're doing here is a few things first of all we're logging into the docker registry this is where we're gonna store our docker image then we're building the image pretty simple just the latest version and then we're pushing it so let's go ahead and try this out and see if it works or not again i'm going to commit here to the main branch and i don't want to go ahead and create a new merge request so let's jump back into the job and see what the pipeline is currently doing so the first stage of the pipeline build has worked without any issues and then we're going to take a look at this current one to see what's going on here so as you can notice here something is wrong something doesn't work properly we're getting here some issues so in order to understand what's happening here we'll see here that actually our problems start from the beginning so we're having some issues here with let's see what what's the problem we have some health check error issues we're pulling this docker image and then finally we're getting something like we cannot connect to this image so that's a bit the issue that we are facing right now and one of the reasons why this is happening is due to the fact that this docker installation that we have doesn't have some elevated privileges that are required for actually building the image so if we're looking here a bit on this part with the errors essentially you're gonna see this mount permission denied are you root so this kind of like doesn't really tell us like what's the problem but i can tell you what the problem is and the problem is that our gitlab runner is not running in a privileged way in order to get that to run as well there's still an additional step that we want to do but i don't want to scare you or anything i'm just going to explain a bit why these errors appear and what is the solution to it so that in case you see it you don't freak out and you know exactly what needs to be done i'm going to jump back here inside the terminal and what we want to do here is to edit a file which is the config.pummel file so and use here text editor which is vim and just in case this terminal doesn't respond anymore i'm just going of go back and hit that refresh button it happens from time to time i told you in the beginning that sometimes you know a few things are a bit buggy so no worries about that so sudo vim this is the fancy text editor that a lot of people are scared about but i'm going to take it step by step and we're going to go here in atc gitlab runner so this automatic text that i'm writing here whenever you're writing a few letters you can just click on tab and that will be completed for you so autocompletion and the file that we wanna edit is this config dot pummel all right so we have everything in place here and this is the contents of the file it's not huge we can use our cursor to navigate around here so your error keys on your keyboard can be used to go up and down and left and right so that's one thing so what we want to do here is to go a bit all the way to this line where it says privileged we can simply navigate and you will see here the value is false so what we want to do here is to override this now if you start typing that's not a good idea because it will not change the file we need to enter an edit mode so in order to enter this edit mode or officially called insert mode we're going to press i on the keyboard and you should see here right at the bottom you will see here insert so this telling you you are now in insert mode so that's all right we can still navigate with our keys with our errors i'm going to go ahead and remove here false and we can simply write true right now in order to exit the insert mode we will not press i again because that will simply insert more text here this time i'm going to press the escape key and you will see here insert mode has been disabled it doesn't appear anymore and then here comes the tricky part what we want to do is now exit this file and bear in mind we haven't saved any changes so what i like to do is to simply type column and you will see here it appears right at the bottom of the screen and i'm gonna type x so this means save and exit so column x and then i'm gonna hit enter and you'll get a message here that the file has been written right so this is exactly what we want to see so so far so good the next step would be to restart the runner so we're gonna write here sudo gitlab dash runner restart because we wanna have the gitlab runner start again with this configuration that we have changed here i'm gonna try again status so we see service is running so this didn't take very long to do and then we can go back to our job that failed and we don't need to make any changes we can just hit this retry button and then job will restart and hopefully fingers crossed this time everything will work out just fine what you can see already is that the job is still failing but that's absolutely all right this time we're going to see here that of course we have a lot of logs but just up to this point we don't have anything that looks similar to this one it's actually telling us where the docker daemon has started it's telling us on which port it is listening and everything so from this perspective everything is working just fine we can take a look here and to understand what has happened so docker build has failed and the reason is we don't have a docker file so yeah that's a fair reason for this to fail without a docker file we cannot actually build anything so we're gonna go ahead and create a simple docker file so let's start here from nginx and i'm gonna simply gonna go ahead and copy the index dot html file i'm gonna copy it into user share engine x html folder that should be a valid docker file that we can use the most important thing about these pipelines is do not freak out when you see errors but to try to understand what has happened before how do the logs look in the previous job how do the locks look in the current job what's the reason why did this job fail try to understand as much as possible try to make sense of the logs sometimes they are weird and they're a bit hard to understand in most cases they contain some hints like what has happened why did it feel like in the previous example you know i haven't specified the docker file absolutely fine that can happen will not work without that docker file so we're going to wait a few more minutes for this job to complete just to make sure that we have everything up and running so i can already see that things are looking much better the image has been built and now it is being pushed to the repository so yeah congratulations the job has been successful probably if you look inside the repository we're going to find that image there so i would say that with the current configuration that we have we have really managed to properly test this runner to make sure that the main functionality that we need is working properly and that is we have a docker executor so we should be able to see the docker image that we have specified here our artifacts are being pushed from one job to the other so that's working without any issues and additionally we can properly build docker images because that's also an important functionality so with that being said this is the tutorial for today i hope that you have managed to follow along but just in case you're facing any issues or any problems do check the video description because i'll be putting there any updates also all the commands and all the instructions that you need in order to get this to run as well on your aws instance i hope this was useful if that was the case give this video a thumbs up subscribe to my channel really helps a lot any comment any thumbs up any subscription helps me create more content like this and dedicate my time more to youtube and to helping you guys manage and understand this kind of things so thank you for watching and i will see you next time you

Channel: Valentin Despa

Views: 28,365

Keywords: gitlab ci, gitlab runner, gitlab runner docker, gitlab ci runner, gitlab, aws, aws ec2

Id: HGJWMTNeYqI

Length: 28min 23sec (1703 seconds)

Published: Thu Apr 21 2022