Build & Use Custom Docker images in your GitLab CI/CD pipeline

when you build your cicd pipeline with gitlab the public images from docker hub don't always contain all the dependencies you need so quite often after starting a docker image you still need to do a bit of setup in your pipeline this approach adds a lot of boilerplate configuration that can even slow down your pipeline in this tutorial i will show you how easy it is to build a custom docker image with all the dependencies you need and push it to the gitlab container registry and later on use it directly in your pipeline [Music] i'll be using gitlab.com and their shared runners for this tutorial if you wish to understand the principles behind this approach i highly recommend that you first use gitlab.com and not your self-hosted gitlab infrastructure so this is currently the job that i have it starts here with the base image which is python it just prints out the different versions available and in this case i'm assuming that i want to do something with pi test so for this reason i'm using the python package manager to install pi test and then i'm just checking again if this has been installed properly so if we take a look at the gitlab job we'll be able to see here which docker image is being pulled and then the script that i have added essentially uh installing pi test and you will see here it is downloading by test and then it's printing out the versions this is a simplified example but it shows like what you would face normally when you're starting with a base image and you need to install all these dependencies on top so before you can actually do anything with your job you have to do all these things in advance so in order to create a custom docker image the first step is to create a docker file and a docker file will essentially describe how that image should be composed so i'm going to go ahead in my project and add a file which is called docker file and again as an example it's gonna be a very very simple example this is not a tutorial on how to write really complex docker files i'm just showing up the principles so i'm gonna start here from a base image which is python and additionally i'm gonna add the command so essentially i'm using the package manager to install pi test so super super simple so let's go ahead and expand this pipeline and start building this docker image based on what we have and we want to use automation for this i'm not going to build the image locally and push it to the repository actually i don't want to do anything locally i just want to let gitlab do the automation things uh and have everything in a pipeline and i think it's totally fine to have everything in the same pipeline i will also show you a way on how you can make sure that this doesn't happen too often i mean the building of the image so let's start here with a job which is called build image and we'll have as a dependency docker but this will be not enough to build this image and to push it to the gitlab container registry we also have to use docker in docker this is essentially like when you have docker installed locally you open up your terminal type in docker version this is working but when you're trying to build or to do something else it will tell you that docker is not running this we need docker in docker is essentially the docker daemon which will be available as a service so i'm going to write here services i have to write it properly and the service that will start is docker docker in docker so dint is docker in docker okay so then we can go ahead and write the script and to understand a bit like what we're trying to do we can also go directly to gitlab and you will find here under packages and registries this will be the container registry that we are trying to use and at this point it will tell you that the container has no images stored for this project and will give you here a couple of commands that you need to run and will tell you exactly what you need to do to log into this registry and how to build an image and how to push it these are a great starting point and we're gonna use something very similar to this let's go ahead and copy the first command go back to our script and if we try to do this we'll be prompted to enter our username and password and of course because we're doing this inside gitlab ci we can take advantage of a bunch of predefined gitlab ci variables so if you're not aware of this gitlab offers a bunch of predefined gitlab variables that you can use directly in your pipelines and some of the variables allow you to have for example the registry password so if i'm searching here for ci registry you will have here the registry name the registry user and also the ci registry password so this will be automatically injected you don't need to do anything you don't need to define any variables these are available there and i think this is also one of the reasons why i think it's good to have this only inside gitlab ci so going back here all we have to do is write something like docker login where do you want to log in we're going to log into ci registry and we also have to specify our username and password so i'm going to specify here the username this will be the ci registry user and we can also specify the password with minus p but this will typically give a warning in the logs i don't want to see any warnings there so what we're going to do is we're going to simply pipe this value and use it from the standard input so we're going to do here something like ci registry password i'm going to pipe it to darker login and additionally let me make this a bit bigger and conditionally tell docker login by the way get the password from the standard input so this is the first step here we are logging into registry otherwise we'll not be able to build anything let's see what's next so the next step is of course to build this image i'm going to copy the command that you see here and again this will give us the command that we need to run which is docker build we're creating here a tag which will be essentially the latest in this case i'm only going to build the latest version i'm not going to get into tagging i will assume that whatever i have in the docker file is whatever i need at this point we can leave this as it is right now but again instead of having hard-coded values in our pipeline we can use again the ca registry image so i'm gonna use here the variable ci registry image and by the way if at any point in time you're a bit unsure like how do these values look like simply add to your pipeline something like echo and then the value that you're unsure about don't just use values because by making assumptions in terms of what they do and how they work also try to get an understanding of what's going on there okay so we're building this docker image and finally let's copy the last command so this will be docker push and essentially gonna push this to the same to the same registry where this image is so this will have the full registry name and the full registry image so as you can see our entire script doesn't contain any references to our project or to whatever else we have inside here so this makes this configuration very portable so let's commit these changes now and see how the pipeline does so if you're going to the latest pipeline we can go inside the build image job and we can see here that everything has been completed successfully there are some warnings but this is outside of the scope of this tutorial and then we can go to packages and registries inside a container registry and we'll be able to see here that we have an image that the latest version has been pushed about one minute ago so essentially we have built the image we have pushed it to the gitlab container registry and we can now use this image inside our pipeline we also seen that if we go here in our pipeline that we have two jobs running in parallel and essentially we don't want to build this image all the time so what we're going to do is we're going to define a schedule and we're going to build this image only let's say once per day according to our schedule and we're not going to run this with our pipeline all the time so the next step would be to define some rules here and the rule would be essentially we want to make sure that the pipeline source is schedule so ci pipeline source equals equals schedule okay and additionally what we will be able to do is to add here an additional job let's call it run tests and this run test will be pretty similar to what we had here before so for example instead of the python image we're gonna use the image that we have built we're gonna remove pi test just to make sure that this is still working and additionally we're gonna also add this rules here so i'm going to change here the rules this should not run on the schedule so when we're running the schedule we only run the build image same story for the original job just gonna keep it for for reference purposes and what we also need to do here is to specify the image now at this point this is totally up to you if you want to have here a hard coded value or if you just want to use the ci registry image it's totally up to you if you think this is going to cause some issues then it's better to have a hard-coded value you can also use this image between different projects so you can have if you want a standalone project that builds an image and multiple projects can use this image in that case you have to use a hard-coded value for that image here all right so what do we have at this point we're still keeping the original job we're not running it on a schedule we have the new job which is using the image that we have pushed and it's always because you only have an image which is the latest image we're going to always pull that one and the last step here now is to go ahead and actually create a schedule so now let's push it but also go ahead and create a schedule so from gitlab inside the job cicd schedules new schedule let's call it build docker image and we can run it every day just save it for debugging purposes if you wish you can simply manually trigger it once so let's go back to the project and see how the jobs are doing we'll see here that run test works properly so what's going on here we'll see here that we're now pulling this custom docker image we're not installing anything anymore and pi test is then available because the docker image contains this runtime here 39 seconds let's take a look at the older job this is 40 seconds so as you can see it's not a huge difference in terms of the time it took gitlab to download this image because essentially they have a pretty similar size and what's doing there are essentially the same steps but essentially this is how we can do it finally i just wanted to point out that not specifying a version for your docker images is not considered a good practice as it will always run the pipeline against the latest version of the image so for example here for the docker images that i've used go to docker hub and use a specific tag for the images that you need for example my gitlab ci pipeline should look more or less like this with the versions being specified and if you are anyway at docker hub do take a look at the size of the images you're using the smaller they are the faster your pipeline look for tags containing alpine or slim these are versions with fewer dependencies which you may not need anyway if this tutorial was helpful and you want to help me grow this channel give this video a thumbs up leave a comment below and consider subscribing thank you very much for watching and i will catch up with you next time

Channel: Valentin Despa

Views: 92,548

Keywords: build docker image, dockerfile, docker, gitlab ci, gitlab container registry, gitlab build docker image and push to registry, gitlab use custom docker image, gitlab registry docker

Id: 7I6tHw68DMQ

Length: 14min 9sec (849 seconds)

Published: Thu Nov 18 2021