Using Docker Multi-Stage Builds

you probably already know that in order to run containers we need to build container images and the most commonly way to define how images are built is docker file what you might not know is that we can use docker file for much more than building container images we can use it to build our binaries we can use it to lint software we can use it to run static analysis we can use it to run tests and so on and so forth there are many other things we can do as part of the process of building container images beyond building the images themselves and on top of that docker file is not used only to describe how to build images with docker but it can be used with many other tools it could be kaneko and quite a few others today we are going to explore multi-stage builds how we can create different stages so that we can streamline the process that ends with building an image but starts from the very beginning let's start with a very simple docker file and then see what the problems we are going to face with using something that simple and progress from there until we get to something that is beautiful i'm already inside the git repository it has all the definitions that i need that you need if you want to follow along and if you don't want to follow along the gist with all the instructions is in the description of this video so let's take a look at a very simple docker file which is called dockerfile dash simple so that you do not get confused this dockerfile is based on the alpine image which is very small it's not the smallest one you might want to go with scratch but for this exercise this should be good enough we are exposing port 8080 we are going to execute the command demo when the container based on this image is created we are going to copy the binary demo from my local file system into usr local bin directory inside of that container image and make it executable the order here is important because the things that are less likely to change are higher on the list and those that are more likely to change like copying the binary that is going to change with every build so that the process of building container images can cache you know the top layers and then only process those at the bottom anyways let's see whether we can build the image based on this docker file so it would be docker image build demo so we're going to call that image demo and since this specification is not in docker file but in docker file they're simple we need to specify the path to the docker file which is docker file dash simple and we need to specify the context where are the files located and i'm going to say dot because that means current directory wherever i am right now it pulled the image and then it tried to copy demo from my local file system into that image and it failed and it's normal that it failed because i never built the binary i never built the binary of this application now i could tell you hey how about you build the binary install go compiler execute this command and that command build the binary and then run this command again try to build container image again and then it would work but that is silly that is a waste of time since we can build binaries and we can do much more inside of docker files so there should be no need for anybody to have instructions like do this do that do this do that and once you're finished doing all those things then you can build the image no we should be able to have a self-sustained system that everything required for building an image including in this case this binary should be within the same definition and everything should be done with a single command so let's try again let's take a look at the slightly different docker file which is docker file dash effect this time the base image is going because we need go to compile the binary for this application now don't worry if you're not proficient with go the logic is the same whether you're using javascript java python this or that right you select the base image that contains the tools that you need in this case that base image is going the next two instructions are the same as before we are going to expose port 8080 if this image ever becomes a container we are going to make demo being the command that executes when container starts this is the same as before and what comes next is new i'm adding the whole current directory inside of the image inside the directory src then i'm making the directory the work directory so that everything else that happens from now on is inside of the directory and then i'm running unit tests because it doesn't make sense for me to build a binary if it doesn't work so first test it then build the binary and potentially later on we might go into functional testing integration testing and other types of testing that require run time but for now i'm focused on unit testing starting testing let's say and if tests are successful then we are building the binary here and putting that binary inside of usr local bin demo so that it can be executed and we are changing the permissions so that that binary is executable now if this works and we are yet to see whether it works and whether it works well this should provide everything i need to build the image i have all the tools i need inside of that container image i am we're copying the source code i am running unit tests i'm building the binary i'm making the binary executable and so on and so forth and the final result should be the image with my application so let's try it out let's see whether it really works so the command should be similar like before docker image build and we want to target as let's say demo and then file that contains the instructions how to build the image is docker file fet and we need dot at the end meaning the current directory is the context and let's see now it is pulling uh the golang image it should take only a couple of more moments actually that image seems to be big i'm not sure maybe that's a problem we're going to figure it out later whether that's an issue or no and let's see the process failed in the middle because the tests were not correct i intentionally make the test fail just to show you that actually testing can be part of building an image and it makes everything easy because it's a single command now i need to fake that i'm going to fix the bug by changing the source code of the application i'm going to open main go and just change the here it says fancy demo is false i'm going to change it to true i just made intentional uh problems so you see what happens when tests fail the process fails at that point as well so that's a good thing right uh if tests were not successful the image is not built and there is not much for us to do but to fix the problem so i'm simulating that i fix the problem and now i should be able to run the same command again and if the bug is really fixed it should continue after running the tests and build the image and build binary actually first and then the image and so on and so forth and that was successful now there is one thing that you need to understand the image is supposed to have a single binary in this case based on gold i mean go compile binary that should be relatively small and lightweight because there's not much more to this demo image is just a single binary or that's what it should be so let's take a look at the images we have so docker image list and then was indeed here here's the demo right we can see it and we can see all the information it was created a minute ago but look at this the size it is 875 megabytes for an application that is hello world it is a hello world compiled to an executable and that should be a few megabytes the reason that this is so big that it is almost one gigabyte is because we used golang as the base image so this image with that binary contains whole go compiler and everything else that comes with go i do not need that in production it also contains the whole source code of the application which was copied so that the binary could be built those things are not necessary they're necessary for the process to run they're necessary for building uh in this case binary and few other things but it is not necessary to have all those things as the end result of the image that should potentially run in production so the process works well it executes tests it builds the binary and does everything else that needs to be done but the end result is too big let's try to make it smaller much smaller and for that we are going to take a look at yet another docker file this will be an improvement of what we did so far now this docker file is almost the same as the one we used before the base image is going we are copying the source code we are making src the word here we are running tests and we are building the binary but what is new is another from instruction and what is very important to understand is that final image will be based on the instructions starting from the last from instructions so only this will be the final image everything defined before the last from statement will be discarded this will be removed when the process ends this is temporary on the other hand this is final this will be our final image and inside of that final image we are doing only the things that are necessary and that is to tell whichever orchestrator we are using to expose port 8080 make them will be the command and here comes the trick here comes the important part it says hey copy something from build and build is the name here of the previous stage so copy this file into this directory inside of this image we are basically creating two images the first one is temporary the second one is permanent and the second one will use something from the first one it will copy the demo binary it will discard everything else from the first image except the binary itself the application itself which will be copied into the final image and at the end we are making that binary executable so this is more or less the same as the previous definition except that we split it into multiple stages so let's take a look at the final result of this one and the final result will be docker image build and we are going to give it a tag demo we do not need to specify the file because docker file is the default it is assumed by default but we do need context with cheese dot and now it is doing whatever it needs to do and let's take a look at the images docker image list now the demo image that was almost 1 gigabyte before is now 17 megabytes this is order of magnitude smaller than what we had before what matters is that the final image is small while at the same time we have everything we need up to the point of building image itself defined inside of a single file and executed with a single command a single docker image build command gives us the ability to execute all the steps we need up to the point that creates the image itself now i should be able to run docker image build as many times as i want during the development of this application i could combine it with scaffold if you haven't seen scaffold the link is somewhere there so that it is doing executing that process continuously and we can put this command in our ci cd pipelines and just have it as part of our release process or whatever we're doing docker multi-stage builds are one of my favorite features i love it i use it all the time they've been around for years now some of you might be familiar with it some of you not i think that it is still highly underutilized especially given how awesome it is it simplifies local development i can just execute docker image build every time i need to validate that whatever i'm working on is correct it will build a binary run the test package lean to do whatever it needs to be done and end up with an image that i can deploy somewhere wherever i want that to be deployed i can execute that same command using exactly the same manifests in my ci cd pipelines and i do not need always to use docker i use docker on my local machine and then i would execute the same process using kaniko inside of my kubernetes cluster the definition is the same the process is the same but the tool is not the tool can be whatever fits specific scenario so if you're already using multi-stage builds you probably didn't reach this far in a video if you're not using multi-stage builds you have to use them they are absolutely awesome you

Channel: DevOps Toolkit by Viktor Farcic

Views: 2,899

Rating: 5 out of 5

Keywords: docker multi-stage, docker, multi-stage builds, docker multi-stage builds, container, docker image, container image, devops, devops toolkit, review, tutorial, viktor farcic, docker multi-stage build tutorial, docker tutorial for beginners, docker explained, multi-stage builds feature in docker, docker image creation tutorial, docker image explained, docker image creation, docker image size, docker image from dockerfile

Id: zpkqNPwEzac

Length: 13min 40sec (820 seconds)

Published: Thu Jun 10 2021