Kaniko - Building Container Images In Kubernetes Without Docker

how do you build container images now if you're talking about your laptops or locally the answer is most likely that you are executing commands like docker image build but let's move to the cluster how do you build container images inside kubernetes clusters as part of your continuous integration pipelines or any other processes that you're having because remember we are moving most of the things to kubernetes and that applies to building container images now within kubernetes clusters docker is a bad option no it's horrible option because and this is now important part docker cannot run inside containers using docker to build images within containers or inside kubernetes clusters requires communication to the docker engine running in cluster nodes even if that security concern is not enough docker shim the one responsible for communication between kubernetes and docker is deprecated in kubernetes 1.20 and it will be or it already is depending on when you're watching this removed from 122. so even if you're fine with simulating uh running docker inside of containers even though actually that's only a client that communicates with docker engine running on the node and does that communication through the socket and all that madness even if you're fine with all that which you shouldn't in any form or way docker will be removed now it is deprecated and soon it will be removed from kubernetes clusters that means that you will be using alternative container engines like container d and there will be no docker so you should not be building container images for using docker inside kubernetes clusters and even if you ignore me and you say hey victory is talking nonsense uh who is here to tell me what i should and shouldn't be doing docker will disappear from kubernetes so you must look for alternative method to build images so docker is not an option anymore within kubernetes it is gone kaput disappeared not anymore non-existent so the alternatives what are the alternatives now there are many alternatives over there i mean not many maybe 10 viable alternatives but today we will speak about the one that i like the most and that one is called kaniko it is a tool to build container images from a dockerfile just like docker does but the major difference is that kaneko can run inside containers and that means that it can run inside the kubernetes clusters there is no need for privileged mode there is no need for exposing any sockets there is no need for docker running on nodes of our cluster it does not matter really which container engine we are using for running containers what does matter is that kaneko can build container images inside containers and automatically inside kubernetes clusters so let's explore kaneko in 20 minutes or less actually we're not going to explore kaneko we're going to explore how it works with docker the problems with running using docker to build container images and one of the potential solutions which is kaneko all that in 20 minutes or less [Applause] to understand kaneko we need to understand the problem it solves and to understand what it solves we need to dive into the issue at hand and that the issue is how we build images and what are the problems with running docker image build so we're going to start with showing the issues with the way you might be building container images today and then jump into kaneko and see how it solves those problems and before i continue actually the first thing the most important thing all the commands everything i will execute is available in a gist that is in the description of this video so go to the description to get the gist with all the commands that i will execute so we need the demo application to practice with some source code docker file that we can build and all that stuff so let's open uh repository of the demo up github.com this is me with farsik and it's kaneko demo it's easy to remember right so if open command doesn't work in your case just copy and paste the address open it in your browser and you should be good to go here's the repo and i will i will not even show you what's in that repo i'll show you docker file later for now the important part is that you should fork it so fork the repo i'm going to select this user devops paradox and now we can see what's going on so the repo is forked i will not need we will not need browser anymore so we can close the browser all together and start building container images i forgot the repo i will need to clone it on my laptop and i will declare gh.org short for github organization or github user variable so that you can do the exactly the same so uh put here as a value of that variable whatever is the user or organization where you fork the repo in my case that is devops paradox and i'm going to clone the code to my laptop github.com that's it okay now we have something or in a few seconds something to play with this is kaneko demo and let's take a look at the docker file now one thing that is important is that i will in no way suggest that you move away from docker file that could be a separate subject but i will assume that you have docker file and you want to continue using it right so what do i have here files of my application source code what's or not it doesn't matter what doesn't matter for now is the docker file and docker file is like this very simple one it is multi-stage build that uses hugo because it's a huge application and then it builds it and the results of that build is passed to the second stage will be the final container image which is based on nginx as simple as it can get i'm not teaching a file i'm not teaching you any of those things so let's move on now if uh locally or in a cluster most likely the most common way most of you are used to build container images is by executing something like docker image build and then we give it some tag and let's say devops toolkit that's the name of the application and the context which is current directory and the image is being built right now papa it's building the image it's a small one it shouldn't take long and the image is built right that is how you're used to do it most i mean all most of us are doing are building container images from our laptops like that nothing new now let's try to translate that experience into uh running the same thing inside a container not from my laptop inside the container and for that we will need the kubernetes cluster and today i will use mini cube and i will start mini cube to create a new kubernetes cluster by the way what i'm going to do inside mini cube works the same in every kubernetes cluster so don't think that i'm showing you something mini cube specific what we are going to see works in every single kubernetes cluster that i use so far so no worries about that if you already have a cluster you can use that cluster you don't have to create a new one with mini cube and there we go i have a cluster and uh that's it that's all i needed now i mean that's not all i needed that's that's what i needed i need the kubernetes cluster now let's take a look at the file docker.yaml that is in that repo this docker is the simplest possible port definition i could think of that will show us whether we can use docker inside of a container so we have containers a single container it is based on official docker image and it is set to sleep for a long time so i'm i'm just going to run the pod with the docker in a container that will slip so that we can exec into that container and see what's going on so let me let me create that pod and see what's happening in the command this cube uh cube catal apply file name is docker.yaml and then i'm going to wait until it is up and running for condition equals containers ready i'm just waiting until the pod is running it's probably running already yeah it's already running cool now let's enter inside of that container and try to see how we can build images docker sh now i'm inside of a container the first thing i need i need to install uh git because i will need to clone the source code inside of that triple there we are and i'm going to clone the code that i showed you already before clear git clone https because remember i'm now inside of a container i don't have the code detect loan that is on my laptop inside of container so github.com slash with rsec you can change this to the username of your fork mechanical demo dot git and doing the same thing that i did on my laptop now inside of container i'm going to go inside of the directory of the local copy of the kit repo and i'm going to build the image docker image build dash dash tag devops toolkit and that's it and look at that error that error says that docker is not running inside of that container it is not there so it cannot build image all we have in that container is docker cli docker client and for docker 2 docker cannot run inside of container so it is running on the node of minicube and we will need to figure out the workaround how to get to the docker from this container to the docker on the node so let me get out of there and delete this was unsuccessful so this just demonstrated that we cannot build container images using docker inside of container without some workarounds so cube cattle delete file name docker and we are going to start over there we are bodies no more now let's take a look at a slightly different definition of docker pod and that is cut docker socket the difference here compared to the previous definition is that we are having now volume mount and and volume section so what we're doing here really in volumes we are mounting hostpath meaning something on the host on the node and that part is the socket docker socket and that is the socket through which client cli communicates with the engine that socket is used even when we run it locally on our laptops even though it's not so obvious so through this socket when we are inside of containers and we execute docker something some docker command of the cli that command will be sent through the socket to the docker engine running on the node instead of being executed in a container directly okay so cube cattle apply file name docker sock socket and then we're going to um we're going to wait why not wait wait for a condition equals containers ready of the pod docker just to make sure that is up and running it is up and running and we can enter inside of that container uh docker dash sh and do the same thing same thing as before apk add we're going to add git because the default image does not act with git and clone https github dot com slash whatever is the user where you forked and kaniko demo dot kit okay and now we can go inside of the directory of the clone repo and execute the same command as before the one that failed build tag devops devops toolkit and the context is the current directory and this time it works it works simply because now we are not using docker inside of the container instead we are using docker that is on the node and this is horrible for two main reasons first because we mounted the socket that means that anybody capable of running containers in our cluster containers themselves can issue any command to docker running on the node basically it would be relatively easily straightforward to take control of the whole cluster relatively easy and even easier to take control of a specific node in the cluster wherever that container is running so this is a huge security issue amazingly probably the if you're doing this this is the biggest security issue you have right now in your cluster guaranteed now the bigger if that's not if you say hey i don't care about security there is a bigger problem i already mentioned the very beginning docker is not supported anymore in kubernetes clusters so you cannot have docker if you want supported version of kubernetes inside of your cluster that means that all this will not work if there is no docker on the nodes right so docker is out this will not work no matter how tempting it is and this is most likely statistically you are almost certainly using this method to build container images uh because majority of people are so statistic maybe your exception right so let's go out cube cattle delete file name uh docker socket and start over let's see how we can we can build images based on something else and that something else is going to be canico so the pod is gone and we can look at how we can build container images without docker because docker obviously doesn't work it's removed from kubernetes clusters and uh you cannot find it inside of containers anyway so start over with the kaniko dot git file let's take a look at this spot this part is slightly a lot different than the definition we had before to begin with we are using the executor image from the canico project so before it was docker now it's something else yeah i'm using it in debug mode because uh it's simply more uh gives me more options than i normally have with the mechanic which is very short on what what can do in the official non-debug release anyways we need to specify a couple of arguments first of all we have the context and context in this case is git so that means that kaneko will pull some source code some git repo from this address in this case context can be different it can be gcs bucket google bucket it could be s3 or aws bucket as well it could be azure blob storage it could be local directory which is very interesting for ci cd pipelines it could be tar file it could be standard input input and so on and so forth so there can be many different contexts or in other words places where canico should find a source code of the application especially where it should find docker file and we are using it it will that means that it will clone the repository for us just like i did it manually when inside the docker container kaniko will clone the repo in this context and then we have destination where do we want to push this image to right we could only build without pushing but in this case by default if you don't specify otherwise uh it will build an image and push it somewhere in this case it is vfr sig devops toolkit which means that it is docker hub my user in docker hub and devops toolkit is the name of the image and the tag will be 100. so apart from docker hub which is in this case it could be gcr it could be or google container registry it could be ecr or elastic container registry from aws it can be any other registry as long as we know how to create credentials for kaneko to connect to that registry there are other arguments many other arguments that we could set uh for example the the one that is interesting to me a lot that is cache we could define where is the cache of the images or the build is faster and so on and so forth now just like the second example with docker we're mounting something as well but this time we're not mounting silly things like docker sockets so that we are completely insecure we are mounting in this case something called kaneko secret which will be placed inside of container in the dar in in a directory kanikos and the file dot docker and here in volts we can see what it is it is a secret and that secret basically it's supposed to provide kaneko with credentials how to push images to container registry whichever registry we are using so we are mounting a secret video credentials to let conico enable canico to push images assuming that we want to push images to canico now i'm going to open that file uh caniko.kit yaml and you i i will and you should also change open it in your favorite editor and change um a few things first of all this fifa six should be whatever is the username github username organization where you fork direct repo docs and the secondary farsight this one should be whatever is your username in docker hub use docker hubs for simplicity reasons i will use vivrc as well but as this you change it to whatever is your docker hub user so you need to change github user just like i did and docker hub user over here and i'm going to save that save that file and i'm going to push those files to the repo that i forked call me the shame changed what did i change registry info and i'm going to pull those files cool now before we run this pod and see how it works uh we need to do something and that's something is that we need to create a secret and for that i will use a couple of environment variables i already have those variables defined because they are very they're the type of variables i don't want you to see so i already defined variables i will not do them but you should define them yourself so follow what i'm doing export registry just three server should be https tatata dot index.docker.com v1 that means that secret will be pointing to the docker hub registry if you use a different registry change this to something else but not for the demo not not now and then we should export uh registry user i will not show you the value of my user because i don't want you to know you change those three dots to whatever is your registered user and export registry registry pass or password put whatever is your password of your registry again i already defined those variables i will not uh export them here right now and the last one is export registry email this is the email associated with docker hub change those three dots to whatever your values are and now we can create a secret cube cuttle create secret and the secret is docker registry and we're going to call them tech cred same as in in that pod and we need a couple of values of those secrets and that's uh docker server equals um registry server right and the next one is passing those environment variable docker username is registry user more docker password is registry pass and the last one is docker email equals registry email oh i forgot to declare this variable silly me i will change it to hardcoded value this is my image this is not a secret we have the secret created with credentials for kaneko to know how to push images to docker hub and now we can create that pod we are doing the same thing is what we did with docker but this time with the canonical cube cattle apply file name is canipo git.yaml and then waiting until it's up and running for condition equals containers ready and pod is kanika it's running and now let's take a look at the logs locks kaniko here are the locks it is cloning the repository right now it is actually a building already the image we can see here those are the locks of building the image it cloned the repo at the very beginning and now it is building the image and almost done there we are almost there actually it's finished it cloned it built the image and it pushed it right now to docker hub that's it it's running inside of containers it is not using sockets it is not using privileged mode it is doing what docker should be doing if it could run inside of containers so this is the replacement this is a vertical date for you building images inside of kubernetes clusters and that's all there is now i will show you one more file just a slight variation of what we did and that is kanikodiriamo this is the same one except with the difference that context there is different here context there is set instead of to git repo to the directory workspace this is very similar to what you would use inside ci cd pipelines because in ci cd pipelines you would clone specific repo specific branch specific commit yourself through your pipelines and then use the local directory local copy of that repo to build container image by using hanako kaniko and uh in that case you would use directory or you could use others other other contexts that you already saw so that's kaneko it's once you figure it out and it doesn't take long actually the documentation leaves a lot to be desired but once you set it up which probably which takes maybe an hour or something like that it is the same basically is using docker but within kubernetes clusters so from user perspective from developer perspective it is still the good old docker file and then when you hook it into your ci cd pipelines or whichever processes you're using to build container images from user perspective is the same i give a docker file and then some pipeline or some process is building my images it's just that this process now happens to be using kaniku instead of docker for many different reasons security first or second and most important one docker is no more in kubernetes gun kaput so docker is still a great option probably the best option to build container images locally so when you want to build container images on your laptop you use docker docker image build and all that stuff jazz works perfectly there is no need to change because the interface is still the same we are using docker file no matter whether it's locally building container images or inside kubernetes cluster so on your laptop docker image build works great inside of kubernetes clusters is not an option and i will argue that kaneko is probably the best candidate to replace your docker socket shenanigans with something that is actually really good so try kaneko and uh before that before you start trying conically exploring please subscribe to this channel it means a lot and if you like this video hit the thumbs up button so subscribe to get notifications hit thumbs up if you liked the video do it even if you didn't anyway if you want to support this channel uh there are some books and courses that i and darin actually that's incorrect darin and i uh are making so please consider getting one of the books or the courses uh that is used to support uh actually to purchase red bull and pizza so that i can work late nights and weekends to create videos like this thank you so much for watching see you next time cheers

Channel: DevOps Toolkit by Viktor Farcic

Views: 13,027

Rating: 4.9746432 out of 5

Keywords: build container images, kaniko, Docker, container, container image, container images, Docker image, build, build container, Docker build, Kaniko, container engine, Docker engine, priviledged, socket, Docker socket, ContainerD, Dockerfile, Kubernetes, k8s, Kubernetes cluster, k8s cluster, kaniko tutorial, docker vs kubernetes, docker image container, docker image creation tutorial, docker image and container, docker image registry, docker image build, docker image creation

Id: EgwVQN6GNJg

Length: 28min 47sec (1727 seconds)

Published: Wed Dec 09 2020

Reddit Comments

[u/andrewrynhard]

With all due respect, this is not accurate. You can absolutely run docker in containers. In fact it is officially supported and called "Docker in Docker" (DIND). Docker in Docker is a bit of a misnomer since the idea pre-dates CRI. We run it in kubernetes with containerd as the CRI.

[u/yebyen]

I have built some stuff that depends on Buildkit features (cache volumes, ssh agent attachment, ... what else I don't know, at least this much) and I was trying to figure out how this deprecation will affect us; I also just heard about Buildkit extension to Kubectl, released by Tanzu last month, which I thought might be the answer. I did not investigate Kaniko or other solutions because I assume they will not support these specialty features of Buildkit that I need.

(If you addressed this in the topic video, mea culpa, I just haven't watched this one yet.)

Do you have any interest in doing one of these 20-min presentations about it? https://blogs.vmware.com/opensource/2020/11/17/buildkit-cli-for-kubectl/

Your talks are great! And your production studio setup is superb as well, thank you for these helpful explanations.