Automate your Docker Build/Test/Deploy pipelines!

hello and welcome to my talk my name is brett i do things on the internet i make videos i teach people i train i consult everything containers and devops that's kind of my gig but today we're focused on automation with docker which has been one of my favorite subjects for a couple of years now because things are getting so much better and easier for us so i wanted today to focus on the ci workflows or the build test and ship workflows that we all have to deal with so the first thing you're going to want to do after this talk if you want to get into these materials is all these examples are in a github repo so fork clone it whatever you want to do it's over at brett.show allhands22 that url first i'm going to give you a quick background on the devops platforms and how i see them today since i used so many of them through all of my consulting over the last decade then i'm going to give you three different diagrams to look at we're going to break them down talk about the different workflow parts of them but the diagrams are really just meant to help you get a sense of where you start when you're sort of new to automating things like docker builds and testings and pushes of automation and kubernetes and all these things right so there's the basics then we're gonna go to intermediate then we're gonna go to advanced and then finally the bulk of this talk is gonna be showing off the workflows getting into the details of the yaml so let's jump right into part one which is sort of a background on how i see things in devops automation space now it's no secret that automation is one of the key principles it's almost like the backbone or the glue of devops in fact it's the only way we're able to achieve higher levels of work we're able to do more things with more systems and that's why it's so important to me and why i'm talking about it today now one of the trends i'm really seeing in this space because there are dozens if not hundreds of tools that help you automate tasks in this world of automation and you notice i'm not saying cicd a lot i'm saying automation because really it's everything and they're not all going to fit in the cicd boxes of labels so to me it's really just an automation platform you might have more than one but the trend honestly is that there's a consolidation happening not just companies buying up other companies which is happening but also that the automation platforms that are becoming the juggernauts or the ones you want to look at first are the ones that are integrated with where you're storing your code usually in git but if you're storing that on github git lab bitbucket those are the places where the automation is really growing rapidly so for me i'm all about github actions because i live in github if you're doing anything in open source you can't avoid github so why not do everything there already now github actions has only been around three four years but it's come a long way if you just looked at at the beginning and you just kind of gave it a pass because it didn't have all the features of your other favorite automation platform now the thing about github that they're going to have a hard time with anyone else beating them at this game is that it's increasingly becoming about the plug-in system for your automation at the end of the day we can all just write shell scripts right we could just automate everything with bash some of us have been doing that for 30 or 40 years and what's really changing now is that we're getting these plug-in systems where everyone else is doing the work in the open of making a previously complex task very easy to do in these automations and so it's almost a level of drag and drop now github having their open standards as they do if you've used some of the lesser known tools i'm not going to name any names but there are some really great open source projects out there that sadly their ideas were fine and great for using container native tooling or whatever but since they don't have this huge ecosystem of actions or plugins or extensions or whatever you want to call it they're really just a non-starter they're they're very limited and you're going to end up spending most your time in bash which we will avoid most of that not all of it but most of that today so if you're not using github actions totally fine you have to take the steps i'm going to use today and you're going to have to translate those into whatever your system is going to be able to do now sadly due to the increasingly advanced nature of a lot of these steps i'm going to be using today and how good they are in github you may have to do a lot of work to actually get them moved into your platform because it may not support all the things that i'm going to show you you can do now what should we automate with docker i have maybe like a top 10 and that's what we're going to go through is kind of the the big things that i think everyone should be doing and lastly i'm going to assume that you're using a pr based workflow because a lot of this is based on the two big actions in your daily work is taking a branch and making a pull request out of it and that kicks off a lot of these workflows that are going to be testing and automating those tests and the different levels and linting and all the stuff that's going to happen before the pr is merged and then once the merged we're going to do other things like deploy things and finalize stuff maybe push images with a special tag something like that so that's the workflow i'm expecting that we're all doing hopefully that's what you're doing all right let's look at our first one we're going to start right where i'm at we're pushing to our pr branch so we've made a custom branch maybe it's in a fork maybe it's in the main repo and i have created a pr for it that's going to kick off a workflow again it doesn't have to be github actions but i'm going to be talking about github actions today so you're going to kick off that workflow it's going to be probably through a web hook and you're going to build an image now that image is going to stay right there and then immediately be tested in a linear fashion this green line is signifying that this is typically your test some create some code make a commit push it it builds a new image it tests that new image and then you need to do some more things because you realize you didn't get everything done in the pr so you just keep doing this over and over and over again then once you've merged we're going to take that image that we had built and we're going to push it to a registry somewhere that is your preferred registry maybe stock or hub maybe it's something else and then you're going to deploy the server so that's a pretty basic workflow again it's two main phases now we're not here to talk about linting but let me just tell you something for a minute linting has started to grow up in the last few years and we've got a couple of good projects specifically super linter and mega linter there's other ones out there too not that they're not good but these are the two that i've looked at and i'm using right now heavily super linter all over the place and all my projects here's my soapbox moment for you linting is not something that we should be opting out of anymore it's not complicated it doesn't take a ton of work but it reaps so many benefits in the long term all the languages all the file types into a consistent linting experience and you want to be doing that in your prs on all your commits that are being pushed to your pull request so you're seeing ahead of time the linting that you need to fix before the pull request can be approved you should be doing that i am doing that in all my projects and helping all my clients move to 100 lending coverage now we're up to the intermediate level so hopefully you've graduated from basic and you're starting to get more advanced in your workflows so you'll see new things here like we're now linting everything you'll notice that these are meant to be in parallel so when we do a pr push we're seeing that we're building an image and we're linting at the same time so these might be different workflows or different pipelines the image build by the way is to a testing stage so i am a fan of using my docker file and my application dependency files as a way to control my testing frameworks versions of my testing apps i don't want those apps have to be installed on my build servers i just want them to build in a separate stage that i layer on top or if you're looking at the docker file on the bottom of the docker file i'm layering these stages and i'll have a testing stage that adds in everything i need for testing that stage will not make it to production but it's there while i'm testing that way i'm using my production stuff which usually goes at the top of the docker file and i'm doing that from the top down and so it's essentially the same image that's going to production but it's got extra stuff on top of it again using the docker multi-stage stuff now we're going to break out the testing a little bit we're going to first do unit tests the simpler ones that are usually pretty fast we're going to have integration tests we might use docker compose here to actually spin up databases or back-end worker processes that we need to test against with our integrations and then cv scanning so it's another step we can use something like sneak or trivi both free now this firewall i've put here is essentially the gate in github we know that as the checks it's all the things that happen at the bottom of your pr that all have to be green before you're supposed to click the merge button before the automation on the merge side that happens and that's all the production stuff that's rebuilding the image with that production stage really again it's just the subset of stuff that you had in the testing image and then we're pushing that to our preferred registry and then deploying to one or more environments and finally this is the coup de gras obviously you can even go more complex from here and have so many different levels especially when it comes to the deployment side where we possibly could be deploying to staging or phasing in applications in production using canary that's a lot of the cd stuff but right here what we're really focusing on is everything before that i find that this is the gray area right when we get to production deployments there aren't that many tools especially when we start talking about docker and kubernetes we've got things like flux argo cd pollumi fleet there's a few others but these tools are relatively rigid in their design so that we kind of know what we need to do there we know we're going to have a cluster somewhere probably running kubernetes we're going to have that tool in the middle that watches our github repos and then deploys stuff to the cluster but everything after you write code and commit it all the way up to the point of pushing it to servers that's a big gray area so that's why i wanted to talk about this today you can see here that one of the things we're doing is we're adding a new layer that's running in parallel that's doing an sast type of scanning this is code ql on github this is sneak code as well these are different tools that actually scan your code to see if you're writing it correctly and securely and you're not in creating your own vulnerabilities in your code so it's a little more technical requires more work to implement but if your team's willing to do that it's well worth it you'll now see that we're doing this all this stuff in parallel so we're adding even a new test where we're doing a simple deployment test to a temporary kubernetes cluster that we can build in seconds and i call that the smoke test basically can i deploy this image that i just built and tested and scanned but can i deploy it to kubernetes with the kubernetes yaml and have it actually come up and pass a health check that's an easy smoke test to do nowadays so we can do all that we have our gate right here this is the github checks that all need to go green as well as maybe have somebody review your work and approve your pr then once you've merged you're back to the production stage back to the registry and deploy so that that part really hasn't changed now let's get into some examples i know that this is the meat of the talk so we're going to rapidly jump into it all right in our first example we're really covering that first diagram so if you don't know anything about github actions it's okay this isn't actually going to be a full introduction of github actions but there's two main ideas you need to understand here is each file is its own workflow so one workflow is one file and in that file there can be many jobs each job has many steps now the workflow kicks off based on events these are github events and in this case we're only going to be really using two all day we're just using the pull request and the push what that means is is that this workflow will kick off whenever you push and commit to a pull request or if you do a push to the release branch in this case i'm using the default of main so anytime the pull request is merged into my release branch of maine it will also kick off this workflow and in this very simple example which is something from the basic diagram we just saw that i'm all i'm doing is one job the only two steps we're really doing here is we're logging into docker hub or your registry of choice and then we are building and possibly pushing the image you'll notice that by the way these both use a special action but this is the action format where it says basically the organ the org that is creating this action and then the name of the action and then the version of the action that's how we define each one of our steps also known as a github action so we have two actions these are the official docker actions there are technically six of them we'll be using four or five of them today but we're logging into docker hub which is the same as doing a docker login and then we're hard coding the tag so this would be the same as doing a docker tag on an image after i built it so this step will always build and then it will tag it with the two tags i've given it i've given it a comma so i can have a list here and i'm pretty simple here i'm just hard coding my docker hub image with two different tags and then i've got this if statement that basically says push this image to the registry only if it's not a pull request so how that works is is on a pull request this will run it will log into docker hub and then it will do a build the build is essentially our test we're not actually running other tests like unit test we're just building the image and that is all we're doing so if that fails in the pr you'll get notified but it's not going to try to push it to docker hub until the pr is merged that way we're not filling up docker hub with a whole bunch of testing images that are just temporary now let's go to example two we're gonna add to that the build kit cache now build kit you've probably heard about it it's been around four or five years it's basically the new building engine for building docker images the images are still the same standard oci format but it's how it builds it it's faster it has more features it does all kinds of neat things but if we enable some of them we can get an instant speed boost and benefits out of our build that we just did so you will see in this example this is the second one you will see by the way that i will highlight the beginning and end of new sections so that you can just focus on what's new because we're layering on the complexity as we go so here we have we're starting the build decks and what does that do for us well we're allowed to use caching now there are lots of different ways in buildkit to cache what am i even talking about with cache well in buildkit caching usually means caching the layers of my images while i'm building it so when i rebuild it it builds faster each time because i don't have to build all the layers right that happens normally on your machine but in a distributed building environment where you're not using the same servers every time to build your images that gets way complicated so most ci systems out there have their own caching mechanism well it turns out very recently as of 2021 build kit with github actions now can use the github cache api to store this cache between builds which means when it destroys this server at the end of the workflow which is what happens and then it rebuilds another one fresh in the azure data centers of github it's going to be able to pull that cache back down from the github cache api and have it ready for my next build so i save time future builds it's great all right now that we've improved the speed with enabling caching across servers we're now into the idea that maybe if you're like me and you be lucky enough to get one of the new apple m1s you now have to build for multi-platform ideally your developers who are on intel are using an intel version of your image for your developers that are on apple m1 or another arm based laptop that is increasingly getting more popular with everything including windows and raspberry pi's that they're using an arm 64 based version of the image all we're going to do is enable qemu and then tell it which platforms we want to build and it'll build them in parallel for you with just a few lines so down here i'm adding a new step that sets up qemu which if you didn't know is already in docker desktop so if you're using docker desktop locally you have this as well but in the cloud and github actions we're enabling it here and then down in our docker build section we're adding a new line this is the same platform list that you will see for images in docker hub now let's get on to metadata so the metadata is a specific action you can get from docker just called docker meta or docker metadata and it allows you some intelligence around labels and tags for your images because quite often you'll want certain tags for a pr and then different tags for the release when you actually merge the pull request and then maybe you want different tags if you actually tag the git repo like tag a release maybe you want something else there and docker has a great plugin thanks again crazymax that allows you to basically make a few lines of yaml to automate all of that and it has a bunch of intelligence in the background that looks at what events are happening in github why is this being run and makes decisions for you on the fly so in our case again we're on the same two events like always but down here we're adding another step you can see the theme here we're going to keep adding steps we're going to add a step before we build to add in our tags now i've given it a whole bunch of tag rules here and each one of them will apply at specific times and it's quite possible that any time i push the image up it might have more than one tag but again i don't have to write this out manually in some script i'm just giving it some rules and the documentation is great so i'm not going to break it down for it too much then i also down here need to add those steps so i'm enabling push true to say always push this up to docker hub even on pull requests now because it's going to be smart about those tags it's only going to push certain tags on pr's and other ones on a merge to the main branch next up we're going to add a comment so this one i'm going to fly through pretty quickly because it's not specifically about docker but i'm just going to have my pr add a new comment to the comment thread in the pull request of what my docker image tags are so that i know when i push them i don't have to go digging in the logs to find all the different tags that were pushed so there happens to be one in the marketplace and if you scroll down to the new area it's a little bit complex i mean there's more than a few lines of yaml here but basically if it's a pull request it goes and finds the pull request number and then it goes and creates the comments through the api and it spits out down here the docker image tags that come out of that docker meta step so i passed information from one step to the next one without having to do any sort of fancy environment variables and scripting i just use these different variables next up cve scanning this is near and dear to my heart because i feel like we should all be scanning there's now free really good tools there's paid ones that have even more features if you want them and we can implement this relatively quickly in any of our workflows so today i'm choosing to use trivi and that is by aqua security you can see it right here this is the action i'm using and i'm telling it basically to use the image of the github run id so you will see throughout these examples when you start diving into the code that i always create an image with a unique tag that is certain to be what that pr run built because if two people possibly are pushing to the same pr relatively quickly after each other it's possible that some of their stuff could get confused with the different image tags so i always use the run id which is unique for each action run it stays the same for reruns but usually only rerun when you have failed github actions but anyway i like it because it's short and it's numerical it's not a huge shaw hash or anything like that it's something that i can quickly copy and paste and even read it and tell if it's the same one or not and that works great so all i'm doing here is i'm running a scan of that image that we just previously built all right now this isn't going to actually block that's why it says here non-blocking all this is doing is giving me all the information coming out of a scan good bad and ugly low high critical all the different cve levels it's just going to put that in my log so it's really just for information purposes if there is a critical it's not going to block it because i have an exit code of zero so this is just sort of fyi as you start to take on scanning your images you'll find at first that you have lots of vulnerabilities in your images because it's really hard to actually get it down to zero so and because it's really hard to get it down to zero sometimes so you want to start out easy and not making exit code one because if any step in your workflow crashes well then the whole workflow or pipeline is going to crash so it's going to stop so you don't want that out of the gate you kind of want to get a sense for where you're at work on it maybe improve your vulnerability list to that get it shorter and then maybe put in what we call a blocking scan where it will error and stop the entire workflow because it can't progress without you fixing that vulnerability first now a variant of this is to enhance it by making it blocking so we're going to do two things here now if we look at this one there's a couple of chunks that i've changed the first one is i've added permissions because i like to be secure by default so i tend to set up all my permissions manually instead of assuming everything has right access to everything in github you can actually click a setting in the repo settings that says only give read access to the get code by default and anything else it needs access to you have to explicitly allow i'm saying here give right access to the security events and then i'm checking out the repo which is a new one we haven't really had to do that yet because technically the docker build step does it for us but in this case i'm going to run a different tool that needs explicit access to the repo code so i'm doing that as well then down here you'll notice that i'm doing this load true and so when i build this image i'm not actually pushing it yet to an registry what i'm doing is is i'm letting docker build export the image right out to docker images so that it's available to any docker run commands which is what i'll be using down here with all of these trivia scans now i'm still doing the original scan that doesn't have an exit code 1 so it's going to give me all the results and i'm just going to keep those in the logs in case i ever need to look at them but the one that really matters now is i am then scanning and then i'm uploading the results of that scan to github security which is a feature in repos that allows you to see security vulnerabilities in your code in this case i'm only scanning for high and critical severities so that i only get the most important ones and i will fail the entire workflow if it finds one of those i'm also saying don't warn me about anything that hasn't been fixed there's a cve vulnerability out there and it's been announced but no one has a fix yet don't tell me about it since i can't do anything about it anyway now we're getting into testing which is the good stuff so we've got some of that boilerplate in there and we've added some image scanning but now what we're going to do is start adding on the layers of testing first up is unit testing the simplest kind of testing it's usually one or two commands for the whole repo and we can do that here with relatively little change all we're doing is adding two steps here so what you're seeing is i'm now going to build the image just for testing in this case i'm going to export that image back out to docker from the build x builder and then i'm going to run my sample little docker command here that's really just to simulate maybe you did an npm test or you're doing a bundle rate command of some kind you know whatever you're doing to run your test you're just doing it inside the image and you're doing it with the image you just built now presumably if that test is successful and it exits zero then we will go and build the production image with all of the proper tags and all the settings that we had before now not much different would be adding integration testing now in presumably in integration testing you're going to need to have other things started before you start your app to test it you might need databases or back-ends or workers or apis or something else available most ci systems have their own mechanism for doing this some of them call them services like they do in github actions where you can spin up other containers or other things in the background that once they're ready it will then run your commands in this case i'm going to be using docker compose because that's more cross-compatible with all the different ci's so it'll work the same on each one of them and it's really a small step all i'm doing is after i build the image again just for testing in my case what i'm doing is i'm taking the image id and i'm exporting it to an environment variable and then i'm using that inside of a docker compose yaml file so all i'm doing is i'm starting my image the all hands image that i just built with this testing image that's coming in from the environment variables then i'm also starting a postgres server and starting a health check that health check is going to make sure that postgres is healthy and it will wait to start my sut which is the real test command i want to run it won't start that until the postgres is healthy you'll notice that i have a postgres condition of service healthy here that's what waits for it is waiting for service healthy now in this case all i'm doing is a curl command relatively simple but if that errors out the way that it handles it here because the problem with docker compose is if it's spinning up three containers which one should it exit code 1 or 127 or whatever the exit code is going to be how does it know which one to take the exit code from well that's what you use exit code from for and i'm saying here that i'm not really testing the postgres spinning up it is a dependency but what i'm really focused on is the exit code and error coming out of the sut command that i'm running inside that special service now this is one of my favorite ones because i'm more on the op side of things myself and being able to do quick test deployments that i mean quick with sample code or even a helm chart or a customized yaml or whatever you've got that's great because a lot of us don't get to do that step in an automated way especially not on every commit every time we're pushing to a pr but it's not that hard and it's actually quite fast in this case i'm adding a few new steps so we're going to now start using the github container registry as our testing registry and then we're going to use docker hub as the production one you don't have to do it this way but in this case i just want to show this additional example of how you might use two different registries for two different purposes in this case what we're going to be using is the github container registry that's already there available for our repository we're going to be using that just to store the temporary pr images while we're testing and if everything goes to according to plan then i will push to docker hub as well as github container registry so we needed a way to get our image that docker built over to our kubernetes cluster which in this case is going to be k3d this is a great action that will build you a cluster running k3s or keys inside of k3d which is a docker container it does this in less than 30 seconds and i told you earlier that there would be some shell script and this is really it this is me creating a secret of the docker registry key then i'm exporting an environment variable that is the image tag that i need to test and then it's not even using customize or helm it's just using a straight up yaml file that has an environment variable in it i'm using a little utility called env subst it's environment substitute essentially and it allows me to take environment variables and put them into yaml so if you just looked at this simple little deployment file the only thing in here that isn't standard kubernetes yaml is the image name and then i overwrite that with a one line command here and then i apply it to my cluster so it's simple it works i wait for the rollout of that deployment to finish and then i run my specialized command inside that deployment i actually just run another curl command to test localhost and if it fails that curl command it's going to exit with an error that will crash my workflow and tell me that there's a problem and all of that happens in way less than a minute usually around 30 to 35 seconds for all of that so it's pretty fast and i love it now for the bonus round we're going to take this whole thing and we're going to parallelize it as much as we can remember that that advanced graph that i showed earlier with all the different parallel tests happening well that works well and most ci solutions have parallelism and in this case that's what we're going to do so we're taking that all those things we just did all those different examples we're adding them into one big workflow that's hundreds of lines inside yaml and we're breaking down jobs starting with just building the test image so all we do is build that image we push it up to github container registry with that specialized one use run id that we're going to be testing against then we have a whole bunch of different tests that need the name of that first test so this need is how we create the dependencies so it builds the image and then it runs a whole bunch of these jobs and then at the very end if they're all successful it will push images to docker hub so where do we go from here well the repo brett dot show slash all hands 22 has links more information for reading obviously you can look at the actual github actions running in that repo and see what they look like dive into the logs it's all open source and i encourage you to take a look and join my community which the links will be in that repo as well so thanks for watching hopefully i'll talk to you again someday

Channel: Bret Fisher Docker and DevOps

Views: 13,450

Keywords: docker, devops, github actions, github, automation, ci/cd, continuous integration and deployment, continuous integration, ci/cd with docker, docker testing, kubernetes testing, build docker image, build docker image github actions, build docker image ci/cd, test docker image, test container, test kubernetes yaml, kubernetes automation, kubernetes devops, docker kuberentes pipeline, docker pipeline, kubernetes pipelines, docker workflow, docker github action workflow, gha

Id: aZzV6X7XhyI

Length: 33min 14sec (1994 seconds)

Published: Thu Mar 31 2022