Using Checkov with Terraform and Azure DevOps

have you ever been interested in integrating static code analysis for your infrastructure's code in a ci cd pipeline well hopefully that isn't all gobbledygook to you and that's what we're going to cover in this video [Music] what's up everybody it's ned bellevants ned1313 on twitter and welcome to terraform tuesdays uh it's it's actually wednesday sorry about that so i i got a little tied up this week couldn't get this video out on tuesday like i originally intended and also i'll struggle with azure devops which is kind of the topic of what we're going to be talking about today before i get into that just two quick things number one i have a terraform certification guide that i wrote with aiden ermey it's in the process of being updated for terraform 1.0 if that's of interest to you check out the link down in the description he's going to come up again good old aiden cause he kind of helped me out even though he didn't realize it help me out with this video and secondly if you're a potential sponsor if you're a vendor who thinks that your product might benefit from being featured on terraform tuesday hey you know i am open to the idea so please reach out you can do it via twitter ned1313 my dms are open let me know if that's of interest to you i'm open to doing like a a early ad at the beginning of the video or also just a full-blown sponsored video if it's a product that i think is awesome and want to make the world a little more aware of so without out of the way let's talk about the topic at hand which is using checkoff from bridge crew to analyze your infrastructure as code i'm not going to lie this has been requested by several people so hopefully this video meets up to your expectations we are going to be using chekhov and we're going to integrate it into an existing project that i've been working on which is developing a full azure devops pipeline for the checking and deployment of infrastructure's code with terraform that's a project that has taken up two videos already so this will be the third video in the series so i'll throw a card up there that links to the previous one so you can see what i've done so far basically what we have at this point is a azure devops pipeline that's pulling in code from a github repository and when new check-ins happen it kicks off the pipeline that's the trigger is when something when code gets committed to the main branch that causes a trigger which has the pipeline run and right now all it does is that we'll say all it validates that the terraform code is valid it then runs in in it and a plan and then it waits for someone to approve the plan before it goes to the apply stage so that's what it does today and what i'd like to do is add some static code analysis on top of validating that my syntax is correct so that's the idea that's the core concept so let's get into what chekhov is from bridge crew and how it's going to be integrated into my pipeline check off is a static code analysis tool for infrastructures code and it isn't just for terraform it can analyze cloud formation arm templates and other formats that are out there and it has a bunch of built-in policies that it's checking for by default so if you just run it without telling it which policies to run it's going to run through the full battery of policies that it has and see if any of those apply to your infrastructure's code if they do then it's going to check whether or not the code matches up to the expectation that's in that policy if they don't match then the test fails and there's something you might want to fix in your code or maybe you'll say no it's okay i know about that i'm going to add an exception there for that one policy you can also write your own policies the policies can be in python or in yaml and they'll define something custom about the code i'm looking for this resource and it should never be configured this way and that's something you could add in as a custom policy on top of all the existing policies that they have so that's the basic idea now the other thing to call out here is this is static code analysis which actually lines up pretty well with infrastructure as code since we're not trying to dynamically run an application and then test everything about that application infrastructure's code is generally static in nature until you send it off to the api of whatever cloud provider you're using so static code analysis tools actually work really well and there's two places you can do the static code analysis you can do it against the terraform code itself you can also run it against the plan if you convert the plan into a json file then checkoff can go through that json file and validate that everything is cool so that's what we want to do there's two goals in this project first when a new version of the code is committed to main i want check off to run through that terraform configuration and tell me if anything in it violates the conditions now there's a little caveat here because i'm going to be using some modules so i actually want to initialize terraform first so that all my modules are loaded and then i want to run the static code analysis against both my code and also any code that's in the modules because although i'm getting these modules from a registry that i might trust who knows what they've changed in the meantime so i want to make sure those modules are living up to best practices as well so that's one place where i want to run check off and then i want to go through this standard terraform plan process and produce a plan file and then run check off against that plan file and make that part of my review process and once everything looks good i will approve the change and it will go through to the apply stage so that's where we want to insert check off into our pipeline so that's the central idea that's what we're going to be doing in the demonstration so let's head over to visual studio code i'll walk you through where i'm adding it in the code and then we can actually kick off the pipeline and check that out as well all right here we are in visual studio code i've got the files open on the left you'll note that the date of the folder is 2021.0622-a-d-o that's for azure devops that's the pipeline we're going to be using and it is yesterday's date in terms of when i'm recording this because like i said i thought i was going to get this recorded yesterday and then things happened life happened so that's why the date's a little bit off but anyway that's what the code we're going to be using if you're looking for these files you can always find them on my github repository it's called terraform tuesdays pretty easy to find and let's get into the code itself we've got two directories here we've got a setup directory and a v-net directory we're not worried about the setup directory that was used entirely for setting up this entire pipeline auto-magically and all the resources that it uses so we're not going to get into that too much let's instead focus on the v-net contents we've got a main.tf file that's the thing that's actually going through the pipeline to be checked and then deployed and then we have our azure pipelines.yaml file that's the file that actually defines our pipeline so let's open up the pipeline file first and take a look at what's in there so i'll go ahead and open that and let's hide the terminal because that's just that's just taking up room we don't need that let's go all the way up to the top of the pipeline now as i said before the trigger for this is that a change happens on the main branch and it includes the path for today's files i had to alter this because i made a copy of the previous version all right but that's not really important another cool thing that i discovered in my digging process i was using a task to install terraform as part of the beginning of each stage in my pipeline and you don't need to do that because the ubuntu image for the hosted runner or hosted agent on azure devops already has terraform one well the latest version of terraform installed isn't that nice so we can totally skip that it does not however have chekhov installed so my first stage is validate validate does two things it uses terraform validate to do analysis of the code and make sure that it's valid terraform and now i'm also going to add in installing check off and running check off against those same files so in our steps one of the steps is to run a bash script that uses pip to install check off and it's really just that simple you just run a bash command and it installs check off so we have that added into our pipeline in our validate stage then in the next step we have to download some azure key vault secrets these are secrets that i'm using as part of my pipeline i covered that in the previous video so if you're wondering how azure key vault is being used go back to that video and check it out scrolling down a little bit more we're going to initialize terraform in part because we want to download the modules so we can do analysis of those modules as well as the code that's already there scrolling down a little bit more we get into the validate command so we're simply running terraform validate against the working directory where our configuration is and then lastly we get to our verifying module files with check off so you can see i've done this as a bash task and we're simply running check off and then we're pointing it at the directory that has our files and i'm first going to verify the module files that have been downloaded so you can see i'm pointing it at the directory v-net dot terraform that's where all the modules are going to be after we run terraform in it you can also see that i am skipping some of the policy checks here the ones that have to do with docker if you know anything about the way that modules are published when you download the module you're really downloading basically a github repository you're kind of like cloning it and in that clone is going to be a docker file and some other things those don't impact how the module is used by terraform but they may fail some of the checks for check off so i'm doing skip check ckv underscore docker underscore star which means all the docker checks are skipped because we don't care about those and then i'm doing the output to a junit xml file and hat tip to aiden ermey for this one this is awesome what the reason we're doing this is we're going to output the file if we scroll all the way over to a file called check off module report and then what are we going to do with that file well if we scroll back here the next step in our pipeline is to publish the test results of our module test and that is using the task publish test results 2. so this is going to show up in the testing in our azure devops pipeline whether or not it was successful and it has the whole report in there of what happened that's well that's kind of useful now that we've checked our module files scroll down a little bit more and we are going to verify the main files with check off so again we're going to do a bash task we're gonna run check off dash dash directory and point it at the v-net directory itself this time not the dot terraform directory we'll do the same output to a junit xml file and then if we scroll down a little bit more we'll publish those test results for the root module using the junit format and this will happen whether it is successful or it failed so that's important we're going to publish the test results whether or not the test succeeded or failed all right so that's everything that needs to happen in the validate stage if the tests with check off fail it's going to stop right there it's not going to move any further test failed now if it does succeed it'll move to the next stage which is plan so let's scroll down to plan in our plan stage is going to install check off just like we did before and this is because you're going to get a new it's a new stage we're going to get a new vm image out of the pool it's not going to have chekhov installed so we have to do that again and we also have to do the same thing with azure key vault and we have to initialize terraform so we're going to do that and then we're going to run a terraform plan and under the command options for the terraform plan task we are going to specify we want to output the file to the working directory main dot and call it mean dot tf plan so now we have our tf plan file now we want to verify that plan with check off and the way that we do that is we're going to run a bash script again this time we're going to go to the directory where that tf plan file is and i want to make sure i'm in the right directory so i have an ls here this is the thing i was really struggling with last night when i was trying to get this all working and now we can run terraform show dash json and give it the main.tf plan file and it will convert well won't convert it'll output it as json and then we can just direct that to a file main.json now we have our plan in json and we can use check off again to scan the file main.json and output that to the junit xml format awesome now that we now we scroll down a little bit we see i have to include a bunch of environment variables here and that's because terraform show requires basically that you have terraform initialized and that has access to everything you use to initialize terraform that's another thing that i stumbled over pretty hard because i don't use terraform show all that often and in this automated fashion you needed to be in the right directory and have the right environment variables so i figured that all out so you don't have to and then lastly we're going to publish our test results for the plan whether it succeeds or fails now it will move to the next stage which is approved and it's a manual approval the approver can now go back look at the test results and then make a determination of whether to move forward with the deployment or stop right there so that's everything that's in the pipeline let's take a look at our main.tf file in the main.tf file we're deploying an azure v-net and we've used this module before so no big surprises there but i've added an additional resource in here and that is a security group and i've added a security rule inside the security group that allows ssh from anywhere so checkoff isn't going to like that it has a check that says do not allow ssh from anywhere so this is going to violate and our first test should fail and then i can make a change here so that it will pass so i already have this queued up as ready to commit so i'll go ahead and commit this change with this new security rule so i'll just uh add sec rule here i'll commit it and then i'm going to push it there we go it's now been pushed up to the github repository and that should kick off a run of the pipeline so let's jump over to a browser and here is the azure devops project that i'm talking about let's go to pipelines and we should have a pipeline currently running we sure do it's my tacos pipeline and you know you saw that the commit was sec rules so this is the commit that we just had let's go ahead and watch the progress in the validate stage so let's expand this out and we can see it's in the process of checking out the code now it's going to install check off this might take a moment or two but after that it will go through the rest of the stages and it should fail on one of these stages so i'm going to speed up the video a little bit so we don't have to sit through the entire run and we'll see what happens once it gets to the publish check the publish check off root test results all right we've reached the end of the validate stage it's doing its post job completion but we can go back and look at this failed test here verify root module so first actually let's go up to the verify modules with check off this is where it verified the modules and there's no output here which implies that nothing was wrong it was okay and it sent all its output to that junit xml file now let's move to the verify root module and we can see there was one error here but we don't see the output because that was sent to our testing so let's go back to the view of this pipeline run in general and you'll see there's a tests tab here and if we hit our tests we can see that there were three total tests and one of the tests failed and it tells us what the test is it says that we failed the check ensure that ssh access is restricted from the internet we failed that check and we knew we were going to fail that check that was very much intentional if you want to see all of the tests that were run what we can do is change the filter here to passed and passed on rerun so we can see all of the checks that were run and this is kind of there we go it's not the greatest interface because i don't think i can slide this at all but as you can see it ran so i ran other check off ran other checks that applied to what was in our configuration and only one of those checks failed but it did in fact fail so now let's alter our configuration so that this check does not fail all right we're back in visual studio code and all i'm going to do is simply change the access from allow to deny and now it'll deny that traffic from anywhere we'll go ahead and save that change and then that is a new commit so remove access will be my commit message and go ahead and push that and that will kick off a new pipeline run so let's go back over to the browser and go back to our pipelines and we should see a new pipeline kick off in a moment go ahead and click on it again sometimes it takes a moment there we go it's kicked over and we can see the commit message is remove access so this is the new one that is running and if we go into the stages we can see it's invalidate now validate all the stages should pass because we removed the offending rule so go ahead and pause it now and resume when the validate run is complete okay our validate stage has completed successfully the portion that we were failing before this verify root module this time it exited with a zero so all the tests passed on that we can see those in the published tests in a moment but let's scroll down and take a look at what's going on with plan so plan is going through its steps now it's going to install check off go through the initialization and plan for terraform and then it's going to use checkoff to verify the plan and send that to our test results so this is also going to take a moment so why don't i pause things and we'll resume when the plan stage is complete okay our plan stage completed successfully and if we go to the verify plan with check off task it looks like that also completed without any errors so if we go back to the main pipeline view well too far go into here and go to tests we can see it's got six passed and zero failed it performed three checks when it went through the initial analysis of our route code and then it performed those same three checks when it did the analysis of our plan if we go here and check on include past in our view we can see there's the check off root scan and all three tests passed and then there's the check off plan scan and all three tests passed for that so if i were the reviewer for this and wanted to make sure that i wasn't violating anything by policy i could see that everything is good the other thing i discovered is that there is now a terraform plan that will a terraform plan option for the terraform plan task that will send that terraform plan output so you can review it here in a nice screen as opposed to trying to review it in the logs and so this is going to tell us exactly what it plans to do it plans to add one and actually if we go to summary we can see it tells us that it's going to create one resource and then it's waiting for our manual validation that's what we want to do well everything looks good to me so i'll go ahead and click on review and say looks good all right and then click on resume and now our pipeline will move to the last stage in the pipeline and if we want to take a look at that we can see down here apply is not started but it's about to it's in the process of starting and that apply process will well it's not going to change anything about the vnet because that already existed but it will add that security group with the rule that denies ssh so that is using check off to check the code both of the main code itself and then also check the plan generated by terraform okay so we successfully integrated check off into our azure devops pipeline and you can absolutely do this too if you are walking through this process it was really not that hard to get chekov working and i definitely am going to include a link to aiden ermey's post about how he went about it he actually ended up using a docker container to run the checks as opposed to installing it on the hosted agent you could also create your own agent that has check off pre-installed and that would probably speed up the process a little bit now let's talk about future plans for what i'm going to do in the next revision of my devops pipeline and there's one big thing that i want to do i'd really like to run the validation and static code analysis when new code is checked in to a branch and then a pull request is created to merge to mean that's when i think i would want to validate the code as as a sanity check before i accepted the merge and also run check off against it before i accepted the merge so that's where i want to run that piece of code and then i want a totally separate piece of code when the merge is actually accepted into main i want a separate pipeline that then walks through checking the terraform plan with check off and doing the application of actually deploying out the infrastructure from the merge so i want to split it into two pipelines basically so that is one thing that i want to do and the other thing i want to do is stash the output file somewhere if i want to do a manual approval and i want to look at the plan before i approve it i want that plan.tf file to be available to me so i can ensure that whatever is run by apply is codified in that tf plan file and not just apply running as an auto approve so those are the two big improvements that i want to do in the next video dealing with azure devops pipelines with terraform all right that's gonna do it for me for today hey big thanks to aiden ermey for that blog post that actually helped me tremendously when i was trying to work through this shout out to bridgecrew for creating an awesome tool that works really well it was so intuitive i was like i must be missing something because this is too easy i mean azure devops made it a little more difficult but the checkoff tool itself was actually really easy to use so good job on both of that and hey thanks to you for watching i do appreciate it if you want to subscribe to the channel i love it i love you thank you so much for subscribing if you want to like and share with a friend i appreciate that too if you're interested in supporting the show in some way i do have a patreon so you can sign up for that and you will get early access to videos as well as a weekly newsletter and hey if you're a vendor out there that wants to sponsor an episode reach out to me i'm available that'll do it for me for today until next time stay healthy stay safe out there bye for now i swear this is the last thing i'm gonna show from my parents garage that i found but this is a ridiculously funny book it's really really good it absolutely holds up and some of the people in here are i mean they're pretty famous now you might recognize at least one of them from brooklyn 99 but yeah just an awesome group state by state with the state it's very fictional it's very silly and i've been enjoying rereading it so hey are you enjoying rereading a book you found in your parents attic let me know about it down in the comments see ya

Channel: Ned in the Cloud

Views: 910

Rating: 5 out of 5

Keywords: HashiCorp Terraform, Terraform Automation, Azure DevOps Pipelines, Terraform Checkov

Id: 7wbhs7PFdoA

Length: 25min 6sec (1506 seconds)

Published: Wed Jun 23 2021