Setting Up Pod Security Policies in Kubernetes

hey everyone so in this video we're going to be taking a look at setting up pod security policies in kubernetes I find pod security policies be a little bit challenging to wrap your head around when you're first getting into it but it is a pretty important thing to consider especially if you're trying to harden the security of your cluster admittedly I've seen a lot of clusters out there in the wild where this was either turned on incorrectly and caused issues or just not turned on at all because it wasn't quite understood so the few sections that I'm going to walk through in here are going to be talking about how to set up the admission control how to set up service accounts so that you can choose who is authorized to make pods in a certain way the policies themselves that describe the rules available in the available privileges and then our backs deciding how you can give certain service accounts the access to those appropriate pods now like always I'm going to be going through some different code samples in this video or really Hamill samples around these different topics and you can find these samples on my website if you want to follow along in your own cluster as well the website link will be in the description of this video ok so the first thing that we need to know about pod security policies is that it is an admission controller now admission controllers are basically something in kubernetes where we have the ability to add something in such that after the auth and authorization oriented pieces like who is asking for it and what group they're associated with what user they are and so on has occurred we can go in and ask this admission controller in this case pod security policies whether that particular user should be authorized to do what they're asking to do I'll start off by SS aging into one of the nodes in my cluster so this node is on port 22 alright great now this is a pretty simple cluster it's effectively a cube ATM bootstrap cluster nothing special happening Calico is a CNI plugin and I'm actually just running a one master cluster at this point to make it simple so do note these might be considerations you'll need to take for all of your masters now let's start off by taking a quick look at the static manifest now depending on how you install kubernetes this could vary but nowadays a lot of us install the cube API server in of a static pod so basically for those not familiar with static pods manifests that we put inside of this manifest direct really directory the cubelet will bootstrap and instantiate onstart such as the cube api server so looking inside of the cube api server for a moment it's a pod and inside of this pod we have a couple different flags now in particular there is an enable admission flag now your mileage might vary depending on the version of kubernetes you're using here there used to be an old flag I believe it was just called admission control or admission controllers and that Flags biggest difference is that the order mattered enable admission the order does not matter and it's just like admission control in that it's a comma delimited list so if I append to the end here pod security policy now I'm going to have these two admission controllers put in place it's important to note that just hitting save here in a cluster that you care about is not going to be a trivial change I'm adding pod security policy here to show you how things work behind the scenes but suffice to say that once I turn on pod security policy in the fact that I don't have any policies uploaded to my kubernetes system existing pods are not going to be able to be recreated new pods will not be able to recreate it I'll really kind of lock down my system completely until policies are available so this is something to consider when you're writing the automation that sets these things up by default in your cluster but let's start off by going ahead and saving pod security policy just like that okay and I'll exit out of here come back and we are going to take a look at some of the different areas of this cluster as I change deployments around and so forth so first thing I'm going to do is I'm going to set up a watch on the pod replica set and deployment of the default namespace so that we can keep an eye on this now I've just made a change to my API server so it might take a minute for it to bootstrap there we go now if I come back inside of here I have two deployments I'm going to be showing you throughout this video one is a host network nginx deployment and one is just a normal engine X deployment if we take a quick look at the nginx deployment there's really nothing special here it's a deployment it runs an nginx container that's it now if we take a look at the nginx host network deployment the big difference here is this one has host networking set to true this is going to be our example for enforcing pod security policies this example is actually a pretty relevant one a lot of times folks will put in a lot of work to do things like set up proper network policies and secure the network but if you let pods get created that run host networking then there's a lot of abilities to bypass that and do things from the host level of its network namespace so that's what we're going to try to lock down first now the first thing that I want to do here is I want to look at the host network that is or sorry that nginx that is not host network enabled and we're going to apply this to the cluster so I'm going to apply the deploy post sorry keep saying host network here but I mean deploy nginx and send that to the cluster now if we go into the watch here and take a couple seconds to let it update you can see here that while I'm listening pods replica sets and deployments the only thing that actually got created was the deployment nginx deployment and then the replica set was created as well you'll notice the replica set is never going up and he counts at all so this is the key thing we have pod security policies in place in the actual account that would be instantiating this pod the replica set account does not have the ability to create any pods because we have no pod security policies currently in our system so this is we're talking about the idea of service account access is especially important so I'm gonna come back to our nginx deployment and I'm going to delete this deployment all right now that that's deleted it should clean up and there's one more thing server side I actually want to give you a quick peek at before I show you some of these different pieces so if I go into my node one more time and then I run the same command to them into a manifest but this time I do the cube controller manager one of the flags you may want to make sure is although it is the default for a lot of installers and bootstrapping tools today is that you are setting the used service accounts credentials line effectively what this is going to do is this is going to make sure that there is a separate service account for each controller in the controller manager and once we get deeper into pod security policies you'll see how this will enable us to get extremely specific with which controllers we allow access to these policies as a quick example if I come in and do a get four service accounts in the name space cubes system you can actually see a lot of these dedicated service accounts that we're using so if I want to make sure that the daemon set controller can access certain pod security policies but the replica set controller the one we were just looking at can't I actually do have the flexibility to do that type of change all right so that's some of the pre-work in the system the admission control and making sure that service account is enabled as well now the next thing that we're going to do is set up some pod security policies in general if I come back to these different files here we're gonna take a look at the policies now in this case I'm gonna set up a pretty simple policy structure in fact a policy structure that might work for a lot of deployments it's going to contain a restrictive policy which is effectively going to be the policy that gets used for everything in the cluster and then a permissive policy which is going to be kind of a heightened permissive like privileges when certain pods should be created such as ones in cube system so you can go into the documentation for pod security policies and get an idea for what some of these do in fact this restrictive policy is not as hardened as it likely should be for example I'm letting people run as any user and I don't have many restrictions here but for our example one of the key things I wanted to turn off is to make sure that not just anybody could create a pod using host networking so that is set to false so let's go ahead and create this pod security policy inside of our system so I will do policy PSP restrictive alright great so that's created now now I also want to create the permissive policy as well now very similar has similar field you can just change different values in an art case the main focus for simplicity we're looking at here is going to be this host network line which if anybody can resolve the permissive policy they're going to be allowed to create a host network enabled pod alright so let's go ahead and create that now as well so this will be the permissive policy great now just to verify everything's as expected we can go ahead and do a cube cuddle get one more time except this time we will do pod security policy and I will get rid of that great so now you can see permissive in restrictive pod security policy you're going to get a somewhat of a nice view giving an idea for what privileges are available now the key thing to note here is I still can't create pods just because we have these two pod security policies in place doesn't mean any of my users or accounts can actually access them so what comes in handy here is the ability to leverage our back to set up access to these specific pods security policies so that we can actually create deployments pods all that good stuff again so let's come into our back here and take a quick look at this now our back can be a little bit tricky basically if you remember our use case what we effectively want to do here is we want to provide a default restrictive policy and that is going to basically allow every single creation of a pod whether it be in a cube system namespace or other namespace to be able to resolve the restrictive policy which gives us basic access to normal pod creation but limits things like host networking and then we want a more permissive policy to be resolved in certain contexts which could be things like running in cube system and so on so before we get too deep into how we can set up that kind of escalation like setup we just need to note that we're going to be creating two cluster rules now cluster roles are things that allow that define access to stuff and eventually are bound to through bindings cluster roles are completely cluster scoped they're not attached to a namespace or anything like that so we're going to start off with a PSP permissive cluster role and then a PSP restricted cluster roll as well let's go ahead and apply both of these from our are back directory so I'll do our back and I will do the PSP we see here the art it's prefixed with our back so that our back restricted cluster roll and then the our back permissive cluster roll and make sure we get rid of that typo okay so now we've got the permissive cholesterol here that we can bind to and then we've got the restricted cluster roll down here we can bind to but we obviously need bindings to make these effective in any way shape or form and again if I didn't call it out explicitly note that all these cluster rules are doing they're giving access to the use verb for a pod security policy namely in this case the restrictive one and that's repres pected to the permissive one up here okay so now we've got these in place and we need to set up some bindings now what I'm gonna do here is take a moment to show you kind of a graphic that gives you an idea of what the logical flow might look like in getting these bindings to resolve certain PSPs now in this case let's say we have a controller creating a pod and the one that we've been using is a replica set controller but it could be daemon set controller anything and let's say that we had a flow where we were trying to create the cube proxy pod which actually something that needs elevated privileges inside of cube system now the bindings that we're gonna set up in a moment here we're gonna start off by setting up a cluster roll binding to that restricted policy so if we give the daemon set controller or all controllers access through a cluster roll binding to the restricted policy then those controllers will always be able to resolve the the restricted policy regardless or restrictive policy regardless of where they are operating in so in this case the flow would look like us going out for cluster roll bindings pod security policies saying okay your cluster roll bindings or global policy would be Orca policies even will be restricted policy and we say okay now then it's going to ask do we allow this pod and in this case we don't because it's not it's a host network-based pod coming from cube proxy going up a step it's going to retrieve the pod security policies for that particular namespace and we can achieve this using role bindings so while I use cluster role bindings to define the global restrictive policy access I'm gonna use a role binding which is namespace scoped in cube system to give access to a elevated policy so it's going to go out and say give me pod security policies for cube system and then it's going to return the permissive policy in this case it's going to ask do I allow the pod and cube proxy which requires host networking is going to say yes in this case now this flow is not meant to represent what kubernetes is actually doing behind the covers but it is going to or behind the hood it is going to represent what kind of logically you can think about this process as is you're kind of getting your bearings for how policy is going to be set up from a global and namespace level so let's go back here and first start off by setting up the global cluster role binding for restricted now in this case what we're going to do is something a little bit different than what I was talking about with service accounts we're going to provide the entire group system service accounts and kubernetes the ability to resolve PSP promise or restrictive excuse me such that whether it's a daemon set controller a replica set control or whatever it is they can always resolve that sort of default admission controller now note that I'm doing all of this for service accounts for the controllers and things that would actually instantiate pods users still would not be able to create pods directly they'd have to rely on deployments replica sets things like that so if you want your users to be able to create pods you're probably going to want another group in here that can give users access to this PSP restrictive cluster rule so let's apply this next and this is going to be the cluster role binding for restricted all right so now we've got the pieces set up such that we should be able to create deployments again let's check it out if we go back to our nginx deployment the non host networking one here and we apply this deployment again so I'll apply deployment engine accent before I actually hit enter on that let's set up the watch one more time all right so I'll go ahead and deploy that coming back to the watch now you can see it is once again creating or not once again but for the first time creating the pod engine X deployment which is great so now the replica set has the ability to create an instance of a pod in this case the engine X deployment now we'll make sure that it creates before I leave this point but just to go back and show you an example of the heightened post Network one what I'm gonna do next is I'm going to remove the old engine X deployment and bring in this host network one now again since we've only got that cluster role binding in this case we're only getting the restricted policy back there's no heightened privileges on a namespace basis so in short this one should fail we can come back we can see nginx is running to keep things clean let's go ahead and start off by deleting that deployment I just did so I will delete that come back to the watch and now we can see that it's terminating now let's go ahead and apply the heightened one so this will be deploy and we will do the host network based nginx create that and now we're back in the state that we were previously which is expected we're trying to create this deployment this replica set and it's telling us no I'm not able to do this let's actually dig in this time and see why it can't do it so if we grab the name of the deployment or replica set in this case and we do a cube cuddle describe to that replica set and we specifically need to call it a replica set we can actually get some good information from the event API namely you can see hey error creating pods engine acts forbidden I wasn't able to validate against any pods security policy and it even calls out and says hey host network was a conflicting value that is not allowed to be used this is great for troubleshooting we can tell exactly why this pod was not able to be created alright so all is well let's go ahead and delete that deployment we're creating pods as we'd expect and protecting our system against pods we don't want but we still have a problem here like I was showing in that diagram with cube proxy there are some things like cube proxy that we actually want to be able to do these types of things so when operating inside of the context of the cube system namespace it's important to us that we can have this extra cluster scoped ability to access the permissive policy and that's what we'll look at next so if I jump back to my are back let's now look at the role binding for permissive I'll open that up and this role binding as you can see is a little bit more specific so what I'm going to do in this role binding is I'm going to provide this role binding to the namespace cube system it is going to be available to the job controller the actually we don't even need the deployment controller in this case we need the replica set controller so let's change that typo replica set controller there we go okay back to it so the replica set controller the daemon set controller and the job controller is all that we need this access to so let's go ahead and apply this for PSP permissive cluster all access so we will apply in this case it's gonna be in my are back directory and it is going to be the role binding so let's find that and add permissive it all right now if we bring this watch back up one more time I should still not be able to create and we'll bring up that nginx pod just to have in front of us exactly what we're working with so this pod still not be able to be created inside of the default namespace so just as a sanity check let's apply it one more time to the default namespace we'll go back to our watch and we'll see once again it is not able to be created great so let's delete this deployment all right now let's go ahead and create this deployment inside of the cube system namespace in based on what we had set up based on our pod security policies we should add the ability to create an in cube system so we will go ahead and switch our watch over to cube system for a moment so watch and in this case cube system now note this will have a little bit more inside of it but still nothing too crazy to follow let's go in and hop back we'll create this once again which is now in cube system oops I ran delete that time so we'll apply this in come back to the watch alright and now if we follow it up we should be able to see that this is being able to be created so we can see the deployment at the bottom nginx host network deployment we can see the replica set ID the 597 number right here and then we can see the instantiation of the pod for z11 s on the replica set five nine seven C for C so we now elevated privileges to the replica set controller inside of cube system and other controllers as well which is fantastic especially important if we were to kill something like core DNS or Q proxy we want to make sure that we have these elevated rights inside so this actually gives you a really good idea of how pod security policies can be set up and how this flow can look there is one more use case that I'm going to end with here in that use case is what if we go back to the default namespace in a hypothetical situation we want to provide access to a specific service account so what I'm getting at here is we still want the namespace to be completely locked down we want to make sure that one specific pod is able to run with some elevated privileges we can do that an interesting thing about PSP is you can actually provide access to use these pod security policies from the workloads service account so that when it starts it can actually tell oh I'm allowed to do this and it goes ahead and stands up that particular pod so to give an example of this we're gonna go back to default here in a moment alright and side of this setup we're going to add a new role binding in and this role binding is going to be in the namespace for PSP permissive and we're going to effectively say if you are the special SI service account for the namespace default you have the ability to resolve the PSP permissive but anything else that gets started in here will not have access to do that so the first thing that we need to do is we're going to need to create the service account special sa that you can see in this manifest all right now we're going to go ahead and apply this service account our back so our back role binding service account permissive we'll add that in all right now any particular deployment using the service account is going to have the ability to create one of these more privileged pods with host networking turned on so let's go back to the deployment for host network let's add the service account into the spec for this pod now again this is inside of the replica set inside of the pods back so we'll do service account and we're going to provide it with the service account that you just saw special sa and save that now again we have no pods running in the default cube system or not cube system in the default default namespace and we're going to apply this replica set in deployment one more time so let's apply this deployment specifically for the host network base engine X and hit enter and it's created we'll go back to the watch and boom we've got the pod running because that particular service account has access to that more permissive pod security policy now there are some other things that might be causing this to fail so in particular and why host networking is a dangerous thing I completely forgot about the pod I had instantiated inside of the cube system namespace so now this pod is contesting for that piece or that port I should say port 80 so again it does show if we go back to the watch here while it is failing to instantiate the pod it's actually kind of making our case for pod security policies to some degree which is that we really need to be able to limit this kind of stuff and make sure that we aren't just letting anybody hop on hosts networking or create any host port they want and so on so now you can see how this is set up now another interesting detail that I'd like to share with you and we can still see it even though this pot isn't creating in fact to keep it cleaner we could just go to the cube system namespace let's do the cube system namespace and let's look up the nginx pod all right so here's our nginx pod that did successfully grab that port what I'm gonna do is I'm going to output the Y amyl of this pod so we'll do a get one more time for this particular pod and we'll ask for the ml output and that will be in the name space of cube system all right now when troubleshooting it can sometimes be helpful to figure out what pod security policy resolves so if you have a lot of pod security policies kubernetes is just gonna find one that resolves and use it and if you're trying to figure out why did this get allowed access and so on you can figure that out by going into the annotations when kubernetes resolves oops when kubernetes resolves that annotation in this case scrolling back up here the PSP permissive annotation that means it's the one that it found resolved and applied to this pod and that is why it's running so a really cool tip and trick when you're debugging and figuring out what's going on in your system now the last thing I want to leave with is just one more note on the bootstrapping so obviously it's unlikely you're gonna go into every single one of your clusters and do all the things that I just did manually in fact if you did them in the order I did it'd be quite dangerous because you're kind of leaving your system in a fragile state so what you really want to think about doing when automating this be it through ansible or some other automation tool is how do you put all of these bids in right out of the gate in an order that makes sense so again you may want to do is set up both inject injection of the policies upfront because even if pod security policy isn't turned on as an admission controller kubernetes still accepts those API objects you may want to go in and create all your are back all your roles make sure everything is set up and then use the alteration of that API server flag as kind of the on switch for pod security policies now again it all depends like if you're just bootstrapping clean might prefer to just set up the API server with pod security policies on and then have a job that sweeps all of these are back things and PSPs in right after the cluster gets instantiated all different options just be considerate when you're going in and thinking about how exactly you should set up this automation and bootstrapping so all in all I hope you found this video helpful and setting up pod security policies and hopefully it'll save you a little bit of stress that I took on when I was first learning them as well so thanks for watching again the website that goes through this all in plain text is in the description of this video feel free to check it out and I'll see you in the next video thanks

Channel: Josh Rosso

Views: 9,710

Keywords: kubernetes, podsecuritypolicy

Id: zkTsj-5o5YI

Length: 27min 19sec (1639 seconds)

Published: Sat Dec 08 2018