Command and KubeCTL: Real-World Kubernetes Security for Pentesters - Mark Manning (Shmoocon 2020)

so a topic I'm gonna go ahead and and let's get started with our our next presentation I don't know about you but real-world kubernetes security is something near and dear to my heart so please give a warm MOOC on welcome to mark you guys hear me okay right now I think you can okay so the title of this talk is command in coop cuddle because I refuse to call it coop control and make this pun work of my own presentation we're gonna be talking about kubernetes from a pen testers perspective or red teamers perspective people that have work in security and can kind of harden kubernetes environments however you see them we're gonna do this through three different case studies and these are amalgamations of projects that i've worked on or things that we've seen in the industry that are common misconfigurations common attack vectors and we're gonna go through an entire kind of attack chain of finding an issue finding something we can exploit going to the kubernetes cluster and see if we can exploit it you know I hope I did the right sacrifices to the demo gods etc the the things that I want to do is though as we exploit these two the different attack chains I want to talk about the different technologies and the components and why they fell over or what misconfigurations happen there so my name is Mark Manning I go by Auntie tree on Twitter I'm the technical director at NCC group and I have a bunch of researchers that work with me on just containers and Linux kernel stuff and orchestration stuff and we do a lot of different container assessments which is a vague thing now because there's so many things that bacon containers now we see them in firewalls in cars and like whatever you want to have run a container in you're probably gonna have like some definition of a container if it's just like namespacing so we do a lot of assessments there I'm from Rochester New York anybody from Rochester is just saying something last night this is the only conference that like more than one person from Rochester is gonna like pipe in I run 2600 group in Rochester we've got a b-sides in Rochester this b-sides for b-sides Rochester is our tenth anniversary we've been running it a really long time it's in March CFP is open right now and it's a cheap fun event for anybody to come to so in the beginning we had containers and I really got into containers because I was coming from the world of Android and mobile platforms and I really liked os's that had application isolation your Android OS has you know apps that aren't allowed to influence other apps and I was like okay cool here's this new container technology that's going to start getting worked into the desktop and the server environments and it's not really how it worked out but in an ideal scenario a container might be a few processes in a isolated environment that is designed to perform one single task like a micro service and in reality containers end up being more of a security opportunity if you will we can lock down containers but no one's gonna call a container a sandbox alone jesse Frisell has done a ton of work a long time ago building things like set comp BPF profiles for docker and locking down and actually it was successful preventing a bunch of O'Day's in the docker service itself and then what do we do sometimes we like disable all that work that they've done so my example for this is like well you know how much do we trust containers and would you run a piece of malware in like a hypervisor in a virtual machine some of us might I have never seen anybody that says like I'm gonna run a malware in this container because I trust the container security model and then came kubernetes we made it even worse actually the kubernetes was this thing that we kind of said was it was really all related to containers we said in the first place containers were this cool technology that's gonna allow us to scale up and scale down it's gonna have all these different enterprising features but really docker doesn't do much of that itself and the container engines don't do much of that all of the work is done at the orchestration level so kubernetes being the orchestrator that one the the orchestration wars that alas said for like two months but they provide authentication controls authorization controls things like storage management and networking becomes complicated when you've got multiple servers in multiple containers that need to connect into each other they do this through like just running clusters right like if you imagined what you needed to build if you wanted to run a bunch of containers across different servers you'd go well okay I need to run these containers on this server and probably need to have like an API that phones home and says like hey here's the status of this container and it communicates to other containers and stuff like that and that's that's really what kubernetes does it's this glue of a bunch of different api's and plugins that you can use it just manages containers at scale and and this was the big thing that started turning I think containers into a more production-ready travesties so we said you know in the beginning nginx could be running in a container it's it's running as maybe UID 0 we had this isolation and we have this kind of security boundary and then one of the first things that kubernetes does is they say we should split this out into pods so we dropped the idea of a security boundary in just a single container now we have pods that kind of share some of the namespaces with other containers so you might be running like nginx container next to a non-void container and there's also this init container that starts up kind of in the background that you don't know about so we're starting to just blur the lines of what the security boundary is and what the security expectations for these vitamins are and if you're coming from like a network pentest background and you'll like you know understand vulnerability assessments and you understand IPS and nmap and Metasploit and whatever you might think of the kubernetes topology like this where we've got this control plane that does a bunch of administrative stuff and we've got a bunch of workers and they're called nodes that are at IP addresses and you might have just stumbled across this like once a month at least I get one of my workers messaging me saying like hey I've taken over a pod what do I do now and it's because I'm not even on like a kubernetes assessment they're just on some kind of internal network assessment they roll into a VP C that's got kubernetes there and they accidentally exploit it but we can't think of it as as a network service per se we can't think of it as 10/10 10.3 is this pod that runs nginx and I've just exploited nginx because they're ephemeral they should be running for minutes they should be running for maybe a couple days at most usually so I like to think about it more like this diagram that I've created is like an OS stack right like if this is like a Linux OS stack you'd have a kernel down at the bottom and kubernetes has the the kernel of the cluster operating system if you will which is the kubernetes api everything goes into this API authentication control controls go through go through here extra plugins will all communicate over this API this ends up being the thing that we really want to target and then we have nodes which are just servers or bare metal or you see two instances however you've deployed like kind of your workers inside the cluster pods we already mentioned can run a bunch of different containers and we have this idea of name spacing which we'll talk more about later a namespace is mostly like a like a logical separation of kubernetes objects if you've got like a bunch of these types of workers you can throw them in a namespace and then we also have services that we use sidecars when we said like a pod is multiple container so you probably have like a primary function of that pod which is maybe nginx and then you might have something like sto that runs what's called a sidecar and it has a separate function to help support the primary function of that container and what's the name of the bar at this hotel seemed like perfectly kismet so everybody should have a drink at sidecar after this but why are we doing this talk I think there's a lot of challenges for pentesting kubernetes environments I think that we as an industry we as InfoSec people and we as hackers don't have all the tools and don't have all the knowledge I think as an industry to easily do this one of the reasons is the the threat model is always up for interpretation and this is true for any customer you work with a business you say well okay you've got a different threat model than this business over here but we haven't seen it at a platform level like this before I don't think where you'd go in and imagine seeing like okay here's you bun - it's running PHP or something like that you kind of have an understanding of the threat models that it provides or the security controls it provides but then you have kubernetes which can be configured in so many different ways and so many different security expectations and extra controls bolted on top of it that you can't really make any assumptions and you need to do a lot of work to just understand what they're trying to do in the first place the other part of this is that there's tons of MIS configurations that are really subtle this isn't a thing where you go and you point and click and you know compares against the siskiyou Bernese benchmarks with which we wrote nothing against them it's just that the benchmarks themselves don't always fit in together with themselves you can't just check all the secure buttons on kubernetes and and things are secure kubernetes comes out with new releases every three months which is insane of an idea for like a lot of production environments and even more crazy is that every they only support three previous versions right so only nine months of support for every cluster is possible and besides that besides the support it's not as big of a deal as the API is they get deprecated really quickly so from our perspective we're writing tools we've got all these like hacker stuff that we load that we like to run and then api's get deprecated they don't work anymore we need to update these tools we need to update our knowledge and our expertise constantly so it's a it's a long struggle and I mean the point of all this is I'm obviously biased I work for a company that sells services that humans like do reviews of but I think that humans are still the best at doing security assessments of very complex systems like this so as we talk about the things that we're gonna go the different nuances of kubernetes and the misconfigurations of kubernetes I've created three completely random businesses and we'll talk through their threat models and we'll talk through how each one of them is able to respond or defend from the same attack which is the running service on port 5,000 the service is exploitable and an attacker can take over a pod so the first one I want to talk through is power maverick and again you can see they're running a service on port 5,000 is exposed to something called a web admin and power maverick if you can imagine is like a company that sells power analytics on Prem for their customers so they're basically selling kubernetes as a service to deploy into your own private cloud into your own bare metal systems into your data center and imagine they're doing like some number crunching on just how much data or how much power you consume so their expectations are 1 everything has to work they've got a product team it says make this thing work I don't care what you need to do and - they've got they've got to make sure that they don't affect the rest of their customers environments they can't introduce new vulnerabilities that when this thing is compromised affects the rest of of their infrastructure so they do this with a couple of different security controls they leverage the latest kubernetes defaults and if you take nothing else away take away that you should never do that kubernetes knows that their defaults are insecure you know their their design is not to make kou brain secure it's it's to make it functional so they're using kuben eyes defaults they're using everything deployed into the default namespace and we'll talk about some of the implications of that later and then the cluster is deployed into a dedicated V PC and this is one of the smartest things they could do where they say to their customers look take this kubernetes cluster but I want you to deploy it into this isolated environment so that even if it gets compromised it kind of mitigates the the attack from of affecting the rest of the infrastructure but the real rub here is that they're using something called helm - how many people here have heard of helm okay right nice so I'm not gonna go to me too far into it cuz it's kind of beating a dead horse at this point but helm - and there's a new version helm 3 used a service called tiller and this service was notorious for being easily deployed into a stirrer and it allows you to push down different what they call charts and it'll just run like an application imagine like a like a Play Store for kubernetes so you can just say here's this chart I want to run it runs ends in X for me cool it did this by setting up a service that sits inside the cluster that you would communicate to and say hey here's what my pod looks like go deploy it for me the problem being there was no authentication and no authorization controls built into the service by default that's that's how they told you to set it up the other problem is that there's no isolation from the web admin pod that we've compromised to this service and literally copying and pasting this command that I have here helm to give it the host which is always the same hostname in the cluster point it to say install and here's a chart that that I've created that is malicious that will dump all the secrets of the cluster back to my house it will compromise the entire cluster now that's not really cool to talk about because I really want to the point of this is like the results of this attackers are able to compromise the pod they can steal all the data and analytics for the customer there's no PII there's not a lot of sensitive information there's not like tokens or stuff about the company they've they've owned this specific you know application there that's run by power maverick and they should obviously fix it but when we're factoring in mitigations and and impact here it's understood it's important to understand the the overall risk and you go back to the business say look I've owned your entire cluster here's how I did it and they say ok thank you very much we're gonna keep this design and we're just going to patch the original remote code execution in that application that allowed you to get in they're not going to change the cluster and and I don't agree with it but it's a valid decisions because it's low cost to maintain they have their own business that they need to factor in but the point being is that here's one specific threat model here's a business that chose this threat model for this kubernetes environment and we need to kind of deal with that second company's more interesting second company is cyber shmoo and they are a business that has only one customer and that customer shmoocon and cyber smoo handles all of the tickets for shmoocon so they are a multi cloud architecture that load balances between Google cloud and between like a sure so they've got different kubernetes clusters and different clouds and their expectations are basically they need to sell tickets three or four times a year for like you know point three seconds so it just needs to have that super-high load for a short period of time and it needs to work the other one is that it's a super high value target for hackers right we're at a hacker con we're selling hacker tickets and hackers are going to hack into my system to get hacker tickets so we've got skilled hackers that are able to you know invest time to get these tickets and I don't know how much they're worth on eBay hopefully anybody buy tickets on an eBay or no they don't want to admit that okay the other the third part of this that for security expectations are they don't want they want to limit the risk of an insider attack they want to limit like being able to you know knock dark Darth null on the top of the head and be able to steal tickets and passwords for accounts the way they do this is direct access to the cluster through coop cuddle will go into in a little while but just understand that like you you don't have administrators that are able to log into the cluster itself everything is abstracted away using terraform something like spinnaker so that you need to have access to the terraform scripts that are deploying this stuff so most of the time you'll see more like a CI CD flow for the infrastructure that just gets deployed into the info into into kubernetes or modified scuba Nettie's the other part of this is something called node pools we'll get into later but node pools is something that kubernetes is investing a lot of resources in in being an isolation barrier so that even when we compromise that web admin service we mentioned before it shouldn't allow you to compromise the rest of the cluster and the third one and we're seeing them a lot more of our customers doing this with their leveraging hypervisor based container engines things like G visor firecracker when they need to have like a workload that is super sensitive they'll still put it into a hypervisor because we know how that the results of this are one it's like a high barrier for attack so mission accomplished it's really difficult to break into this cluster even when you can break into the cluster the node pool will isolate it so it's not compromising the entire cluster even in that case we've got multi cloud architecture with a load balancer we can flip over to the other cloud environment when we need to the problem being is like there this is great and I and I like this model and I really like the the scenario where we're abstracting away direct access to the kubernetes clusters and we're seeing more customers doing this it's super expensive to maintain infrastructure costs sres that know how to deploy this stuff we can't really expect that everybody has the the resources to be able to deploy clusters like this but we're kind of talking about like a Goldilocks scenario everyone's too cold one's kind of too hot I don't have the resources to build that this next case study we're gonna go into more of a demo this is e cloud this felt like something from Silicon Valley I don't even know if it exists ich loud is geographically diverse startup they've got companies or they've got like developer groups maybe in Singapore and in Canada and each developer group is able to deploy their own code into the crew Benes cluster so the organization has an IT operations that sets up the cluster in the first place and then they grant access to the developers to be able to deploy their own code developers push their own code developers also make their own security choices for the applications that get deployed the way that they do this is they isolate with a namespace and this is the first example of kubernetes trying to do multi-tenancy stuff in our case studies so they're trying to say like one development group represents one tenant another development group represents a different tenant and all this is restricted with roles and all of this is restricted with pod security policies we'll get into in a second so let's see just how successful any of this stuff is so I want to walk through an attack chain hopefully we can demo this stuff if the networking is working and go from finding a service to exploit going through the cluster and and seeing where we can take it from there so to start off we've got a service to exploit which is an one of the first pen tests I ever performed that stitch allowed just a URL parameter to to execute commands into the cluster super-nice we can dump env we can you know do whatever commands that we want in there so cool my first demo was successful that was pretty stupid this is what it looks like if we walk through the slowly for 5,000 we go into the web admin service we've compromised the pod we we can do whatever we want in the pot what do we do this is still the most common question I get from co-workers that are like I've compromised a pod what do we do from here so the first thing we need to understand is the idea of service tokens and these are jots that are just stored in the same path every single time VAR run secrets kubernetes is service account / token and that jot is able to authenticate to the kubernetes api remember i said the kubernetes api is the kernel of the cluster operating system why would anybody put an authentication token in every single pod that gets deployed to allow you to access the you know the kubernetes api one reason is they think that pods need to understand how they're running in the context of the cluster so they need to go ask the cluster some questions a more common one is there's a service like if anybody uses hashey core vault there's a secret injector for for-4 hashey core before kubernetes that uses an authentication token to authenticate back to the Hashi core vault service so that's kind of like service to service communication that makes sense but the most common thing we still get is you know it's there because it's the default it's there because changing it might break something and that's not because you know that's not to make fun of somebody that makes that choice that's because kubernetes is so complex you're really scared of changing whatever defaults they've set up so if we go back to this and now we kind of have a targeted approach of what we want to look at we can look at the the the token if we just cat var on secrets kubernetes i/o and I'm gonna paste that over here really quick and we can look at some other stuff too let's find the kubernetes endpoint this happens to be an environment variable so 10 27 to 41 is just exposed and if I did a command like this you can kind of break it down I'm hitting the kubernetes api I'm just patting the token into the API and I'm saying like tell me what endpoints you have and this is interesting to us because we have the public endpoint of the kubernetes api so you can see that stuff in the back but i just pasted it in the public api this gets us something like this coop cuddle get bods scare pod ok cool so far it's working we now have access into a part of the cluster we don't have access to everything in the cluster let's just see as an example I tried to query the for running pods in the default namespace it doesn't work I query the secure namespace it does work so let's walk through this a little bit more I curled the endpoints that are exposed in the kubernetes api I got back the endpoints that were that were publicly accessible I'm now just connecting in from my laptop through that IP address into the the kubernetes api and i'm using that service token so the tool I just used was coop cuddle coop cuddle can do whatever you want in the communities API is the command-line tool for for all that stuff it doesn't imply that you have access this is just a tool that allows you to do certain things if you do have access so there's two angles of this you can start saying like look I've taken over the pod I can do whatever I want in the pod I'm gonna curl - you know all my favorite binaries I'm gonna curl bash coop cuddle into the pod and I'm gonna configure it to start interacting with the API or if the cluster is exposed publicly and by default for a lot of providers this is always true you can just connect you remotely from your machine and that's that's how we're doing this now we have access to a token what do we do the service account doesn't mean we have full cluster admin it doesn't mean we can do everything we want and hopefully the service account you have is constrained and if it's not we can see what we can do from there the other part that I'm skipping over here is the things that you can do inside of the container one of the first steps that I'll say to my co-workers is oK you've taken over pod run a tool like hamachi which is a tool that we wrote and go that dumps all the available cysts calls that your container supports it dumps all the mount information and gives you some perspective on potential insecurities so maybe there's just a built-in breakout to the pot itself but I really want to know what I can do in the context of the cluster so we already noticed that we can't do everything we can we can kind of access some areas but if I ran a coop cuddle off can I nice little API command oops just spell things right can't see this because I'm got it's so big basically this is showing that you can't do much you can access the health Z API you can access open API version information if I do the same command to the secure namespace it's going to give me a different result I'm trying to make this as big as possible but on the very top it shows verbs start out star basically permission to do anything that you want you also looked at it in this way there's a tool called access matrix and this tool will just give you nice pretty check boxes for all the different things that you can do most important one to us is this pods API which we're going to leverage so this is list create update delete I can list a pod I can get information about the bot I can create a pod so let's see what we can do from there all of this all these controls are set by a role in a roll binding and our role as we've just found out looks like this star star star star within the context of the secure namespace so again like we're looking at a development environment the you know the group in Singapore has access to their their own namespace and they've decided to create a service token that can do whatever it wants within the context of that namespace but theoretically this that whatever they're doing in Singapore shouldn't affect whatever they're doing in Canada our back and our back auditing is its own subject so there's a link to a blog post if you're interested in some of the other tools that we would use to further out at that we're now around to the phase of like attack pods what do we do now that we know that we can deploy into the kubernetes cluster what can we you what kind of images and things can we should we deploy so if you make your own attack images you could put your favorite tools any customized terminal commands all your precompiled binaries all the kind of fun stuff into your favorite image and that's what I did with this tool that called brick is brick is a blunt object with literally every tool that I could think of just thrown into the image it's not you know this amazing thing but anybody can go out and build this let's see if we can't deploy brick into this environment so first I want to just try if I can run a Paso coop cuddle run just say like a busy box image so we have full control over the secure namespace and means of the the namespace is secure image is busy box a name to my shell this seems like it checks out and I've created a demo that is designed to fail not many people would say this this is blocked out and and why is that because of a pod security policy and this is something we're seeing more and more of pod security policy is a policy that goes on top of your cluster admin roles or whatever permissions you have access to and it further restricts what you can do in the environment so we have full access to that namespace we can do whatever we want but there's a PSP on top that says you can't run these certain conditions or you can't do certain things so on our environment we see there's two namespaces so far secure and default so secure has a PSP default does not and as we bypass this I want to kind of point out how this works when we want to create a pod or deploy a pod we make a request that goes to the kubernetes api and says hey here's the pod that I want to create here's what it looks like it's a yamo file and the kubernetes api says okay cool i see that you've been authorized to perform this task it then sends it to what's called an admission controller and in this case the PSP admission controller which reaches out and looks for a policy inside the cluster that says hey this person that's allowed to do stuff in the cluster wants to make a pod that looks like this and the PSP says yes or no what depending on how secure or insecure that it is so our PSP looks like this it's got two main things one it says privileged false which means that you can't just run a privileged container that would just bypass all the security controls and - it says must not run as root meaning you can't run a container as the UID 0 or anything with those capabilities the problem being that PSPs monitor at at the initial point in time like when you create the pod it only checks it at that time it doesn't go back and monitor it so there's nothing preventing something to say well my pod has changed it's been updated there's no controls there and I'll show you like how that's going to work with command you may have heard of called pseudo but why do we even need a root in the first place a lot of services like in I think an 1.15 they changed the permissions on that that service account token that I said was in that same path so now you need to be a privileged user to access that count token some things TCP dump for example you might need to be the user but most likely when you're managing storage and you're using file system permissions in your storage stuff and you mount it inside of a container being the root person just automatically you know you've AIDS any kind of permission checks that are there so how do we get around this we mentioned there's the Brick pod and I'm going to deploy it into pull it into the secure namespace and all this is saying is cue cuddle run unnamed it my special pod and I'm telling you to use an image from a third party host and that's gonna be a big takeaway later this is a public host that's an image registry and I'm saying please deploy this image that I've created myself let's see if it's done creating it so it is really a brick it's like four hundred Meg's they just downloaded so it's now running in this service and I can run something like port forward to my special shell port 8080 and this creates a loopback connection into that that service I've just created so this is all still through the kubernetes api you know Who am I I'm nobody wait a second now Who am I oh I'm rude stupid demo so here's some processes that are running looks pretty legit this is the normal processes you'd see in a pod I'm running something called Gatti anybody who's got E I really really like it's a go TTY like a web - - I use it a lot of time but let's get into the next phase of this and look at let's do something like this we'll come back to that in a second let's just review or we've got so far so we've evaded the PSP we have now deployed our own image into the secure namespace we now have full access to that pod to do everybody want in that pod this is kind of review we've gone from finding a service to exploiting it taking the service account token bypassing the PSB now we're at the point of talking about namespaces we keep mentioning it but let's let's understand namespaces a little bit more we have two namespaces default and secure and we have roll bindings and PS PS and different secrets that only apply to one single namespace right like you normally can't have secrets that are scoped to the entire cluster you normally would have them say like here's this namespace secrets here's these other namespace secrets these are just a logical separation this is the the huge takeaway this is not a security boundary this is a logical grouping more like a folder there is no network controls on any of this stuff so as I'm doing a port scan in the background right now there's nothing preventing one PI from accessing another pod in a namespace alone cannot provide that stuff we can do things like sto and envoy that attempt to do that but by default none of that is possible so we just ran that net scan oh we found this cool service on this IP address let's go back and see if we can't access that anybody heard of so cat again all these all these modern tools like pseudo and so cat you've never heard of I just created a so cat pod with a little bit of extra configurations there we'll walk through that in a second let's just port forward to it secure namespace and what are called so cat 299 so this is another service in a different name space and hey looks like that other service that we exploited earlier which is super convenient for my demo right well let's just let's just walk through I'm going through image that I just deployed into the secure namespace I'm telling it to forward into the other pod in other namespace it's in a different location those are scoped differently and they have different security controls on them so is this a privilege escalation that's what we want to try to figure out so the the seok adding thing is like kind of like a server side request forgery right like at scale we can connect into our from our laptops and connect into other pods and other services so again we have the perspective of we can tool up the images that we can now deploy and try to attack it just from command line inside of a terminal or if we can forward back out to our laptops that has all of our favorite tools why not attack it there and this is the the command that I ran basically just running so cat and giving it a so cat configuration this is a good opportunity to mention crew and be heard crew Doku oh man okay forget everything else and just remember this one right here crew is like pip for coop cuddles the coop cuddles the binary tool that can interface with with the kubernetes api crew is a whole bunch of tools I think they're up to like 40 or 50 different tools that allow you to automate some of the common steps that I just did so that long so cats during that I just did in a couple of demos I said you know run run so cat here's this configuration here's his IP address and then oh create a port forwarder I wrote a tool for crew called net forward that allows you to give it arbitrary IPs and arbitrary of ports into the into the the cluster so I'm not gonna spend too much time on it but if you're interested in that tool it's open source and inside crew right now so it's literally like you do coop cuddle crew install this plug-in it automatically pulls stuff down and the value for pen testers and people that are working on on Auditing or kubernetes cluster is it's not deploying anything into the cluster you're not affecting production if you're working in production you're just running tools on your own laptop that help speed up the process of exploitation so let's see we're a we've got a new token let's copy this out let's modify our other token you can still see in the back all I'm doing is token swapping my coop cuddle config file when I say get pods this is a different environment and let me show you why if I go back and I say show me the secure namespace it says you can't anymore so I've hopped from one space namespace to another namespace and I now have different permissions in that namespace so let's see what we can actually do get my special pod that I've created just really quickly the important parts here is I've I'm running a privileged pod and I've got something called a host mount and I want to do a coop cuddle apply this special pod and bypass some stuff their pods been created port forward again to that pod that just created brick something ok so there's some processes running let's just turn it to root for fun basically the same amount of processes let's do something slightly different in this one to route command so time-machine going back to the 90s when we use to route environments I have a lot of different processes that are running now and if I run docker PS there's a lot of different containers that are running inside of my pod no that doesn't make any sense what I've done is I've to routed the the host machine into my container so I've taken like the slash directory of the Linux box underneath it and I said hey mounted in this directory I just happened to name to root and then Trude into it which basically is rebooting the entire box underneath it the node underneath it into my container so I'm still in a container but from a security context I've taken over that entire box so I now have full control over the underlying node let's see where we can go from here inside that node there's a convenient the coop cuddle command the tool has already been downloaded for me and if I export a specific path I can interface with the cluster again from inside of the node we're getting kind of like recursively confusing so now if I do something like all namespaces we see something that we haven't seen before this is all of the pods that are running inside of the cluster which doesn't really render correctly but this is coop system this is the secure namespace this would be all that all of the developers environments that that you can that you can imagine so we now have the ability to do whatever you want in the cluster see anything that we want in the cluster but because we're using this couplet config file were restricted so let me just show if I just wanted to do like a run image is a busy box I don't know if that's right but here's an example of a failure and I just do that anyways you can't run that because the kulit config is specifically restricted so if I did access the this path Etsy kubernetes manifest nodes have special permissions inside the cluster they're not allowed to just do anything that they want so even though it looks like I have access to do anything in the cluster I'm still slightly restricted so if I did that command terrible rendering what you can see is three to five seconds ago a new container was just created I do the same thing of cou cuddle get pods all namespaces will see that there's a lot of interesting new positive got deployed one of them being this one right here so what I did was I created a DML file I pasted it into the the this known path and the way that nodes work is they will mirror that path whatever yam will file that you put it in there and deploy it into the cluster and what's interesting is that it deploys it into the cube system namespace and we didn't have access to that before so let's see what's over there cube system namespace the name of the pod is this I'll put the yam oh and here's this interesting IP address for the service we just deployed and we go back over to here we use that tool that I just mentioned and give it the IP that we're just looking for and the port we wanted to connect into and we've created a listener on it $49.99 I think and we now have okay you may not understand what commands I just ran so let me walk through that really quick what I just created a mirror pod I just created I just compromised an entire node I was restricted so that I could only execute things on one single node all of that happened and and when I deployed the mirror pod into the cube system I create a remote connection into it and that compromised all of the nodes in the cluster and the entire cluster is completely owned the end thank you I [Applause] think you're applauding just for kubernetes for its like work in a consistent way twice so the issues that we went through in a summary you know there's an over privileged role bound to that secure namespace that's what we leverage in the first place that web admin pod that we popped granted us the ability to do things in the cluster we shouldn't we see that a lot of the time you might say like well you just made up this environment we still see a lot of environments that have full permissions to do these things from the service token level PSPs we're seeing PSP is deployed everywhere now fantastic not everywhere we want to see it more but we're now seeing a lot of scenarios where PSPs can be bypassed and and and they're done in subtle ways a big thing that not everybody agrees with me on is that your kubernetes cluster shouldn't have access to docker hub and it shouldn't have access to GC io by default we should be running private image registries because one of the ways that I was able to pivot through this stuff is by making a custom image hosting it you know externally and then just pointing to it to download so we should all consider docker hub is just like a malware repo the other part of this is that there's no network segmentation between any of the namespaces we might say like Oh namespaces I understand this in Linux but in kubernetes it doesn't actually have any security context whatsoever it's just kind of groupings and routings that are isolated we can reinforce namespaces you can create multi tenant namespace clusters with networking isolation but you need to install something like calico or sto and then they have their own kind of their own kind of issues but we're seeing a lot of customers doing the multi tenant types of deployment just like this but if falling over one way or another the other big thing was the reason I was able to take over a node was because there's the entire cluster allowed privileged pods to be deployed and this is still the default and we still see it every single time we're starting to see a couple of PSPs now that will restrict it but there's like if you go to install kubernetes today they'll say nobody should ever install privileged containers and then the kubernetes directions on how to install kubernetes is like install these for containers that are all all privileged so we're at this weird scenario so in summary you know we've got these three companies one is you know didn't invest a lot of money and resources into securing it kind of still works cyber shmoo threw a ton of money on it and it was relatively successful ich loud had the multi-tenant environment which is really complicated and complex to actually deliver we see this environment more often than not we're different developers are allowed to deploy into the cluster and the one we as like infrastructure people just go well you know it's the developers fault if they skew something up and that's true but the whole point of this you know where is the security boundary I originally got into containers because I was like this is a really cool isolation model and it's just like leveraging Linux primitives but is it at the pod level is it at the namespace level is it at the node level it all depends on how you how you deploy it and all depends on how you decide to implement all these controls so hopefully kubernetes has all the security abilities that that you need for your threat model in your organization but just not always true they're not building for security yet there they're working hard to make things more secure and to bolt on security features but it's a it's a cat-and-mouse game where new features are coming out very very quickly we saw a namespace isolation is difficult we saw that privileged pod still exists we saw that you know our back is really tough to get right and even when you have it right it gets applied in really super vague ways go back to that blog post that I was I was mentioning earlier to see what that's talking about and noticed that somebody was asking before like a journalist is saying like you know are you dropping any on any o days on kubernetes I'm like no we're not at that face we're at the phase of like there's still so many miss configurations that these are the ways that we break into kubernetes today hackers are breaking into the kubernetes by just looking at the common misconfigurations or accidentally things that are exposed so i'd love to come back in a year or two and say wow kubernetes is super secure now we've got these new days we dropped two or three CVS this year and you know it got patched and it got fixed and stuff like that but I think this is the better conversation to have as an industry is talking about misconfigurations and how we can fix them from either the InfoSec side or from the container side just to finish up a couple of things I want to call the shots into further research and hopefully you know if you're interested in the stuff areas that you can look at to service mesh is a technology that's being deployed everywhere SDO is super common look what happens to your SDO deployments if your pot or your container runs as UID 1 3 3 7 it's a fun little little thing that will we'll talk about more in a blog post later I'm really looking forward to eating cold water and Brad's coop can talk that's hopefully gonna go into more attacking kubernetes from like an from an exfiltration an evasion standpoint because we've got twistlock and aqua we've got cystic and things that are attempting to find us already like just detecting TT wise so how can we evade them and how can we start playing that cat-and-mouse game and I haven't gone into really practical compromising of saying like you know here's a store that's running or here's an engine X instances do something specific how do we break into it how do we how do we mirror it how do we really compromise a you know real-world scenarios like that so I think I mean we know how to do it I just we don't have enough time to talk talk through it right now I think we'll see more talks like that and then finally we do a ton of work with EBP if anybody experts or when I say experts anybody work on EBP F at all right now so this is a topic that I think again in the security industry we could really leverage into an attack tool and that's something that we've been building we've we've done a couple talks at DEFCON and a CCC about EBP VF e BPF is currently used as a defense tool for monitoring the colonel in a very performance way but there's nothing to stop us from doing all these nasty things in in attackers ways so two new tools to release one go pillage registries this is a tool that if you understand how a registry works and if you understand how show Dan works this tool will go through and automatically look through an image registry look through the metadata of that image and an extract secrets from it automatically at scale net forward we already mentioned before and there's links to more information about it Komachi is a tool that we wrote that that dumps information about the containers once you've exploited them auerbach tools this is just a link to a blog post that goes into some of the existing are back tools and how we audit them so before I finished I want to thank a couple people Jack lead furred is somebody in the company that gave me a lot of feedback on this and Rory McCune as another person in the industry that that is really pushing that I think the security boundaries and all of these people I hope to meet one day and thank them but they're on Twitter and these are the people that we should be helping support then and helping secure kubernetes so that's my time thank you very much [Applause] you [Applause]

Channel: 0xdade

Views: 2,543

Rating: 5 out of 5

Keywords:

Id: cRbHILH4f0A

Length: 49min 51sec (2991 seconds)

Published: Thu Mar 12 2020

Reddit Comments

[u/matisys]

Great talk. Thanks for sharing