Jay Beale, Attacking and Hardening Kubernetes | KringleCon 2020

hi i'm jay peel i'm cto over it in guardians here to talk about attacking kubernetes at kringle con where i am freezing my took us off i think it's because i keep reaching into the ice to get the freshest of the sushi so this talk is all demo it is tied to last year's story from the holiday hack challenge where the tooth fairy was the villain and we're continuing by saying that she left a logic bomb behind running in a kubernetes cluster and if we don't deactivate that logic bomb it would be bad like egon spangler bad so uh one of her developers luckily has found redemption and wants to help us out and kent's going to try to help us get the logic bombs password before it causes bad things to happen so what is kent what's ken's starting point he has complete administrative privilege in his namespace a very common thing for organizations to do but outside of his namespace he has very little privilege he's kind of got read only and he can only read some things so this cluster is staged uh in gke google's managed kubernetes service um which creates nodes as virtual machine instances gc instances uh in a google cloud project so kent's got some ideas about how he might do things um but let's go straight into our demo so if you like what we're doing if you like what you're seeing here if you're really interested please come take a look at in guardians kubernetes work in general and especially our open source paradise program so with that said i'm going to go right into demo so kent is uh on the kubernetes cluster he's just doing a quick like look at all the namespaces that are out there there are a ton he knows that the thing he's looking for is probably a secret um that's in one of those name spaces but he doesn't know which one and he probably doesn't even have access to list the secrets um he made a list of namespaces we can kind of see the one that he's got developer kent um and developer kent he can list secrets and get there and get their contents and even create new ones um but outside of that in the cluster at large um if you were to even go namespace.namespace you can't list you can't list or read secrets in the other things so he's going to start by just creating a quick um pod in kubernetes in the kind of one of the simpler ways we're going to use coop controls dry run and tell it hey can you give me a quick definition a quick manifest for pod that's going to run nginx i'm going to call it kent to the dry cleaners a reference to the movie real genius which you absolutely have to see so so we've written out that manifest it says uh it says a yaml file and um we can take that file and just run this command one more time or copy and paste and write it out to a pod file we're gonna run coop control create and that's gonna create our kent the dry cleaners pod and now kent's gonna exec into this pod because that'll put him in the cluster instead of on his mac laptop um he's also gonna install jq real quick because it's json query um as uh carlos venturamini at in guardians has taught me it's an incredibly useful tool um for you know for going through and both querying and changing json on the fly really easily so um kent's first step is to go and talk to uh the cloud providers metadata uh api and uh because uh cloud instances will very very often have service accounts kind of like i am accounts um and uh and so he wants to see you know those those accounts in gcp they'll have they'll often have real read only but read to every bucket um and that can be pretty cool so he's gonna go and basically query and find out what service counts he's got it looks like two bus to cube and default but default is assembling basically to bust a coupe so he'll ask we're gonna do another query and basically ask the metadate api if we can have a uh a token that we can use to authenticate to the to the google cloud services so we just throw token uh the default and then the token on the end of that and then we can take that and we're just gonna parse it out with jq um so we're going to um toss that into jq and get at that uh access token field and so we get that access token field we throw through said to pull off the quotation marks and now we have something that we can put into a variable we could call that variable token and token is that is now that token it's only going to things only alive for 3 600 seconds but during that time it's useful to us we could pull one another one if we want to we'll need the project id as a number not just as a name um but the metadata api will happily give that to us at which point we can start doing authenticator requests we're going to do um an authenticated request against the storage api where we're presenting a token as a bearer token storage api for bucket storage we do our slash b which means i want to list all the buckets that are available in this project and the first thing we get is basically something that says nope sorry no good um this is basically uh workload identity it's that's uh hitting us here uh a gke feature a friend of mine aaron small helped ship and this is um and this is making it so that we're not getting we're not getting the nodes uh kent's not getting the nodes uh uh service account he's he's getting one that's mapped to his to this specific workload which is turning out to be useless so we're gonna have to try something else and so now kent's gonna try something else he's going to create another pod and this pod it's a little hard to follow with the word wrap so we'll do it a second time this pod is going to um it's gonna be the same kind of nginx pod um he's gonna call the pod privilege as a name but use the same image but uh we're gonna we're gonna have the the dry run kick this thing out as json throw it into jq and have jq set uh the specification for one of the pod's containers and make that container have the uh have the security context privileged and privileged pods privilege containers you should basically never have um you'll have it with a few of the control plane elements but you really shouldn't have anybody else creating their own um so this is a json version that manifests where security context um has privilege set to true and so we've written it out to a file called yaml but it's json and uh code control doesn't care so we tell kube control here you go stage this it creates pod called privileged and kent execs into it and the first thing we see is this is what makes a privilege product privilege pod um it's among other things we have all of these devices from the host from the machine that this container is running on published into the container um and that is no bueno for defenders luckily our defender here is tooth fairy and uh and we're trying to you know uh we're trying to uh fix things so we've got uh one of those devices is dev sda the the actual hard drive raw device from the uh from the virtual machine so um we do a quick after just to see which partitions are which looks like sda1's probably uh going to be really really useful to us and so we mount sda1 into slash mount we trued into there so that we basically as we explore the file system um in this shell we're gonna we we're in the file system for the host um for that hard disk not the file system for the container and so i know i can find in home communities bin a copy kube control which i'm gonna want and here or kent's going to want and so we'll find some other things like take a quick look at os release to see yup my the container may have been debian but the host is ubuntu we can see the the name of this it's a gk you know that you know ends in that fj9 um we can see it's a kubernetes you know look at the message of the day just kind of get a feel for yup this is a node so let's look at something else that's really really useful it makes one of the ways that we can really really make this attack count there's a kublet and there's a coup proxy both of which on on every node and they both have their own search to make requests against the api server um we're going to use the kublets because the kubelet has to connect the api server to the container runtime whether it's docker container d or whatever and it has to the cool thing is that it for any pods that are running on this machine on this instance this virtual machine the kubelet is going to be the thing that's that's uh getting the secrets from the uh that's getting the secrets um from uh from the kubernetes control plane and passing it into the containers um we're staging these connectors with those and so the google will be able to read all the secrets for anything that's staged on itself but not on other on other uh but not in other instances um so there's a coupe config which refers to this pki um this pki directories you know client cert uh if we look at that client cert um we find something we like which is it's going to have not just a public key but also private key so this is going to let us authenticate to kubernetes as the kublet's service as the kublet's kubernetes service account and so um we're just doing quick lists like get all the get a list of all the pods um we see that you know while there were tons of name spaces the only three really in play or kube system with the control plane and developer kent that we're playing with and then this other thing gps a reference to last year's holiday hackathon which was impressive and fun as heck and there are all these pods in gps called logic bomb passwords remember we're looking for the logic bombs password so that's good for us um so we need to find ones that that have that are that are staged on the same node on this fj9 node and so we're going to grab for just fj9 and these are the pods on the cluster that are staged to this one and we see a bunch of control plane elements but we also see four of those logic bomb passwords and um and so let's go and look at the um let's go and look at the logic bomb passwords kind of status and definition to find out what their secrets are so that we can go and request those secrets so we'll take our we'll take our coop control um continuing to use the kublet's cert we'll get the pog definition for this specific pod we'll ask for it in in uh yaml um we forgot to enter the namespace we have that namespace in and we see there's a volume called logic bomb shutdown it's provided its contents come straight from the secret called logic.bomb.shutdown and they're passed in and placed on the file system in the etsy directory um so what we're going to do is basically ask for that ask for the copy of that secret um and so if we um so just one more time noting that that's this that we're going for secrets that are only held by pods that are running on this node that's that's really critical to understand we're going to uh do our coupe config ask for the secret you know the first one just gives us the metadata from the secret so if we ask for an output in json or yaml we'll get the actual secret and so we ask for json realizing we don't have jq in here we asked for an enamel and then we're going to grab out part one um so we can just get that uh the data and it's uh it is base 64 encoded so we're going to parse that throw it into awk um try to toss it in the base64 but there's a space so we'll just we'll change the way we're using awk and we get it and here's our password it says first password is a line from real genius the kent says you will rue the day um but it says you'll need a second password got you got one flag you got one more flag to go and that's going to be found in a storage bucket in a gcs storage bucket so we know our next target is to get that storage bucket access so it's time to it's time to try harder on that on uh on the uh on the gcs um on the gcs attack path so let's go and try something else so kent goes and basically says okay i'm going to try a different kind of thing instead of privileged now i'm going to make my pods start with with the host network set to true which means it's going to instead of starting its own network namespace it's going to use then it's going to use the network namespace belonging to the virtual machine it's on which can evade workload identity and so um we're going to stage that um pod kent's going to exec into that pod and now he's going to curl the metadata api we see a different service count this one looks like what we're used to when metadata when when workload identity isn't in play so we get asks for an access token um i'm going to try to parse that out with jq um realize jq is not there so add jq real quick since this is recorded i was able to skip past that and so we've got the access token um we're parsing out with jq and ripping off the double quotes and um that gives us and that gives us what we need so now we're going to set gcp token to that and now we're going to construct our curl request to list buckets against that you know against that uh bucket storage api using a bearer token now the first time i write this i use token instead of gcp tokens so we don't use the right bearer token and we're not authorized but uh i or kent figures this out and uh uses the right bearer token and here we go we've got a list of storage buckets there's only one there's only one bucket it's called tooth fairy second password uh that's a pretty good name uh that's a pretty i have a lot of hope there and uh and now we're gonna do b then the bucket name then o for objects give me a list of objects and there's only one object and that one object is uh a file called secondpassword.text and so we're going to take you can take this nice little media link and paste it right in so after our you know after our headers we can throw in that media link and we're gonna get the uh renew the file and we got the contents that file the second password which is in the immortal words of socrates i drank what so hey thanks for watching if you enjoyed this please dig into kubernetes and cloud native security some more i'd love to talk kubernetes with you here in the north pole or even collaborate with you on in guardian's open source kubernetes penetration testing tool paradise um also kubernetes is cool in that there are a ton of people who are not just doing great work but they're also eager to share with others and help people get involved i'd recommend following a number of notable kubernetes security folks work and i've listed their twitter handles here but go looking also for their talks their blog posts their books their tools um uh by the way the kubernetes project has a new special interest group sig security and it's a really welcoming community looking for more hackers so thanks a lot i'll see you around kringlecon in the holiday hack challenge bye-bye

Channel: KringleCon

Views: 892

Rating: 5 out of 5

Keywords: Holiday Hack Challenge, KringleCon, SANS, InfoSec, CTFs, CyberSecurity, Cyber Security

Id: S4ySed0k7uE

Length: 16min 12sec (972 seconds)

Published: Wed Dec 09 2020