Jay Beale - Attacking and Defending Kubernetes - DEF CON 27 Packet Hacking Village

so now this is this is an interesting timing for a talk because I want to introduce an old friend of ours who is this is the first time we're going to have a talk but then immediately you're gonna have to hop over to the next one for a workshop so this is like the prelude to your workshop isn't it without much ado JBL everyone hi hi hi hi I'm gonna try really hard to stay within range of the mic and let's see how I can do with that I'm not really good at staying in one place I kind of like to walk around if you've seen me at many def cons in the past I like to dance so anyway with that said I'm going to talk to you about attacking and defending kubernetes I it's a topic that's become incredibly important to me and dear to me and interesting to me so I have a bio slide but we all know bio slides are just big C's of text so I have a graph of the bio slide which basically says this is stuff I've done or stuff I'm interested in the I really I really enjoy this is stuff kind of going back way back to the back in the day writing writing at security focus and doing CIS work and giving some early early Def Con talks and nowadays teaching a teaching a class at blackhat and enjoying a heck of a lot of work getting guardian's as a CTO who likes to still be a very technical CTO so anyway so let's let's get on with what you came for so what are we gonna see today we're gonna talk about we're going to talk about kubernetes I'm gonna do I'm gonna do a bunch of demos we'll do a little bit of choose-your-own-adventure if all my if the tech tech works the choose-your-own-adventure part is entirely new so the nice thing is cutting-edge fresh the downside is oh oh so hopefully this is hopeless goes really well so I'm gonna talk about what kubernetes is I'm gonna talk about how you what you attack and how you attack it and then a little bit about how you defend we're gonna do some demos let's just start so the first thing I do anytime I talk about kubernetes is talk about what the heck this thing is for and because and I want to make sure I'm coming through on the mic am i coming through on the mic and like the back of the room perfect okay so what the heck this thing is for and and you know people are like I know that this is hot I know that I should use it that's kind of the same way we all felt about docker so you know why why do we use this or what's made it so big and the basically the the kubernetes came out of in a lot of ways it came out of what Google is doing with the Borg right that's it's an open source project that initially got released by Google and now has many many many contributors but kubernetes basically came out of this need we know that Google we know that Google had just has a ton of computers and a ton of workloads to be clear you know it's not exactly the most transparent company all the time but they've had they have some public statements they made one is back in 2014 they were launching two billion containers per week if that were evenly distributed to be 3,300 containers per second what's a container it's like a workload it's a program it's gonna run for a little while and do something we don't know how long that while is but if you're launching 30 300 a second you can't have a human being decide which virtual machine or which host you know each program each workloads going to go on so you end up needing a tool for that now what else well they did this they were they were launching all of those workloads on to 2.5 million servers at least that was their number back in 2016 and in 2016 hard drive failure rates were just right below two percent so if every server had just one drive from that's a big you know hand waving thing but every if every server had one drive that meant they had a hundred and thirty three drive failures a day or one every nine minutes so now you've got now you've got a resiliency problem right like you wanted to guarantee you know you wanted to guarantee so many nines of uptime and yet your physical machines are going to you know entire physical machines are going to die on some regular basis like say raid notwithstanding one every nine minutes so what features would you want if you wanted to be able to stage a ton of workloads over a ton of servers figure out how to use the servers of the greatest as close to their capacity as you could so you weren't buying lots of extra machines and you wanted to make sure that as programs crashed and as and as hard drives failed and so on that you still had availability so you end up with what kubernetes is which is basically a software-defined data center and we do that by container orchestrations so what do you get so basically this is your feature set this is what you end up saying you end up saying well I sometimes I have a lot of people checking their email on Gmail or or searching right now for the thing that just happened in the news and sometimes it orb as many so I'd like a whole lot of scaling I'd like that horizontal scaling I'd like to be able to say I have four copies of that workload no wait I have 400 no wait I have 50 and so on and you get automatic bin packing which is that like which workload should go on which machine so we can keep all the machines as full as we can and you get self-healing like that hard drive crash thing and you get a whole bunch of other cool features you get like automated rollouts and roll backs where you can basically deploy multiple versions at once and see if one of them gets see if one of them ends up producing more sales or more of whatever it is you're trying to produce or see if one's buggy and then be able to basically route all the traffic's the one that isn't and so on you get you get the other kinds of things you would need if you're going to do micro services so you get service discovery and you get load balancing and then you get orchestration of your storage and you get secrets and configuration management so that's kind of that's what the purpose was I'm gonna do a little bit more theory so just kind of like what's a pod what's a container so right we when we're talking about kubernetes we're talking about containers but we say pods a lot and basically pods are the smallest unit of computing kubernetes so all the containers in a pod are sharing an IP dress the only reason you put multiple containers in a pod is if you want them to absolutely positively land on the same host like they have they have need for very very quick network access or what have you maybe they need the same local storage an odd thing about pods is all the containers in a pod share an IP address okay so then you've got communities nodes simply plot a node is any system you're going to run kubernetes on and uh notice any of them any of the systems in your cluster whether those are virtual machines or physical hardware the nodes run a well first and foremost the nodes to be a node you've got a you have to have a container runtime so you have to have something like docker run C or what have you something that's going to basically take instructions about takin structions from kubernetes about creating containers and create them and maybe destroy them and so on you also need a couplet and a couplet kubernetes couplet a couplet is just this thing that runs in every kubernetes node and it ties the it ties the container runtime or the node into the rest of kubernetes it talks to the API servers and so on then you've got a CUDA proxy and that thing manipulates iptables rules to create pool networking it also serves as an actual proxy so yeah we'll get into that a little bit more and then services services are big because in kubernetes and services give me a great give me a great occasion to to kind of describe how you talk about the names of things in kubernetes when we're doing the attack demos we're gonna be talking about this pot and this pot in this pod but for the most part in kubernetes you don't ever think about which pod names you want right it's the whole idea is that the individual pods right the containers they're very very much supposed to be cattle they're not supposed to have names they're supposed to be interchangeable I'm supposed to be able to trash them start new ones up I have to I'm going for I'm going for that so what does that mean I mean if I want to talk to a pod what I do you don't I mean you might but rarely mostly you talk to a service and if service is mapped to multiple pods think of it as an internal load balancer now now if you've got like why my what did I say that would get me into something cool that gets me into labels and basically you have a load balancer you can't say this load balancer should be pointed at these five pod names right first you're gonna have a whole lot more they're gonna get added second those pods are gonna die at any moment third you're trying to go really really fast remember you know not 3,300 containers per second being launched right so what do you do you end up basically labeling everything so you label all the pods you give them labels and those labels are basically sometimes they're by convention sometimes they're whatever you chose but you label the pods and then you create a service and you say everything that you know this service is gonna be a name and IP address a port and everything that goes to that name and port or IP address in port is going to be sent to one of the pods that have the set of labels the service specifies it does kind of open an interesting question like if you are allowed to stage pods on to a cluster could you start intercepting other people's traffic by just giving yourself the same labels that their services are looking to land try to land things on okay interesting last thing we talked about last big big you know kind of simple glossary concept we need is namespaces the namespaces are just a logical grouping for things so we'll see those they're often used they're often used to separate projects or separate teams to separate an individual you know maybe two separate well anything that you anything you choose so there's some way of chopping things up and it's an it's important to say a kubernetes namespace is not the same as a docker or Linux namespace so much so that when I teach classes on this I one of the other folks in guardians Jared Fred s asked if we could please call urban and ease namespaces space names since docker and Linux you don't already have namespaces so it was you know don't over don't overload the term so here's a quick glossary slide I'm gonna say a tiny bit more on theory and then we'll have some fun so so kubernetes is basically works in this declarative model you don't generally say you're gonna see me generally like hey do this thing right now but you don't generally tell kubernetes to do a thing right now you don't say hey take this you know take this container image start a container on that node over there with it right you tend to say you know it's it's very it's very delegating and hands-off right so the idea is you say kubernetes I have this program I've created an image maybe I've placed that image on docker hub maybe put it on in some other container registry and I'd like you to now like download that him like have some nodes download that image I'd like at least say five to be running it like if you need more scale-up but you know I want five no less than three like let's you know just do me a favor and get that running and kerbin and he says sure and it's got a bunch of like control loops that basically first make that the state of the world so you describe the state of the world you want it it basically spends its whole life trying to keep the state of the world where you want it where you've said it has to be and then it basically loops and it says off the machine fell down I guess I'll have to reassign those workloads or ah looks like something's not responding I'll have to you know I'll have them maybe I'll kill that off and I'll start something else so you just guard the state of the world I think this is basically my last theory and I hope I haven't given you too much so the kind of target components they're a bunch of components you'll target the big ones that you hear about are especially on the first two slides here so you'll target the the API server the thing that everybody's basically doing all these all of these HTTP and God forbid HTTP requests against and asking to do things and the cubelet which we said bridged the bridge to each node into the system the container runtime you can often think of docker and often say docker but there are many there are multiple container runtimes and that container runtimes the thing that pulls the container images from the registry you know like docker hub or what have you and instructs the kernel to start up a container you have you have an etsy D server that maintains all the state and if you ever get access to that thing yeah there's not really a lot of chopped up authorization so if you get access to it you kind of own all the you own all of the persistent idea of what the system is and so that's very good for you a coupon IDs dashboard that was the thing that got as I recall that was the thing we got Tesla in trouble where their dashboard was available to the Internet and I think was allowing anonymous access to do anything but that's a web interface as I I think it's being deprecated there's metrics components and different kinds of klutz cluster services it'll give you good information those those are your usual targets just to throw in two more two more components that people that it's really helpful just you know if you're operating kubernetes defending it or attacking it or useful to know one is the controller manager and the controller manager it makes it sound like it's like the controller manager would have lots of separate controller programs and it kind of does but they're all just compiled into one big they're just compiled into one big program so the controller manager has all of the controllers the controller's of things that basically do those loops and say is everything the way it was supposed to be is everything the way it was supposed to be ah something's different I'm supposed to have five copies of this I've got four I'll go talk to the next one the scheduler and say excuse me scheduler can you find somewhere to can you find somewhere to put this can you find employee node we can put things on we tend to target these components only through the API server so attacks on kubernetes this is where we start to finally get to attack and defense attacks on kubernetes they start from they start from the perspective of compromised pot so you can attack you can attack the kubernetes cluster from across the network and it's even less effective than attacking most companies you know across the internet has been right lately it's it's we've gotten pretty good at perimeters kubernetes is pretty darn good at perimeters because for the most part it doesn't allow already try there's nothing there's no traffic being allowed in unless it's had and unless it's had a some kind of ingress defined for it so so for the most part you attack a cur bananas cluster from the perspective that you are in a workload that you're in a container and that container is compromised or otherwise under your control so what are the you know what are the one of the one of the ways the attacker might end up with that kind of control one is yeah as you'll see in some in our demos this is the this is the vector I like to I like to show it's the the attacker found a vulnerable thing that you were hosting kubernetes got remote code execution and now they're in a container and from that container they may have the ability to do much more far-ranging things because we're at an orchestration system a very automatic one they may be able to do far more wide-ranging things than if they just busted that you know that if they just busted a a you know some web server some or something that was hidden a there was in a cloud environment or in a date or in a word an on-prem data center so how else might the attacker gotten and the attacker might have just fished somebody Brad giesemann who all who's whose name I probably just butchered and whose Twitter handle I'll have at the end of these slides had it has a great slide in one of his presentations about how coop control apply and that a URL is the new curl URL into shell right if you one of the great ways to fish somebody would be if you put up a how-to article on Stack Overflow or what have you and the you know part of that article was oh just apply my you know apply my llamo from this URL you can get anyone who applied the animal from the URL then you've got places for your crypto mining to go so and our attacker could just be some internal thread actor who's looking to escalate their privileges I often you'll you'll often hear me use the word user and attacker interchangeably on the defense side that's not really cuz I think all my users are hostile I just don't know which ones of them might be I don't know who's being coerced I don't know who's you know getting curious I don't know who's actually got a axe to grind and I think we all know at least privilege is a good idea at this point so okay so so if an attacker is gonna be in a pod what are they gonna do with that I'm gonna give you a few I'm gonna give you a few things there's even I'm sure so this is kind of the general weights they could use just the network access that they get provided by the pod to access other services and I'll say very very very often especially in the micro service model a kubernetes cluster has a bunch of things that are all supposed to talk to each other and it's actually it can be a little difficult or at least takes some work to sit down and say okay what should and what shouldn't so once you're in the cluster you have access to a whole bunch of thing to a whole bunch of things many of which might not be intended to be to be communicated with by anyone on the internet or anyone in the anyone that the service that you've pwned is trying to is trying to serve so what else so the attacker could attack other containers that are in their same pod multiple containers in a pod they share well I told you they share they share an IP address they share a localhost so it's a cool little Linux name space thing which we won't get into unless somebody asks me later on what else so the attacker could make requests to the API server or to a couplet and ask those to maybe run commands in another pod maybe even interactively to start a new pod that has privilege and that privilege could be all kinds it could be a privileged pod which means it'll start on the host and have access to system devices and be able to among other things maybe mount the hard drive of the node but but there's but there's just there's all kinds of privilege you can get out of out of starting new pods and I'll show you that they might gather secrets that kubernetes provides to pods because kubernetes has a whole thing where it's it it it needs to provide it needs to provide configurations to pods because you're really supposed to make those pods come up with a static image so the idea is the pod starts with a static totally replaceable image and pulls whatever information it needs from from elsewhere you might connect to the kubernetes dashboard to perform actions maybe it wasn't accessible from outside the cluster God got hope or gosh Oh what else the attacker might find a way to interact with the sed server and change the cluster state a nice way if there's if there were network access if if there are network access control rules the network policies in place that are getting in your way if you can talk to the thing that stores them yeah you could you could potentially change them and the other one the one that's really fun to me for anything that's either in you know it's either in a cloud provider or even in an on-prem cloud is that the attacker might interact with the cloud service provider and because many workloads will one way or another and I'll show this to you too will have privilege on the on the on the cloud provider so I'm going to show you a bunch of demos and these demos are real they're they're often things that there are certain things that we do in classes I teach there also I'm gonna tell you later on we have an open source project and it's a intentionally vulnerable kubernetes cluster call it like damn vulnerable kubernetes cluster this one's called Busta Kubb we've got an Easter Egg bust-a-move video you can find if you go to the Busta cube comm website but so some of the things I'm showing you are basically available as scenarios that you can play in this open source free den vulnerable communities cluster so I've done a thing where I've made my videos part of the slide and so now I'm gonna see if that actually works please work and if it doesn't we'll have to use YouTube one way or another or something else okay Color Me very embarrassed let's just see if this one's the one you should always test your demos even when you record them so so I tell you what we're gonna we're gonna find another way to do it and I am going to let's see I'm gonna unplug from the projector for a second because that means I'm going to go and look at my own file system and I'd like to not share it [Music] and here we go just embed the videos and the slides they said it'll be fun and they said anybody old enough to have played to played Warcraft and you click on the guy and he's like what and you click on him again and he says something else and he says you know join the army he said see the world he said okay so here's our here's our demo and hopefully entering oh well that's not a demo is that a demo that's not a demo either anybody's seen my screen there we go let's just hit this on mirror and call it a day okay cool so that's not the right don't mind me I'm a I'm a I'm clearly a noob I'm a need alcohol let's so this is this is one of the this is one of the best of Kubb this is one of the bus to cube scenarios so you go up and you see a wordpress site and the WordPress site I will tell you is hosting is hosted in kubernetes and you fire up maybe dirt Buster to go looking for hidden content and enter Buster takes a little bit of time but we can run it and let's go a little faster and what we find is backdoor dot PHP and so when we surf to backdoor dot PHP this is what happens I get this thing where basically this wordpress instance like so many other wordpress instances has been pre compromised for your convenience so i type ID and it says your dub-dub-dub data and hopefully I shut down Bear Buster because it always choose a ton of resources something we always say in the class that was my laptop's running slow have you turned off there buster no it's 500 requests a second well that'll that'll do something so I'm going to stand up a quick little I click a quick little web server in Callie because I found a I found that reverse shell so in a stand of little web server and my purpose the web server is to be able to you know hand a be able to hand a meterpreter or a you know or if I wanted to you know whatever some other kind of River shell but I'm gonna go for a motor pretty and so I just put in here like hey could you please pull this meterpreter binary I've created from my web server and while you're at it make it executable and while you're at it run it and I like to fast-forward my videos because who doesn't and go look at the you go look at the web server and mrs. Ben has been pulled the meterpreter binary but also a copy of coop control the the command-line utility that you that you use to do damage I mean to administer or use a kerb entities cluster so I'm gonna pull coop control and so the other thing I'm gonna do is just take him make sure that that it's landed here so here's mrs. Ben my meterpreter binary Hughes Cube control cool and now if if recorded J will type a little faster we'll run the meterpreter binary and I'm gonna go over to my Metasploit console and actually be like oh wait I have them I have an incoming meterpreter that needs to connect to the console I should probably start the console this ordering is just great huh but that's often how you work so I'll interact with that session and I'm now in a pod I know it doesn't look like it it looks like an ordinary Metasploit you know pwned Metasploit instance so but i see my copy of coop control I'm happy and when I run mount I get a tremendous number of file systems I'm not looking at one hard disk or what-have-you and a bunch of them put there by docker and by kubernetes and so the one that's really interesting to me is this one that says run secrets kubernetes service count io service account as soon as I see that directory mounted I know that I'm gonna have some level access to the API server and I'm happy so I'm gonna go and first I'm just gonna go and collect my flag I built built this scenario as it captured the flag so let's look at our little flag and it says hey you found the server's backdoor you might notice this isn't your usual Linux system you're in a cluster my friend it's time to bust a cube this container doesn't have any more flags can you find your way to another pod intent so cool thing in kubernetes is like I said one of the things you can do is see if you can move to another workload so you're in a container can you move to another one so I'm gonna see what my service account can do I'm gonna go and look in that directory and that run secrets kubernetes i/o service count directory if if recorded J will type just a little bit faster and I look at that namespace and it says you're in the marketing namespace so the totes of that directory has a token it has the certificate authorities public cert and it has a name it has a description of what namespace you're in and that namespace like I said is kind of a way of organizing so we're in the marketing namespace we'll come back to that later and then I'm also gonna go and look at the environment variables that get pushed into every pod and my environment variables will tell me where to find the kubernetes api server so there's this assumption in kubernetes that every workload will need to ask carbonates to do things i don't know if that's really a the greatest assumption but i think let's just call it not the strongest of defaults so I've got the API servers IP address and port and I'll clear my screen and I'm gonna set up a quick little cube control where I'm saying alias alias coop control to the path to coop control the token paste it in from this long path to the token the certificate authority path the namespace and the server that I just got and so every time I type coop control all that stuff will be passed in so let's just kind of skip and let's see what did I do so I said let's do it quick get pods and get pauses my is honestly my go-to just to figure out if my setup is working and so it says oh there's two pods you're the WordPress one and I ran hostname and it says that's the one you're in and then the WordPress my sequel one and that's another one so let me see if I can move over to it because that was what the hint said so I'll do coop control exec IT looks a whole lot like docker exact IT huh and what program I'd like to add to that container I'm gonna add a shell I get this kind of error but I'm in the I'm in the container I've just moved over and have probably moved over to a different host like I'm probably somewhere else in the data center now so there's another flag and we'll go get our flag and I can't clear my screen because my terminal is not set so I set it to VT 100 because that was one of my first back in the BBS days because I'm seeding the old and it says hey you found your way to another container actually you're on a whole new virtual machine we're moving from one container to another and kubernetes system often means switching nodes your end goal is to get the developers nan containerized host so is to get not just not just to own the cluster or potentially but to but take but to get to a machine that's not in the cluster so to get there you're gonna want to see if in hop tenants tenants are usually separated by namespaces so let's see what we can do I know I'm fast-forwarding through my stuff but so I'm in I'm in the other and that my sequel want to try get pods and the problem I'm running into is like hey you're not allowed to get pods so the cool thing is I'm in a different service account I was in the marketing wordpress service account it's allowed to get pods the marketing my sequel one can't so over time I might want to like go around and pick up all these service accounts and like hold on to them remember it was just tokens so if I can store all the tokens I'll have like I can end up building privileged wider than just what one thing had so I did get nodes and I found out the IP address for node here's a name but this node has an IP address this is this one is this one's running in a cloud provider and there's a cool thing so you can talk to the couplet and this is thank gosh finally not a default but some clusters will have it you can talk to the couplet on it's got a couple ports 10 250 I'm gonna go and say can you give me a list of running pods and it gives me JSON because everything in kubernetes is JSON yamo so I'll take that and I'll run that again and I apologize for this big block here so I'll run that again and I'll add some JQ JSON query really good tool to learn and I'm gonna say give me all the pods kind of better formatted give me their name and their names face and what I'm looking for is to see if there's a pod that's in that's in another name space besides the marketing one and I quickly find one there's a dev and that dev pod so kubernetes loves to tell you about anything you want and that's really really nice as an attacker especially if you can script it so I've got dev sync I've got dev web these are two containers that are running in the same pod I've got the image they came from so they're up on docker hub you could pull them yourself if you wanted and as an attacker that's often what you're doing is basically saying well what workloads are running it can I go look in container images and see if there any secrets or anything that would be really helpful to me it's very very nice of the defenders to can view descriptions of their workloads so I'm gonna look at that I'm gonna run another command and this is I did running pods I'm gonna I'm gonna tell the couplet hey would you mind just executing a command for me so I'm gonna pass it run and then the namespace the pod and the container name and I'll do command equals ID and I get an error code that says basically no either there's no shell in there or or I'm not allowed I will tell you that there's no shell in there so but remember there were two can there were two containers in that pod so I saw dev sink and dev web and that's a really common a really common thing you see outside of containerized environments like you have a web server and you have something that's you know something that's there to pull it content in from wherever that contents being created or stored so I was hoping I could get to dev sink because it'll have that it will have the credentials for pulling content I'll have an SSH key I couldn't so let me try dev web so devil whip I can run commands in Iran ID says yeah that thing's even running his route very nice and so let's try a different command let's look at the file system oh look another flag I like flex I love doing CT UPS and I love doing CTFs actually like first doing them as an attacker and then going and basically doing them as a defender and saying how many didn't have the attack can I break with proactive stuff with stuff that you could have thought of before you knew that you had the vulnerability so this thing says hey you're not in the marketing department anymore you're in the development departments namespace so can you go take over the development departments non containerized machines and their big fans the matrix so it's name the mainframe name for the machines mainframe that may or may not have been in the second and third movie depending on your religious beliefs there there are many people I know who acknowledge the truth as they believe it that there were no second and third Matrix movies only quests for more money so here we go so we've got turn into a gat pas we're trying to do a get pods so I am I'm in one pot asking the couplet on another on another host if it mines running commands in another pod and I'm asking that pod to run coop control commands I find out okay dev app isn't allowed to list pods but it's kind of cool I now know that I've got a third service account I've got I've got the dev dev app one so I'll try to get secrets let's see what secrets are available oh nice there's a machine there's a mainframe login and an SSH key well the two pods in there the two containers in there are gonna likely have access to the same secret they've been placed in the same place they don't necessarily need it in dev web but what the hell let's go and ask for a copy that's secret and I promise the the next demo is hackers themed and choose-your-own-adventure so we're gonna have some fun so here's our SSH key you're like that doesn't look like an SSH key and that's because it's a secret it's it's okay Glee encoded its base 64 encoded so we can decode it and there's our SSH private key and we can do the same thing with the other secret here so we can find out basically what is it used for so I'm just gonna fast-forward a little bit and tell you that it's the that it's a user it's a user and a host and AWS but not part of the cluster so now we could SSH into that and we're in the developers mainframe and it says hello you made it's a machine mainframe can you for extra credit can you get credentials to take over the whole cloud environment and then we can do that but I'm going to set the rest of that aside because we do that in another demo so so let's see let me go back to the slides for a second and here was our and I'm sure that'll be because the vagaries of having the videos you know in a separate program I'm sure I'm gonna be doing a little bit of going back and forth and having the whole screen do this so here's our tack path we got a batter preneur in the WordPress container we moved into my sequel container with the coop control exact we used the my sequel containers totally unfettered network access inside the cluster to reach a couplet that we wouldn't have been able to reach from outside the cluster on the master node and that couplet led us basically take a look at a take a look at take a look at run commands in another container in another namespace and from that other container and that other namespace we found we figured out there was a there were secrets there was an SSH key so we got that SSH key we authenticated to the high value developer machine and later on we ended up getting to take control of the AWS account and we're gonna skip that right now because we're gonna do that in a much more fun way so we're gonna play hackers choose-your-own-adventure and I'm gonna drink some water for a second whoa so I like the movie hackers it's a it's a fun movie it is an oldie but goodie it is thoroughly realistic we all know that whenever we're hacking we we see like strange translucent 3d models and and you know of buildings and we fly through a city and and it's really cool and all of that was on was clearly done with the technology that had Apple twos and floppy disks so I mean I remember 3d rendering back then I wasn't doing and I was watching people do it and yeah no not on that so here we go so we're gonna start here and and again this is where I have embedded video so we will unless I click on that and it works let's see oh hey look at that now can we fullscreen that is there a way No so we're going back to oldie but goodie and here we are so first this is gonna look a little familiar so we've got a we've got what's that Oh while you're on the slide and I'm Emma and I'm on the video ok boom and switch come on I'm gonna windows computers soon and that's a lot for me to say trust me so anyway here we go so we've got the we've got the Gibson control room we have Penn from Penn & Teller who's apparently a techno weenie according to Ben from short-circuit who somehow was playing an indian and talking about mr. Johnny Five but mr. Johnny Five here was the plague and and we have this nice little form that's called the Gibson control room and the Gibson control room clearly lets you like leave a message so it's based on a it's based on a actual on a kubernetes documentation vulnerable app they didn't mean for it to be vulnerable but it's only a little vulnerable so I've added a vulnerability to that so the so what we've got is I look at the source on that page and I find that the forum is basically being provided by some JavaScript so if I go look at that JavaScript what happens when you submit the form is your your referencing guestbook dot PHP you're sending to guestbook PHP you're saying hey command set key messages value this okay really you're gonna let me just you you're gonna let me decide what what key I'm gonna set you're not just gonna have me set whatever key you know was meant to be set by this forum so that was there bug and what I've added to it is I have something that actually runs code out of a different Redis key in this you know back in Redis database so we go and take the Gibson control room and if we were to type a message you know it shows up this is the intended this is the intended purpose but instead we're going to paste that same thing that happens when you submit the form except instead of the message key we're gonna try a different key or we'll try a let's see how it works I'm gonna hit foo and we'll put in just some key some random key oh cool okay so Java scripts probably updating that key let's try going to status PHP so I'm asking you to imagine that we did there but sir or looked at a robot's texts we found a status PHP I'm not doing it because well I'd like to spend our time on kubernetes so I go to the status PHP page it says hey no command to run I'm gonna set the reddest key the Redis command key to this so that you can reload the page and and that will get run so if we reload the page I get the output of you know I get the output of go to crawling that web server it says all services are operational it here at what was it Ellingson mineral or Ellington mineral Ellingson I think so as long as there's a command key let's go and set it and I've set my command key now to environments to run environments and like we did before to find a kubernetes cluster to find the the communities API server and here we go so we know that once we're in a work if we reach out to ten 9601 we'll get to the kubernetes api server so I'm gonna go a little further kind of similar the last time I'm gonna curl in I'm gonna tell this thing excuse me would you just pull in a meterpreter binary I created and connect back to me so I have it pull it in I set that key and now when I run status the page hangs I love when pages hang when I've just given them binaries because it generally means the binaries running successfully so if I go over to my terminal where I've got a menace point handler rating this time I find there's a session waiting for me and the first thing I'm gonna do before I start interacting with a shell there's I'm gonna upload I'm gonna upload a copy of coop CTL which I've stored on this cally system in this directory so I plug coop CTL and I'm gonna upload some the amyl which we'll get to later so I've uploaded some Yamma which we may or may not get to later depending on your choices and I've asked for shell head PWD looks like my video is cut off by one character a hit ID it says your dub-dub-dub data user I try SS aging and alright well I don't know what I try because it's because that was the end of that part and now we're gonna do this part so that we're a little bit gentler I'm gonna ask you so do you want to try the yamo files that I put up worse were ones that would mount a volume on the we try to mount a volume on the host like they try to mount the hosts home directory the old you know docker eggs you know start a container mounting the host so do you want us to start the hosts mounting pod and see if we can take over I'm gonna see if we can take over a node or do you want us to exec into another pod I heard a host I heard the host mount first so we'll try that so let's see boom and boom here we are I'm in that front end and we'll set coupe control executable so we can run it we'll go and set up our coupe control we'll try to run coupe control apply realize wait we've got to set all those parameters we said before sometimes you get lucky and so I'm gonna take that same directory that same run secrets kubernetes io service account that's being mounted into the pod I'll set a variable called third of that export der that just to make this a little bit easier to type a little more manageable so if I look at the door got the CA cert namespace token just like before let's go and run alias coop controls gonna mean that same way we did before so servers out on 10960 one that was what we found from that environment variable and now let's look at our Elias that's that token being pasted in so coop control get pods cool we've got a front ends we've got front end pods we've got a Redis master we've got we've got an Apache status that thing we saw before we're in the front end so let's try launching a pod could control attack applause so good control apply I like it as OOP control attack attack pod and it says sorry your service count front end isn't allowed to create resources wha-wha so and this is where I get to my slide so we'll do that real quick I was considering for this one oh damn it let's just keep the me screen mirrored so I was considering for this one doing a spandex quote from the movie and I decided that wasn't a that wasn't a sentiment I wanted to voice so I see that I am I see that I'm going to make sure that I'm moving on time so at this point we'll go back to our original we'll go back to the other thing we could do so our two options where do we want to launch a pilot or exactly the back end so let's exactly a back end and let's try exacting into another pod so we'll go like we did the other like we did in our other dental try exacting into a pod keep wishing we had a better terminal coop control get pods I see that there's a Redis there's a Redis that Redis setup remember I was pushing Redis keys I might try to investigate the Redis and see what else now that I've got direct I've got more direct access to it but because Redis is very often well almost always on authenticated but let's go to the Redis pod so I'm rooting there I'm Redis master I'll grab a copy the meterpreter binary I've been using just because that's a very convenient way of putting files into something and I'll background that and I'm gonna I'm gonna switch over now that I've got the I've got the rightest master that pod and running my interpreter so I'm gonna switch over and catch that session and so in that session I'll go and upload the same coop control that I did before I'll also upload that attack pod that I just used before and I'll also attacks and add something called attack demon set just in case we need it so now we'll get another choice so you're in the back end do you want to try host mounting pods or do you want to check out the cloud EP ice cloud api's call ok so here we are and I'm going to show you this in two ways and can I get a rough time check how much time we let me be late Oh less than 5 ok so this is where I normally kick on the speed control so ok so we're in a pod we're in that Redis master pod and we say hey I'd like to talk to the metadata API so every cloud provider gives you one six nine two five four one six nine two five four that part's common what the calls are that you make against it they document wonderfully and you read the documentation so this is on this is on GC P so I'm gonna go and say can you tell me what the project ID is I need that project ID to do API calls but then the more interesting one is usually where you say would you mind in addition to the project ID which I'm gonna set into a variable would you mind telling me what service accounts are assigned to them to the machine I'm on so the machine is pods running on and there are two but default is mapped to that other one so I'm just gonna use default because it's easier typing and I'm gonna go and say okay for that service account would you do me a favor and just give me a bearer token that I can use to act as that service account and it's a short here's your token it expires in like a half an hour or an hour but I've got a token so I'll set it variable to that and now what I'm gonna do is say let's go and I've set that token to that and I mean I need to refresh that token so most of my commands will pull that token over and over every command so I'm gonna use that token type out authorization bearer token where the tokens provided say I'll take JSON and now I'll say hey for this project I'd like a list of buckets and it says oh there's only one okay well what's in that bucket and so I basically say give me a list of objects and the way I do that is it's like /b for list of buckets then it's the /b that bucket /o for a list of objects and then for those objects I can run through them now on this one I don't find anything that's all that useful I don't find I find I don't find what I find in some cloud environments I'm going to show you in another demo somehow and but I do find a coup B and V so I can pull in a metadata instance for this cluster that was set by the Installer and when I do that what I get out of it is among other things a couple service count tokens for the cluster and that ends up leading which I won't now show you but that that ends up leading to win and so just for the end of our for the end of that that then leads to our crash and burn so if if Ming will let me have one more minute I will show you a cooler version of that we have a tool that were that we keep soft releasing but never kind of like putting up on Twitter so it's called peratis and peratis will if I if I actually just I'm just nervous so it's in here but somehow I'm let's do this so peratis is basically a remote access trojan for for kubernetes you run this and it does all kinds of things and I'm going to start this from the beginning but it is under two minutes and then I will get the heck off the stage and tell you that you should ask me for stickers so it's got a bunch of things one of those I had said ok 13 go get the IM credentials from GCP the same thing we just did get the cloud service account and then come on J type faster highlight faster so that's for that same one I just got and now let's take that and let's pull let's go looking in GCS in blue and in the button storage buckets for service count tokens oh look this installer this one was cops in left left service count tokens therefore all the components including a cluster admin which would be perfectly useful and so that's the like basically that's probably roughly what happened at Capital One recently where it was talked to the metadata API get your get your get your cloud account use that to go raiding for information or access so anyway so now once I've done that varieties will basically store every every count it gets and then you basically say ok I'm in a switch context and you go and you go through and you collect all of the service count tokens then you basically have this big old store that's your loot and it keeps it in memory and and you run around and basically get more access and get more access we've even got a this isn't this video will be up on YouTube but we've even got something that basically takes over the nodes if it's allowed to if it's allowed to mount the hosts path volume it'll go and add a cron job that will send in that cat shell back to your back to the IP address of your choice so once we have this cluster admin we're allowed to we can go and do everything which included like hey now show me all the secrets and all the secrets includes all of the tokens all the service counts for every up for every component and and varieties will not just grab and will not just list these and get them but also store them let you use them so it's a fun little pen test tool and it's up on github we'd love some help we'd love some help producing it so it's an in Guardians tool we do a ton of communities working Guardians and I will now get off the stage and take questions all the way out in the hallway so if you see me please ask me for sticker

Channel: DEFCONConference

Views: 2,300

Rating: 4.8947368 out of 5

Keywords: DEF, CON, DEFCON, DEF CON, hacker conference, security conference, information security conference, information security, conference speakers, hackers, hacking, hacking videos, security research, Packet hacking

Id: 2fmAuR3rnBo

Length: 54min 50sec (3290 seconds)

Published: Thu Nov 21 2019