Hacking into Kubernetes Security for Beginners - Ellen Körbes, Tilt & Tabitha Sable, Datadog

ah okay um new day new project time to write some code then i'm going to connect my local tooling to my development cluster and then i can get started i'm gonna do cube cuddle apply my um my usual tooling let's see oh wait what uh no part part something so let's let me do this by hand fail to create node parts the parts already allocated okay so someone's playing with my cluster parts i i told everyone 31337 that's me don't mess around with it gosh darn it okay who did this let's see who's around okay there's a lot of people here what are they doing i don't know i can't look at it can i look at them one by one no i can't okay so maybe there's some security in place here uh and i can't see what everyone else is doing okay you know what i'm not calling the security people they are not fun i'll i'll do this on my own um let me check something out here i remember seeing something funny in the onboarding documents so i was looking around the docs and i found this dav database controller so the controller creates new database instances pre-populated with fake data for use in dev environments it hangs often you can exact into the pod and if it's really wedged you can restart the process by deleting and recreating the pod and someone said something here about fixing our back privileges okay so we're using our back and our back is that thing where i have some permissions there's some things i can do in the cluster and there are some things i can't apparently i can't look at what my friends are doing to figure out who's messing with my stuff but if they set our back incorrectly which is very easy to do because it's convenient to set it incorrectly then maybe i can use that cascading effect where i am allowed to do some things i'm not allowed to do others but through the things i am allowed to do i can go around and manage to do the things i'm not allowed to do just in a very roundabout way so let's try that so we're looking at a dev database controller let's let's see if we can find that what can i do normally let's see keep cuddle off can i so i can do whatever i want in my own namespace but when i try to look at someone else's so let's see let's see what catherine's doing here when i try to look at someone else's namespace the stuff that i can do is very little almost nothing let's check something out so let's look at what's going on again and i have this dev environment controller i should be able to create pods in it yeah the documentation was right so for pods i get a star which means i can do whatever i want with pods and i can exact into pods as well so let's let's try and make use of that and see what happens let's see uh let me do cube cuddle get pods and here's one so let's try and execute something in here so cube cuddle execute on that namespace in this pod and give me a shell okay i got a shell inside the controller and that means that i can probably now make use of whatever permissions the controller has so do i have tooling here yes i do thank you when you leave tooling around attackers become very happy and speaking of attackers so what can i do let's see what i can do from within this controller let's say again what can i do would you look at that stars stars more stars like a great philosopher once said we're made of stars but our back shouldn't be and oh boy they were right so i can do whatever i want here which means i can probably uh see what my friends are doing yeah i can see what everyone's doing now so let me filter through this a lot of people are doing a lot of things let's see who's using my gosh darn part ingress pointing here and that's joey oh joey i told you to stay out of my turf joey yeah you're getting grounded that's enough joy so keep cuddle edit on joey's and here we're going to replace joey using my part with joey using 420 because i know the kind of stuff joey likes to do so get out of here joey there you go and now i can probably get out of here and run my stuff finally and there you go now everything works and now i tilt up and my application's working so now i can go write some code and and be productive and joe is going to have a bad surprise in monday and that's his own thought well here's what we might want to do first we'd want to become cluster admin this could happen by either compromising the cluster and becoming cluster admin or maybe you are a malicious admin hello this is tabitha devsecops enforcer joey joey joey calm down calm down i'm i'm gonna help you but you gotta explain it to me you gotta tell me calmly what's going on okay so you you can't get into your dev environment well let me let me have a look okay so i'm going to look at your service first because that's always been a problem oh joey of course it's not going to work for you on port 31337 it says here you changed it to thirty four twenty joey is this some kind of joke oh you you didn't change it oh oh let me check on the devon controller maybe maybe something bad happened to it [Applause] ah devon controller looks okay um joey i'm i'm gonna put you on hold for a moment while i do some more investigation of this i'll be right back if this was changed and and joyce says he didn't change it i'm gonna have to go and look in our log management system to see if the api server audit logs can tell me what's going on okay so i have to first look for our kubernetes audit logs and let's do in the namespace joseph pardella okay there's a lot of logs here so let's let's look at what has been done inside his namespace okay patch that's that's going to be the ones where things were edited let's have a look at let's have a look at these um here's one that says devon controller recently modified joey's ingress okay that's that's interesting why would the dev end controller have done that inappropriately uh let's see what has let's see whether anything strange has gone on in the devin controller okay so here's some here's some typical kinds of kinds of changes let's see who's doing them by looking at the username hmm a lot of service accounts and then ellen what has alan done in the devenv controller recently okay so it looks like for dev maintenance ellen was allowed to execute a show in the devnf controller and then right afterward the devent controller service account modified joey's uh modified joey's service yeah that's that's suspicious girl i think you just got yourself in trouble okay uh better get back to joey and make this right for him hey uh joseph thank you thank you so much for holding i've i've done some investigation and i see a simple misconfiguration uh i am gonna fix that for you and then you should be well on your way yeah yeah hey what's up that's exactly what i was expecting hello hello ellen it's it's you'll you'll recall i'm director of devsecops enforcement i have one of those things i need to ask you as part of an ongoing security investigation do you do you have a moment for that yeah i mean i'm in the middle of a game but yeah sure what's up great thank you so much for taking the time to meet with me here what can you tell me about kubernetes api server audit logs really you need to ask me that uh yeah yeah tell me about all uh sure yeah so all that logs is like when you do stuff to the api server it logs that stuff yeah all of the activities taken on the api server so like do you know how one might enable api server audit logs i i think i saw by default on the managed stuff that you see out there if you're rolling around you you need to enable it by yourself but i don't think we are i mean once it's enabled like should you just leave the logs sitting around on the server um lots of friends of mine would say so but um i i would advise you to put them somewhere um yeah like where could we send them uh if you're doing elastic stuff vlog stash uh i think here we use datadogs so you don't keep them in the cluster because that's what you should be in the data dog yeah that's that's right so like what do you think i can see if i go into datadog and i view our api server audit logs uh sure yeah you can see the uh when resources are created or modified or deleted uh that that activity yeah like what what kind of resources are we talking about here oh you know stuff like pods deployments jobs node ports yeah yeah yeah no node ports so like d do you have any idea where i may have gotten these logs that show you modifying joey pardella's dev environment to point at port 3420 do do you think you'd like to take a moment to explain yourself you got to get your act together ellen you've got to cut out this childish movie hacker like garbage behavior you're supposed to be a professional and this ain't it you gotta get it together you know if security people would listen to developers maybe i would be writing code the past hour instead of browsing twitter ah what is this look it up baby huh yeah that was a fun one i wonder what is this exploit that they're talking about here cve11253 even rhymes that's cute so apparently the cve is improper input validation on some versions of kubernetes and causes the api server to consume excessive cpu or memory possibly crashing okay so this is a dos and you know if the development cluster would be out of commission for a day i would get the rest of the day off are we vulnerable to this cube cuddle version we are 16.1 and what we have here is 16.2 yeah you should be vulnerable where's the poc all right this is cool uh apparently we're dealing with kubernetes vulnerability here and they don't come around very often but when they do they can ruin your day and if we're vulnerable to this it means the security people in our team haven't been doing their job because they should be signed up to the mailing list the security getting the security announcements they should have patching and upgrading and all that stuff as part of our software lifecycle management and if they're not doing that then i suppose they're gonna need to learn their lesson yeah they are okay and we have this exploit set up well i think it's time to sending a billion hunks using a flock of 400 flying geese and is this working uh nope our server does not reply any more time to enjoy my day off ah devsecops enforcement this is temp can you please hold devsecops enforcement this is tabitha can you please hold oh my gosh what is happening [Music] thank you so much for that great demo miss libby having a principal engineer like you on the staff is a huge inspiration to the entire rest of the engineering division now for the last part of our monthly all hands i'm going to give an update about security initiatives let's see how do i share my can can can can everybody see my screen unfortunately so all right today i need to talk to you because of some activities of a questionable nature that have been happening in our environment we're going to need to enact certain new security policies company-wide um why why just now are you saying how long have we been without security in the company from each other and in some cases to protect the company from your childishness and what were you doing the past couple years we're going to talk about three things today admission control network policy and container vulnerability scanning in our ci pipeline we'll start with admission control so admission control is what you need to be able to enforce policy about what settings are allowed to be put on a pod such as privileged or host name spaces so we're going to be deploying new admission control policies that are in line with the kubernetes project pod security standards document the one that's about to be deprecated right thank you ellen no the pod security standards document describes best practices for running pods in kubernetes i think you're thinking of pod security policy which is a feature that was deprecated in 1.21 but will be replaced soon with a new admission controller and that's irrelevant to us anyway because we're going to be using gatekeeper but thank you for your concern so getting back to the presentation the importance of admission control is because our back works at the level of kubernetes objects like can you read or write this pod or service but it doesn't have any opinion at all about whether a pod is a harmless web server or a privileged command shell that exposes all of the node's data so to enable that kind of control we need to have admission controller and admission control then gives the api server opinions about what should be allowed and what shouldn't shouldn't you be doing that yes and i'm enforcing my opinions using an admission controller thank you ellen so this will help us to not develop using dangerous pod features that could help an attacker if they got shipped to prod and it will help to isolate our dev environments from each other the next thing that i'd like to share about is network policy as those of you who are involved in our attempted proof of concept years ago no micro segmentation has been a dream of mine for years but it was way too hard in our bare metal data center so you know you love to dream yes so in our kubernetes environment we already need to precisely define which services our services talk to for use by our service discovery system so we're going to be adding stop talking i wish you'd stop talking to all of us ellen as i was saying our service discovery system will be integrated with the network policy engine so most of the work of writing policies will be taken care of for you and after you've added the last few touches then kubernetes will enforce what can talk to what within the cluster in order to reduce attack surface the last thing that i'd like to share is that we're going to be rolling out container image vulnerability scanning as part of our ci pipeline now that we're web 2.0 scale our developers have been struggling to keep up with the over 12 containers that are deployed in production so with this scanning the the scanner will run every time a new container is built by ci and it will send an email to the developer about any findings there we're not going to be blocking deploys right now because of these results this is only for your information but if we find that these results are getting worse over time there may need to be further follow-up so thank you all i want to leave with one last bit of information or perhaps a warning for some of you we will now carefully audit all logs especially the kubernetes api server audit logs in order to detect any malicious behavior or shenanigans that might be occurring so thank you all so much for your time this afternoon we will now carefully audit all logs yeah over my dead body you are i don't like working in a panopticon and this is way too much of a dystopia no one's auditing anything i'm gonna get in there and see what they're trying to do and if they're trying to make me uncomfortable i'll just take some drastic measures now to do that i'm gonna need root access to the development environment server and they talked about admission control but they didn't mention that it was deployed yet and if it's not deployed it's not so hard to run a pod that gives me a root shell so i'm gonna try my favorite incantation here and let's see if it works let's go back to duffy this is such a classic and i love this forever is this gonna work uh no it doesn't okay what else can i do what if i can mount a host path so that i can access stuff that's on the node from a pod would that work let's see i have this script here let's try and run it and see if it gets true so i'm going to apply and the night admission okay so can i do something else what if i can mount a pod onto the host network is that gonna run and to test this all i need to do is run any old pod and it can be anything it doesn't matter what it does so long as i have this option here and yeah it went through this pod is probably going to crash but the point is the admission controller didn't catch it and if i have access to the host network i can interact with the host and so far i don't know what that's going to get me but let's do some research aha i found something cv 2020 15257 container d 1.4 abstract unix domain socket this will do just fine and there's even a nice write up here about ways to use this so this is perfect i'm going to set up my attack and then we go okay we're all set this is going to be a two-stage tag first i need to set up a reverse shell so that when my exploit calls home i can pick up the call and second this is what i need to run what this is going to do is i'm going to interact with the host network directly and because of this container d vulnerability i don't have to talk to kubernetes i can tell container d itself to run whatever container i want so this means i can run that hostpath container that kubernetes wouldn't like me because even if kubernetes security is set up 100 correctly if i can exploit a vulnerability in something that kubernetes is running on top of i can bypass all of kubernetes completely and then it doesn't matter i'm in so let's see if this works and it's running is this working yes it found home now final stage did this work yes i'm in okay now let's see what those security folk are up to uh oh right i'm not using docker i'm using container g ci ctlps cube api server let's see what they're logging here i'm going to do siri ctl logs here and let's see what they got on me oh boy they got everything look the task hosts not stuff that i was running it's all here this this will not stand but wait a sec i think did i just see something strange what is this this is unfamiliar and i don't like unfamiliar things what is this part it's running on cube's system i don't remember seeing this on cube system before let's check out let's look at the container and oh boy this this i don't know what this is but it looks highly unusual and what this is running is wait nat cat no no no no i'm the one running that cat i'm the hacker here why would anyone else be running that cat unless fascinating a hacker so what should i do i mean if there's another hacker here and they cause real trouble i got a record people are gonna try to pin it on me so i think the only way forward for me is i need to play goody two shoes and at least get my rear end out of the fire yeah we need to call tabby you keep bothering me what do you want now ellen tabitha when i was a child he speaks a child understood as a child as a child when i became an adult i put away childish things q but really why are you interrupting me we've been playing games we were supposed to be on the same side okay yeah and are you familiar with an admission of controller on the cube system namespace executing netcat i mean no yeah why why would that why would that be that oh we should really start the security incident response procedure um let me create the incident record and paige page the forensics team for us we can start filling out the forums together i can take your initial report right now while we're on the call what i'm impressed okay okay so here's the first question all right that's that's the last of them okay thank you so much for bringing this to our attention ellen sure maybe i was wrong about you [Music] you

Channel: CNCF [Cloud Native Computing Foundation]

Views: 6,056

Rating: 5 out of 5

Keywords:

Id: mLsCm9GVIQg

Length: 26min 13sec (1573 seconds)

Published: Fri May 14 2021