DIY Pen-Testing for Your Kubernetes Cluster - Liz Rice, Aqua Security

hello welcome to this morning's session yeah Thursday you're still here that's something close yeah so we're obviously in the best room so we talking to security all day here which is wonderful at least this morning and then I think also this afternoon I'm Maya I was on the program review committee and I'm your track host this morning and I'm here to introduce Liz rice who'll be talking about DIY kubernetes pentesting liz is a technology evangelist at aqua security and I'm sure you've seen her many times before she used to be the co-chair for for coop con and you've seen her on the big stage please just a reminder that if you have a seat next to you kind of move in so everybody can have a chance to sit and make sure to rate the session on shed after their talk and if there's time for questions at the end I'll be running around with a microphone please let's take it away all right thank you very much Maya there are some seats quite a few seats in this corner up front if you're looking for one all right so thank you so much for coming to this talk about DIY penetration testing for kubernetes I work for aqua security and we help enterprises secure their kubernetes deployment but we don't offer penetration testing services it's something that specialist companies offer and penetration testing is where you kind of act like a hacker you see what you can do to exploit a deployment and there's a whole set of tools that both hackers and pen testers will use to explore what's possible to kind of scan what ports are available and then run various different types of tests to see what you can do in a deployment of any kind not just keeping eTI's this is like a set of just some of the tools that are available nmap is one of the most famous ones so famously actually has a whole page on its website about n maps use in movies this is from Matrix Reloaded and it doesn't actually say and Mac there but you don't necessarily want to if you want to do it yourself you don't necessarily want to have to ramp up on these tools you know if you're a busy kubernetes administrator this this is a whole specialist scale during penetration testing the reason why you might want to do some kind of pen testing is because of stories like this where just David just every year ago a bit bit over a year ago there was this situation where Tesla had rather unfortunately left their kubernetes cluster opened to the Internet and that - sorry that had left the dashboard open to the Internet the dashboard has cluster admin privileges and so basically anybody who found that dashboard was able to do things to their cluster what they chose to do was do some cryptocurrency mining and when this happened we thought well a little bit of basic penetration testing a little bit of kind of just checking what's possible out there would have helped Tesla and maybe other companies out there see if they've done anything kind of grossly misconfigured like leaving a dashboard or the API open to anybody to abuse on the internet so we wrote this tool called cube hunter which does some of this penetration testing specifically looking for miss configurations in kubernetes it's not in any way you know a replacement for a full penetration test but given that penetration testing can be pretty expensive it's nice to have something that can check for open issues open ports and so on so what I'm going to do this morning is show you some of the things that are happening inside keep answer like what are the types of attack that cube hunter is going to try for you which you could type in the command line the same way as I'm going to or you could run keep on top and generally the principle is it's looking for open ports mostly looking at the default ports that kubernetes components may or may not have left open and it's attempting to make REST API requests when it finds open ports that is not a comprehensive set of penetration tests but it will help us find some you know potential issues before we talk about exactly what what kind of tests there are I just want to reset a little bit about making these restful api requests so you could be making them over just plain old regular HTTP or you could be using HTTP now lots of software kubernetes components included will yet they're offering an API they're acting as a server for that API if you want to offer HTTP you have to be configured with a certificate that tell that identifies your service and you know maybe that's a little bit of a pain to set up so quite often software is written with an HTTP port as well just to make it easy to set up and easy to run many of the kubernetes components offer both the secure and insecure port the reason why you have them on two different ports by the way is that HTTP connections start their life as HTTP and then get promoted upgraded to include the encryption part and that means you can't offer both HTTP and HTTPS on the same port so if as a hacker we find an HTTP port open that's kind of hmm nice even if we find HTTP port anybody can access that anybody can issue API requests over that perhaps the API is going to ask us to identify ourselves or the connection is going to ask us to identify who we are as a client and perhaps it's going to go further and actually check that ID against set of permissions in kubernetes we have our back for this so you have to identify yourself and then the service is going to check whether or not you have permissions to make the request that you've made over the API so what I'm going to do i I have a cluster here that is terribly badly misconfigured and I hope none of you have a cluster that is a badly misconfigured as mine is and over the last probably year eighteen months lots of work has gone into kubernetes to make it have better defaults so some of the things that I have opened up on you know that that's not how the default administrator installation will appear but it's always worth just checking if you've got a cluster that's maybe two years old maybe it does have some insecure ports open alright so thank configure your cluster the way that I do because it's very foolish I'm going to start by looking for an insecure port so I've got a single node cluster running in a virtual machine is that big enough for people to see shout if it's not always okay let's make it a bit bigger right I am just going to use curl and I'm going to I happen to know that 228 actually I'm gonna go back and find it because I might get the address wrong right so I'm just gonna do an HTTP request okay on port 80 which is the default there's nothing there connection refused means the port is not open far as an attacker I might just issue a car request to see whether there's something there port 80 doesn't come back I might try port 8080 which is a very common port to be used for services and as an attacker if I saw something like this coming back I would immediately have a really massive clue that it's running kubernetes and that might help me decide what I'm gonna issue next what kind of requests I'm gonna issue next I could start looking at the API and getting back enumerations of all the entry points that I could hit and maybe I could try to get a list of the pods in the default namespace and I can because this insecure port has been left open with this information I get to see exactly which container images are running and as a hacker that would be amazingly useful because I could look at what if I know exactly what images are running I might find that some of them have funner abilities I can look them up against some databases or vulnerabilities and perhaps exploit some if I find that a container is running that has known vulnerabilities so being able to access that information could be hugely valuable to an attacker alright so that was the insecure port you could you can and should be configuring your API server to not leave the insecure port open okay so what about the secure port which runs by default on six four four three and I could let's assume it three now as an attacker I don't actually care all of us come out I need to say HTTPS as an attacker I don't actually care whether the web service is identifying itself or not with a certificate but this error is telling us that my clients doesn't recognize the certificate from this service well that's fine I can just specify that I want the insecure version which says I don't care what certificate I get back okay okay so on the secure port this is the same cluster I get 403 I'm I don't have the authorization to make that particular request so when you make a request of the kubernetes api server if you don't identify yourself it will be associated with this service account called system anonymous so that's the kind of identity for anonymous requests that service account can be used in our back so you can give that service account permissions to do things should you so choose you probably don't want to do that you might also be wondering well should we should we be allowing these anonymous requests at all there is there is an option that allows you to turn off to to disallow any anonymous request but there is the thinking goes that if you're going to do things like health checks you don't necessarily want the client that's making the health check to have to identify itself to have to set up certificates just to be able to do things like health checks so by default anonymous requests are accepted but it relies on our back to stop you from doing anything kind of crazy alright so we could do things like get health information health and we get back and okay so as an attacker I would at least be able to tell with the default settings that the kubernetes cluster was up or not okay so what if I were able to identify myself as something other than the anonymous user what if I were able to get hold of the credentials to to make a request under a service account that has permission to do other things well every pod is running under a service account it might be the default service account or you can specify another service account identity and by default the token associated with that service account is accessible from your application code running inside that pod the reason for that is so that if your application needs to make requests of the kubernetes api it can have the credentials to do so so if I I'm going to run this on my other machine I've got a cube control get pots I have got a pod running and I can exact into that well and I'm going to run a a bash shell inside there now and I can get hold off if I can type I can get hold of a token from the well-known location boots again of the secret so it's in VAR run secrets q Vanessa's service account token so that is my token that's accessible to this particular pod and if I do the insecure version if I make a request of the kubernetes api without identifying myself I get a 403 forbidden message but if I do the same request actually let me do it again with trying to get some hot information [Music] namespaces default pods so I get a 403 for that request but if I pass in that taken or a authorization header authorization with the bearer token that particular service account happens to have permissions to access the API so if I were an attacker and I have been able to compromise that application if I'd been able to open a reverse shell I would have been able to access that token and hit the API and do whatever I wanted to do so yeah if your app is compromised your attacker is effectively able to sit inside your app and get hold of that token another way that you might be able to get hold of that token is my accessing the secret that it's associated with so if I go back to my node right if I get the secrets on this on this cluster there's a secret here called superpower token and I could describe that and I can set that is exactly the same token now as a user of the kubernetes api is a key control user I might have permission to read secrets and if I can read secrets I can get hold of that token and then I can get all the privileges associated with that service account so even if I it's the central way of escalating your privileges it'd be very very careful with your are back configuration and who is allowed to access secrets access to secrets could be effectively giving access to a broader range of auerbach than you really wanted to give so everything I've done so far was on the kubernetes api server there are several other api's offered by components within your kubernetes cluster which probably are not accessible they're probably not made available but you might want to check so for example SVD is running on your cluster and you might want to check whether or not you can access port 2 3 7 9 and if you make an API request like that you can you can tell as an attacker that CD is running and you could explore the HDD API and get all sorts of information about the running cluster maybe you could even modify the state of your community's cluster through HCD another API is the qiblah API so every node is offering up this Kuebler API this is another example where there's a secure and insecure version by the way and the secure version runs on port 10 to 5050 even if you get a 404 page not found as an attacker or a pen tester that does tell me that the port is open so maybe I want to explore that further and I might do something like I might try to get some metrics okay I don't have permission to do that what if I have a token if I was able to if I've been able to access a token from a compromised pod or through a secret I could to do exactly the same approach as I did before passing that token authorization token and again there's all sorts of interesting information accessible those the cubular api I could even get information about the running pods on that particular node so it's not just about the API server API there are all sorts of other API in your cluster you might want to be checking whether people can get information from them so I've run a whole load of curl requests obviously that would be a bit painful to do kind of manually if you wanted to do it repeatedly so we wrote keep Hunter to kind of automate all this kind of thing so let me run keep hunter against that goods that cluster 172 28 28 3 and that's running a whole load of tests very much along the same lines very much like issuing those curl requests and it's found a whole load of interesting things including well it tells me that it's found ports offering API services like the qiblah api at CD and a insecure and secure version of the api server and then attempts various things and reports back on them so for example it said yeah I've been able to list pods as an anonymous user ok you can also run keep hunter inside of a pod and then that's simulating what we did before where your step if you have a an application that's been compromised and you can get hold of the token well keep hunter will try to grab the token and attempt to make requests using that token in this pod mode the results you get from that will depend on the particular a service account that you're running the pod under so you might want to run keep hunter inside a pod under all of your different service accounts if you were going to be really comprehensive about it everything I've shown you so far is what we call a passive hunter so I'm just requesting information I've not done anything that could change the state of the cluster keep hunter also has some what we call them Hunter's that do you things that could potentially change the state of the cluster we don't have anything that doesn't nothing horrendously destructed like trying to delete all your pods or something but we do have some active hunters that will try to create things and then destroy them again so I could run this again with active mode 72 28 120 8.3 and this time we find a few additional things you know it has actually tried creating a stir roll and created a privilege pod and this would be something you want to do with caution as I say we don't we're not trying to do anything terribly malicious with these active tests but you should be aware that it could change the state of your cluster so having a quick look inside the code I just it's written in Python and it uses this quite nice observer pattern where I'm just sort of taking some extracts from the code system illustrate how this observer pattern works there's this it works on the basis that you subscribe to events and you can have classes that subscribe to these events and then can go on to perform more tests so there's an event for each IP address that we want to test called new host events and for each host the class would go through a set of ports we're looking at for the ports that we expect humanities services to be running on and for every port where it finds that the network connection is potentially open it publishes an event called open port event and then we can have a set of classes that subscribe to that event may be specifying particular conditions like what the port number is in this example cubelet discovery hunter that subscribes to open ports for ten twenty ten to fifty and ten to fifty five and it will do lots of things one of which is attempting to read the metrics from that connection and if that succeeds you'll say okay that's a read cubelet event which can then be subscribed to by another class which will attempt to get more information from that cubelet ports publishing event for example we've got if it was able to get hold of the cube Nettie's version that also publishes a vulnerability and that's what we end up seeing in that table of vulnerabilities and the reason why i'm showing you that code is because i'm hoping that some of you in this room might have creative ideas for additional hunters it's open source code right I don't think chief hunter is ever going to be some comprehensive that it would replace a full penetration test but there's certainly you know we have some ideas for some more hunters probably people in this room have even more creative ideas for more hunters and we very much welcome ideas implementation of hunters as well if you so choose so if you are interested in running or contributing to keep hunter you will of course find it on github I think we have a little bit of time for questions [Applause] hyah is it set up to use from CI oh so you were running it manually there but if we wanted to run it was part like a continuous delivery pipeline house opposite for that and I haven't tried doing that myself I I guess there's no reason why you couldn't yeah why not just yeah it's Python code just run it wherever you want to run it hi it was a nice talk and it's nice projects and do you plan to extend it with just to use it with some thread pattern databases or just like test four CVEs or something like that right or we should just join the project and develop it yeah I mean we don't have a particular boundary where we'd say we'd never have that particular type of testing in fact just this week we were talking about the possibility of if you get back that container image identity the the hash of the container image we as actor we have our own vulnerability database so we have that information so we were thinking about maybe potentially adding that was just that the idea idea stage at this point that yeah and also you know there are plenty of other types of attacks that are not currently cut you know it's all just restful api attacks at the moment - we haven't at this stage got anything that tries to attack the application code that could be running in your dignity's cluster there's all sorts of possibilities hi how often do you actually see this sort of insecure ports being opened with your customers I don't have a data on that it is definitely the case that there are people who have found particularly that older cluster that's been set up for some time there you know miss configurations are possible I would say that perhaps more common is our back settings that are a bit looser than they necessarily need to be yeah it's the sort of thing as well you know you saw how long it took to to run keep answer there's a few seconds and it might save your save your bacon so there was an effort to get better defaults for kubernetes api yeah is there any plan to test all these popular additional components like east your linker D that expose more api's and we don't know about the status of the defaults I think that would be a great great thing to do yeah I'm fully supportive of that idea any other questions all right well thank you again live [Applause] [Music] [Applause]

Channel: CNCF [Cloud Native Computing Foundation]

Views: 6,418

Keywords:

Id: fVqCAUJiIn0

Length: 30min 27sec (1827 seconds)

Published: Fri May 24 2019