Datacenter Orchestration Security and Insecurity: Assessing Kubernetes, Mesos, and Docker at Scale

so my name is Dino desi I'm the CTO and co-founder at capsule 8 wanted to make sure to start off by saying that this is a this is a group effort led by cap Brandon Edwards the lead of our research team including great work by Russia cats called Bergy and Kent mAh so this is a team effort even though I'm the only one up on stage I wanted to kind of tell you you know a little bit about Who I am and how I approach this my background is in security assessment and I've over my career I focused on different things I started focusing on client-side exploitation kind of in the early 2000s a little early went spend a lot of time on Apple technologies including OS X and iOS and over the last year and as you're in some change I returned to my roots in Linux and they can look at all the cool stuff that's going on with containers and orchestrated environments and spent a lot of the last several months looking at the security of these environments where they are today because we've talked to a lot of people and they are the adoption of these platforms is growing very very rapidly and we've a lot of us have been in security long enough to kind of know what happens when there is that sort of explosive growth so let's talk about a little bit about what this is so common misconception I hear from some people is well all this stuff is just about docker containers right it's like just that's all it is I don't see what the big deal is well not quite there's a there's a lot more to the story and the metaphor that I normally use is of the the modern assembly line and the era of mass production so when people focus overly on like just the container which is just the artifact that you send to production it's sort of like trying to understand the the moving assembly line by the like the conveyor system yes it's one it's an important part it's necessary but it's not sufficient to get all the benefits of that efficiency you need fixed stations you need repeatable operations you need the workload to move to the worker and so on and so when you're thinking about what the security of these modern environments need to look broader than just the container and kind of explain a little bit more about what that means first way to understand this is to understand what people are trying to achieve out of it and what they're trying to achieve is a element of scale and agility and velocity and way to measure that velocity is by the number of deploys to production as a single variable that you can use to measure your efficiency and that's like a call to fitness function and good anecdote about this I found on a medium post was a CTR a CEO of a well-known tech company asked the CTO I've been asked to spend 15 minutes in President Obama to provide recommendations on how he should fix healthcare.gov what would you say so how do you drill that down into one quick thing and the CTO told them suggest the President Obama that the team should deploy to production every day and kind of the genius of this statement is that it's a forcing function to do everything else that you need to do for high quality engineering at high velocity you cannot sustainably deploy a production every day without everything exploding unless you have all these other moving pieces and practices and those kind of force you to do things like have small agile teams independent containers you know various patterns of composition in order to get that continuous delivery of software and my thesis of what when people are talking about DevOps they're talking about containers they're talking about a lot of the some of the tools or techniques that are pieces of the picture what they really want is continuous delivery like that is the real technology that is the real enabler that any company adopting any of these technologies really wants and everything else should be in support of that and so what does it mean to have this continuous software delivery process you need to adopt several key practices building a repeatable automated reliable software for releasing or releasing software automate absolutely everything everything must be version controlled and you should adopt practices where if something is painful like previously deploying to production was a painful painstaking operation so what you do is you move it forward bring the pain forward to make it more efficient and so by deploying production every day you make it more seamless and build quality throughout the process and the only time you mean done is when it's actually released to a hundred percent of your users its rolled out and make sure that instead of having the silos of developments operations and security everyone is involved in a delivery process and I think as an industry we're trying to figure out what it means for security to fit into this new way of doing engineering so there's words we haven't decided whether it's called SEC ops dev SEC ops SEC def SEC OPSEC DevOps op SEC dev you know we're kind of trying every permutation to explain what it means for security to not be an independent function that is a gate keeper rather a piece of the process of a continuous delivery pipeline and focus on this continuous improvement these are the things that when you adopt these practices you can actually have continuous delivery and orchestration and containers are very key components of this so the container is just the software artifacts sent to production it's focusing too much on it is like you know looking at the desktop the PC revolution and personal computing and thinking it's all about like the comm file it's like yeah the basic executable is a piece of the puzzle but it's really a whole bunch of other things but you know all a container really is is it's a reusable composable and suitably abstracted from the environment piece of software and it allows you to just have a reproducible artifact that you can develop test and ship to production and keep it all in version control when you're using these at scale you need an orchestration it's just system to schedule their operation and schedule the dynamically scheduled into nodes and also automatically handle rollover you know rollouts failover scaling and this is actually one of the the key points about orchestration systems is that they are a ideal place to implement a lot of the security functionality that you need in your production environment because they are the software fabric around everything and so what we'll see in let his environments is pushing more and more security into that so that you don't have to influence security in every single application which is is tedious and also very painstaking so my security thesis when I started this research was what I how I approached a lot of things I think attackers are always looking for the new frontier to exploit while defenders are still fighting the last war and so how I saw this pattern manifest itself in this ecosystem or things like a an excessive amount of focus from security practitioners on OS level container escapes focusing on kernel vulnerabilities focusing on these things that kind of impedes looking a little broader and seeing just how exposed the rest of the cluster level api's are from inside those containers because just looking at those escapes is like just running a docker container on a single workstation and that's pretty much only in development not really where it matters in product as in production another you know corollary I had going into this was that when you only focus on the scope of an individual node like you're thinking about okay so how do I assess callate privileges how do i you know install a rootkit on particular node in the cluster you also miss what you can do at the entire cluster level because one of the patterns of these orchestrated environments is that nodes can come and go they're torn up torn down and a lot of companies that are using this type of new set of environments already they don't measure up time in terms of how long the systems go I've been up they don't see that as a good thing they want to measure reverse uptime and basically take that number down they want to have systems be short-lived ephemeral and have that pattern kind of a throughout every stack or through every layer of their stack and the third thing I saw was that overly focusing on immutability of the containers themselves and even the operating systems on the nodes misses a lot of weaknesses in where the mutability of the contain of the cluster level state allows all the types of persistence that an attacker would want so how a normal ego about things as I start with a an attack graph to kind of figure out what the choice is an attacker would make and why in different environments so we start with the assumption of we attacker obtains remote code execution in a container in a cluster in one of these environments so that I'll just use shell shock as high-profile example but one of the things I've noticed over the last few years is that the prevalence of remote command injection vulnerabilities in a lot of you know popular Linux software is pretty surprising and whereas unlike the Windows desktop world we're talking about you know vulnerabilities that involve insane levels of heat manipulation and arcane low-level techniques a lot of the attacks that are succeeding at a pretty impactful scale against production environments are just shell metacharacter injection in the right place so shell shock you just put it in a you know environment variable it gets passed to bash image tragic you have an image conversion routine that shells out other of only bit similar vulnerabilities and go script it's I think as people start looking more at this I'm gonna see a lot of more as a problem and so especially as we are rapidly taking less mature software putting it in you know containers exposing it to the Internet and moving it through that cycle so this is my starting point another traditional route of the official strategy is starting at you know I have no you're looking for approaches its privilege escalation on the node get out of the container get route and then see if I can persist on the cluster however that's not as useful as looking at other things where we can say alright well look we instead of that look for lateral movement opportunities within the cluster directly and from there we can look for privilege escalation within the you know at the cluster level and then cluster level persistence and this is possible from you know just direct container remote code execution privileged access to the node or somewhere that you've moved laterally to and it sort of shows that that's the area that I think is the most important to focus on is from different levels of access how can you escalate cluster level privileges so let's talk a few of the various alternatives for registration systems I'll start with docker swarm the so the service model for docker swarm is oh I would call it the the least complex of the available leading orchestration systems it is largely docker composed as a platform and how it works is you have a service that is defined of of a container one or more containers and swarm will schedule and scale it across available nodes as tasks they will farm it out and manage a configured number of replicas so that if one goes down another one is automatically restarted or you can also auto scale the number of replicas as well how this is implemented is through a docker swarm or swarm mode of docker and there's two two types of nodes in this configuration there are managers and there are workers much like our daily lives and the managers are all in a a raft consensus group so there's a distributed key-value store essentially this is a pattern you'll see on all these systems a distributed key-value store and kind of the masters and the managers of the the orchestration use that to maintain a quorum and the actual cluster state among themselves and so you'll always want a odd number of a manager nodes regardless of whether using graft or whether you're using Paxos but this makes sure that the the cluster itself is consistent even if there's network issues the all the the workers communicate over a gossip network so you can primarily to manage the overlay virtual network across them and the one thing that you'll also notice in kind of it's slightly different than traditional virtual machine based network architectures where there's a very strict separation between control plane and data plane a lot of the orchestrators today combine them on the same network fabric and they'll have overlay networks for the data plane but they'll go over the same links and the to use TLS and various other encrypted protocols to protect the control plane traffic so swarm actually does a really good job at this they spent a lot of time on the security model and the security features and there's sort of the gold standard of orchestration security today and how they how they work is the manager is a x.509 CA the managers form x.509 CA for the entire swarm and they use this to implement mutual authentication between all the control components between nodes between managers as well and the data plane traffic by default is unencrypted however you can optionally encrypt it and IPSec tunnels will be created between all the service dependencies of your workload so container will only have access to the links that you have declared so an app container will only have access to a database that you've declared not access to over the network to talk to the manager is there anything else and another key artifact aspect of how swarm model is implemented is that it is at least privileged design the the containers themselves are always considered by default untrusted the nodes themselves that they run upon are by default considered untrusted and they are only given a middle amount of information to do their job so it's a kind of a it's a pretty good model in addition to this this interacts most concretely with secrets where secrets are taught you know what they sound like objects that are defined to likely passwords to whatever an apple need but they're they're encrypted at rest throughout all of all docker swarm and they're only sent to nodes that have a job that needs them rather than making them accessible to the entire cluster so let you know if you look at what does what would something like shellshock being exploited look like on docker swarm it's pretty uninteresting because you get a shell in the container the attack surface is essentially all you get from a highly constrained container on a single system and there's not light can do there's no API you can talk to there's no there's just not a lot of avenues for lateral movement or anything like that so the attacks that you lot want to do are largely scanning for scanning the data plane for other accessible services or but that you have access to and try to SK privileges through there or traditional operating system level container escapes and there's sort of a lot of other research being done there so that was something that while it's important and necessary not the focus of this talk but one aspect that I did want to highlight about how swarm works is in order to make it easy to use and allow nodes especially in an environment like a cloud environment where you want to scale up and scale down seamlessly they support joining the swarm with a single cryptographic token so this join token automatically authorizes the node to join the swarm and doesn't require an explicit approval or confirmation for each new node and the tokens themselves are not consumed they're easily easy to rotate they're easy to change and disabled and they're only needed when this in the node joins the swarm never after the fact but there have a very regular format so they're easy to find in source code which is a plus and a minus and easy to find in memory as well so if an attacker is able to escalate privileges outside of the container and gain you know gain route on the underlying host there's actually stuff they can do against swarm and so the kind of a lot of the attacks go back to kind of 1990s style hacking with ARP spoofing and denial of service at the network level and what you can do is disable the network connectivity to legitimate worker nodes and a height boosts essentially hijack their IP address however should note that you don't actually get their cryptographic ident it just looks like a new node to swarm it looks like a new node at the same IP address because the cryptographic identities assigned you know through a certificate that is granted through a CSR when you present the joint token so by reusing the joint token you're just getting a new CSR while blocking the inter before using the same IP address you're blocking the legitimate node and what this lets you do is you can proceed to by blocking on legitimate nodes and taking up all the kind of the subnet you will be assigned all available services all the apps running on the on the cluster because the the orchid are swarm will notice that they're no longer reachable that need to be rescheduled and so they'll schedule you them all on your on your rogue node so let's see what that looks like make sure this works so we have you know we're showing both systems right now we're on the manager showing another layout of the of the environment we have two workers and one manager a few database services running on the on this form and we can see that they're they're all scheduled to run and they're running right now so now we're switching over to work or to assuming that we have it you know escalated privileges broken out of the container and what we're gonna do is try and find this join token and the easiest way to do it is if you have root access already is scrape it out of memory from the docker process it's regular string it's easy to find and it is not scrubbed from memory currently so if the system is still up after having joined the swarm you can recover it pretty easily and our attack uses that join token and talks to the manager and implements this where we are proof the exhibit nodes to the manager and so that they can't be assigned new jobs we can take all the available identities and get all the services reassigned into our rogue nodes and once that's done we will get the service secrets for those services sent to us and so we can get all the database credentials we can get all the you know whatever other secrets are are included in the you know for those jobs so I don't just see that again there are several ways to mitigate this first is the I mentioned earlier the join tokens are incredibly easy to rotate so they should be rotated frequently doctor recommends every six months or on events such as disclosing it through checking into version control pretty easy to run like git hooks that can actually detect this or you can use the github API to find that through your repos so you can write a job that will search for this and anytime that happens boom just just expire the token or if you have other reason to believe that a swarm node may have been compromised if you wish or you can also rotate it if you want to guarantee that no new nodes can join the system however when you do this you're preventing this swarm from auto-scaling up new nodes and if you're using like an auto scaling group in something like AWS this might not be desirable or minimum at every six months one of the kind of most important things to do however to mitigate this are to use node and service labels to segregates sensitive services so swarm lets you designate certain services is sensitive or just using labels and nose is sensitive as well and this is all kept in the manager State and even a rogue node cannot give itself a label so when we perform an attack like this we will be essentially an unlabeled node and so we'll only get access to unlabeled services so this requires that you go through and actually you know do all that so that's that's swarm let's talk about kubernetes so kubernetes is one of the largest most active open source projects right now it's pretty startling how active it isn't how fast it's growing this chart I took from a blog post from the cloud native computing foundation shows on the on the Left logarithmic PRS and issues and going to the right logarithmic because our going up is PRS going to the right is a number of commits so you can see that even though kubernetes is a relatively young project has a massive amount of activity in you know PRS and issues and it's moving quickly started moving up moving quickly to the right towards Linux and chromium so it's there's a lot of activity and it's actually pretty remarkable how many large companies have full-time engineers contributing to this this project and you can kind of see where all this is going so while swarm was easy to understand kubernetes is a bit more complicated and the complexity of it is actually a lot of the benefits because for a lot of modern services the model of the service model that is implemented in docker swarm may not be sufficient so the architecture let's go quickly over the architecture of kubernetes it starts with you know you have a again like a central key distributed key-value store in kubernetes this is called Etsy D and this is simply the backing store for all state for the cluster in front of that you have API servers that implement this you know an interface to the scheduling and and operations of the cluster and each node runs an entity called the couplet couplet is you know it presents its own API locally but it also or that the kubernetes master will speak to and is it responsible for scheduling and running actual containers on the node and in kubernetes these are called pods you don't run a single container you run a pod that has you know the kind of forms the Nate the Linux namespace and multiple containers run concurrently in it and kubernetes is a very abstract architecture there's multiple plugins to implement various features so you have a declarative API saying I have in this network policy that I want to happen and there's multiple plugins that you can use to implement that so you'll have things like we've calico and so on to implement these policies that you've declared I'll go briefly over some of the objects that's going to show the scope of what kubernetes actually lets permits you to do so we have nodes and containers pretty obvious I mentioned pods but we also have entities like a namespace we have multiple namespaces so that you can partition out your cluster to different groups and different teams using the same hardware and they're not able to and if you configure things correctly not able to see or interact with entities outside of their namespace so what some people or some environments would use something like this for is to have their production their development you know test staging and production environments on the same hardware however in separate namespaces so you can ratite it can achieve a lower cost and greater scale doing that and just using namespaces to isolate rather than separate hardware and you can also apply the resource constraints to a namespace so you can make sure that jobs that the data science team is doing that might be CPU heavy are not interfering with the latency-sensitive production jobs serving you know for internet services pods all communicate internally through what is called a service service is a virtual IP that allows communication between pods so that's how you'll you'll find each other and and communicate replication controllers are just one of the many other Metapod abstractions that let you scale services up and down and ingress is kind of my favorite which lets you route HTTP requests based on based on path and and host to arbitrary services and pods so you can have you know one hostname that gets basically farmed out to various things inside your cluster this is relatively new in kubernetes but it gives you a lot of functionality that is really useful so the the security mechanisms of kubernetes are being very rapidly developed and it's just about every most of things i'm going to talk about are you know less than are either alpha or beta or less than a month old so the first most important one that if you don't have your entire cluster is essentially one security domain regardless of the user permissions inside a container if you follow like the docker hardening guidelines they don't really matter when you have no are back enabled on a kubernetes cluster only incriminate is one point six in greater is it enabled by default however a lot of installations and guides don't work with it enabled so you if you're trying to install packages from things like helm they may not work with our back so a lot of people are still turning it off another key component that you want to make sure you have enabled is the pod security policy admission controller so this is the entity that runs in the on the masternodes that will enforce security requirements like not letting anyone execute a privileged note you know in one of these you can declare in Cleburne eighties is i want to run a privileged pod i want to run this pod but i want to give it you know full host privileges what i wanted to give it access to the host network all these things are disabled by the pod security policy and this is something that was contributed to kubernetes effort developed first in red hat's OpenShift and then contribute upstream to open source kubernetes and one thing i wanted to clarify is that most of something talking about is all open source kubernetes and there are commercial distributions it's like you know like red hats openshift core OS tectonic google container engine but there's a lot of activity happening and open so any open source distributions and installers that are describing them as production grade production ready that aren't as explicit about how many of the security features aren't enabled or in existence that you would want if you don't want your entire closure to be one security domain so the other one is the node restriction and mission controller this is a very very new this I think it's alpha right now prevents couplets from obtaining resources for pods that are not scheduled onto them similar to Dockers least privilege architecture and there's also Network policy resources which allows you to apply isolation segmentation so that your workloads can be segregated however one thing that I would point out is that this doesn't actually fully let you control segregation from the control plane so a lot of the couplet the master API are not actually restricted by these so the let's go into the attack surface if you get code execution and equipment and a pod in a kubernetes environment the most surprising thing that I noticed and I was so pretty surprised by every pod by default has access to a service account token mounted into the pod so even if you you know don't build this into your container its injected by the orchestrator and this token gives you your and you know even unprivileged users inside the inside the container of cluster level privileges and without our back that's cluster level root token is word readable and the the service account lets you do just about anything like launch new pods launching privileged pods and you can do all this the other model that is kind of unique to kubernetes is that the service model is implemented like the service virtual IP ease and load balancing are implemented on the node not in some central infrastructure so it makes it really fast to scan service VIPs because they're just going through like local IP tables configurations so if you want to do that you know it's pretty easy to find what the service subnet is you'll have an environment variable kubernetes service host that is a service IP that gives you the the host the IP address of the kubernetes master api and so you can just port scan this the subnet for well-known ports look for services to actually interact with another way you can do this is just looking through service names in Kubb dns Kubb dns is the clustered dns service that allows service discovery so you can query for common service name namespace SVC cluster dot local so for instance if you didn't have the IP address for the that environment variable you could just do a DNS lookup through kubernetes not default default is the default namespace SVC cluster a local to get the kubernetes api servers so let's you know see what shellshock looks like if you exploited on kubernetes so exploits shellshock in a vulnerable container next thing an attacker will do is download some tools so one thing they also might be going through your mind is how much this sounds like the common pattern of attacking a Windows desktop you know a lot of you get initial access and then download something like me me cats and then lateral pros escalation these same patterns are actually available at the cluster level so we're gonna kind of go through that same same flow so download a common tool not-not-not a piece of malware just coop CTL the standard tool to interface with the API and use the service account token to authenticate to the API servers and you can adjust escalate privileges through calls the kubernetes api or scan for something useful so i'm picking on one particular service here the tiller deployment service i'll talk a little bit more about what tiller is in a minute so let's look at the environment that you get from a shell inside the container that's specific to kubernetes you have all these environment variables giving you the API server which I mentioned before if you look at the mount points you can see that the secrets are typically mounted as in to the filesystem and by default that service account token is is mounted into the container and using that token with a you know with a downloaded coop CTL or some other client we can start interfacing with the API we can talk to you know list the nodes and Val verify that we have some level of access and then we can see if we have more access by launching a privileged pot so the example I give is just a command-line invocation that will create a new pod and of running the busybox container why busybox because it's it's small it's quick to download and has you know a lot of useful shell tools and I'm giving it a security context of privileged equals true so just means we have root on the node from this one call to the API and if you don't have our back enabled on your cluster or you're running essentially running kubernetes below 1.6 this is what is possible from just about any container so assuming that that is fixed you know that you're running either a commercial distribution of kubernetes that has addressed this you've adjusted yourself for something else there's other a lot of other lack of authentication inside the cluster so one popular way to install software in kubernetes environments today is called helm it is the official package manager for kubernetes and it's actually really cool because what it lets you do is have essentially an app store for your cluster so you can install complicated infrastructure such as you know spark you can install you know Kafka you can install Cassandra application you know datacenter applications that traditionally require entire teams to operate can now be installed with like a click of a button and there's software operators inside communities that will even scale them automatically for you so it's it's pretty awesome but they just haven't implemented anything any access control yet or it's just not enable it's I think it's yeah there's an open github ticket to actually enable it so if you want if you're have a shell on a cluster you can just look for the what's called the tiller deploy service it's the way that helm is built is you have the helm command line and in order to start using it you need to deploy a privilege service inside the inside your cluster to actually perform the package installations that's called tiller so if you do a coop DNS lookup for tiller deploy coop that Kubb - system so coop - system is the system namespace the default port for it is four four one three four and you can connect to it and so there's not a good command-line client for it so I just wrote a quick one it's a G RPC interface all the code is on github so it's just very easy to to write stuff to interact with if you know how to write G RPC clients and then you can do everything the helm command line can do from inside the cluster you can install packages query them uninstall them so because it runs a privilege and all the packages that you're installing have to have privileges even if you have something like a pod admission controller or pod security policy you're like they're likely going to have have to enable tiller to be able to bypass that so you can just talk to tiller instead the kind of drilling down some of my key recommendations for securing kubernetes environments first and foremost make sure that our back is enabled and configured appropriately that's sort of one of those loaded statements of like just turn on security there's actually a lot that goes into this because there's still a bunch of sharp edges in the security policies of kubernetes objects so for instance if you have a writable if you have access to write into a namespace and even if you have a pod security controller that doesn't allow you to create privileged pods but a privileged pod is already running you can create services too early have a pod that's running on the host network you can create services to route onto the host network and once you have you know a host network access privileges escalation is pretty easy through a variety of ways so right now am I kind of if you know even with our back enabled having right access to a namespace with any privileged pods is pretty quiet game over but it's it's you know one step away you absolutely need to have a pod security policy configured if you are if you do if you are letting any jobs like automatic or continuous deployment jobs at the plate of the cluster or think of developers access to deploy to the cluster applauded security policy is crucial also something that I was surprised to see disabled more than I'd expect was the couplet and who knows masters API is not having TLS certificate authentication enabled and that just basically gives you route to the cluster and even some open source kind of installers for kubernetes they would not enable this on at CD so if they enabled certificate authentication on the API servers however you can connect to the sed key value store which if you can connect and there's no space Toft enabled you just have full access to read and write the state directly and that's good privileges that way so that's pretty simple stuff they need to make sure it's all there also the node Oh threatened the node authorizer which is beta right now just released to google container engine I think this month can restrict couplets access to resources that are not scheduled or not belonging to a pod schedule to them and also there are multiple network provider policy Network policy providers such as you know see Liam calico that you can use to implement the network policy API so that you can apply access control list to services so that not every service is globally accessible through the cluster this is you know if you're noticing that there's a this slide is a lot of work to implement in your environment that's sort of where open-source criminais T's is today it's it's sort of it feels a little like the early days of Linux but if it didn't if everyone was root you know on your system for the first several years and they're like gabble will implement that security model a little later so but there's it exists it just takes a lot of bootstrapping or using a commercial version of kubernetes that actually has it all done for you so let's talk about the a call it the quiet giant this is d cos and mezzos it's the environment that you don't really hear much about but it's the one that can the only one that can scale to hundreds of thousands of nodes so it's in some very large very large environments it's a similar pattern to the other systems we have a distributed key-value store this one is called this one is a zookeeper which is used for Kafka and a variety of other internet scale services and kind of it's a bit more complicated than kubernetes in that you have two levels of scheduling implemented by well first you have mais dos which is a two-level scheduler and then you have what are called frameworks on top of it that can schedule cluster resources to some other some other way if you're running like a large Cassandra cluster you might be using directs Bezos but a lot of people use marathon to schedule containers and there's a massive suite of stuff they also have an app store like thing called universe but one thing that is pretty explicit on the mesosphere website is that batteries are not included the security features exist only in the commercial DC OS and there's by default containers run on the host plane those are the host Network so any containers schedule has full access to the host you can connect to a lot of services right there and I've seen a lot of configurations where zookeeper has no authentication enabled so it's send a look at what shellshock looks like there the container environment you can get your host the host IP is pretty easily of the you know the control plane and you can actually talk to it you know it directly and dump the state and get you know the authentication settings and in this example you can see that there's no authentication actually enabled or required so that means we can launch jobs on the node without authentication we can also connect to the master and and do the same but sort of the more most fun one is just talking directly to zookeeper so you can inside the cluster all the zookeeper instances or you can look up through mezzos DNS at Z k1 through z k5 and usually the zookeeper client is a large Java package but I just wrote one and go and you can just talk directly to it and crawl through it and get things like the the auth token secret which is used for all the JWT tokens and then from there you can just mint your own authorization tokens and do whatever you want forever so it's cool so tools that I made to do this I didn't really currently think of a good name so going with crispy tribble for now it is a container image for assessing orchestrated environments you just launched it it has a built in go ssh server so you can just ssh in see that kind of directing that kind of raw environment and has a variety of tools for inspecting what is going on inside the cluster so you have coop CTL at CD CTL to command small command-line tools they wrote called just one called tiller and one called zk to talk to these services and do with it are basically just perform operations on them and when there's no authentication required and background on the name if you just create a project on github it says don't have any not feeling inspired try this one and it said crispy Tribble I was like alright yeah that'll work so kind of high-level takeaways I want people to kind of take you know remember from this is that sadly the open-source distributions of commodities and and mezzos and D cos don't have all the security features enabled they don't have them implemented in some cases and so if you're not prepared to go through a lot of work and a lot of learning curve like I spent probably two to three months learning kubernetes this year because I actually ran our kubernetes cluster for the first part of the year there's a lot you're gonna have to get working to have reasonable level of security and that just means not having you know root root for it within every container on the entire cluster and but they're moving pretty quickly to you have all these features you know imagine API authentication control plane and data plane separation is actually a little farther out actual workload isolation s2s authentication and authorization and a zero trust networking model so that even if the you know control plant or data plane physical networks are compromised it's not an issue and also a least privileged orchestration model and they're moving pretty quickly to this but it's Walt may be like while a lot of the open source systems will say the production ready production grade make sure all these are are enabled in it otherwise you're better off with the commercial version so far but one of the things they'll be interesting for us as a security industry to watch is that you know while we're what what I'm seeing in kind of the desktop ecosystem is a lot more solidification of an restriction of what's possible on devices as everything becomes like a mobile device its firmware you can install apps that's all you can do in order for people to even just move and implement functionality has to go somewhere and so I think what's going to happen is we're gonna see all that flexibility go on to the cost into people's data center is because environments like kubernetes and like App Store like installation of complicated infrastructure will remove the barriers to entry and we're gonna see a Android ecosystem like scenario in data centers because of this and it'll be it'll be interesting and one of the things that I'm actually seeing kind of fascinated by is that the growth of the cloud is actually massively propelling the use of Linux because it is the native operating system kernel of the cloud and it's pretty pretty cool to see that so that's that's all I have thank you everyone for your attention I have two minutes three minutes for questions

Channel: Black Hat

Views: 3,894

Rating: 4.8518519 out of 5

Keywords: Black Hat 2017, Black Hat USA, InfoSec, Information Security, Black Hat, BHUSA, Black Hat USA 2017, Platform Security, Enterprise Security, privilege escalation, lateral movement, datacenter orchestration systems, Dino Dai Zovi

Id: lXggHTqznOI

Length: 44min 53sec (2693 seconds)

Published: Fri Mar 02 2018