Certifik8s: All You Need to Know About Certificates in Kubernetes [I] - Alexander Brand, Apprenda

so thanks everyone for coming super excited to be here my name is Alex brand I am going to be talking about certificates and how we use certificates in kubernetes and all you need to know about certificates in kubernetes the only caveat is that I thought I would have enough time to talk about everything certificates but half an hour is really not enough so that all there should really have a star and what we're gonna be talking about just a smaller scope we're specifically going to be talking about how to use certificates to produce a secure kubernetes cluster so just to give you a bit of background about myself and about aprender we've been working with kubernetes since early 2016 when we got involved with the community we actually created the sig windows so the special interest group that's leading the windows implementation of kubernetes so you can run Windows containers on Windows and then I personally have been involved with maintaining our open source project called the cosmetic enterprise toolkit which is a set of tools that allow you to manage Orchestra operate and bootstrap kubernetes clusters on Prem or on the cloud so our initial implementation of cos Matic was released in November 2016 when our back wasn't quite there yet so our security model we were secure from the from the get-go but our security model was very very simplistic so every single component had the same identity all the users were using the same the same users so after our back came out in April 2017 we actually decided to adopt it because we thought obviously it's a it's just a better model so we adopted that model and that meant we had to just revamp how we generated certificates in cosmetic so this talk today is going to be informed on those experiences and what we had to do to revamp the generation process and how to produce a secure cluster so this is the agenda as you can see there's a bunch of stuff in there hopefully we have enough time to go through it but basically you know we'll talk about a couple things in that you need in kubernetes and that are related to certificates to have and to run a secure cluster so I know we're in a security track but I just thought I would do just a quick certificates refresher just in case yeah so the certificates basically allow parties in a conversation to authenticate each other so if you think about a client and a server relationship the client can authenticate the server and just make sure that it's talking to who it thinks it's talking to and the server can also authenticate the client using certificates so this is all this all depends on the on so one way of doing this is using a third party that both the server and the client both trusts so if they don't know each other they don't have a previous relationship they can actually trust a third party that issue certificates to be able to trust each other so that is where the certificate authority comes into play so the certificate authority is actually going to issue certificates to all these parties and given that they trust the certificate authority they can then trust each other so if you think about when you access your bank's website for example you want to make sure that you're talking to you're actually talking to your bank servers and not some random server on the internet so what you do is you ask your bank for a certificate and then you validate that certificate and given that that certificate has been signed by an authority by a certificate authority that you trust you basically trust that you're talking to the the banks server so that takes us to kubernetes why do I need certificates in kubernetes why do I even care about certificates in kubernetes so how many of you are running a kubernetes cluster or in an Operations team very nice that's awesome to see how many of you are familiar with how certificates are used in kubernetes and exactly awesome very cool so as we all know kubernetes is a distributed system this is a very simplistic diagram of some of the interactions between all the core components in a cluster so we have the API server in the middle the API server is the only component that talks to XE D and then the API server has a bunch of clients including the scheduler the queue proxy the control the manager and the cubelet these are basically the main the core components of your cluster and given that all these components are running on different machines that are connected through a network you just want to make sure that all these interactions are secure and that all these interactions are basically all the components can authenticate each other so I'm gonna walk you through how we can build a secure cluster using certificate so the first thing we actually need is a cluster certificate authority so the cluster certificate authority is going to be the trusted root throughout the entire cluster so all the certificates that are used in the cluster are going to be signed by the cluster CA and this is what's going to enable all the components to be able to authenticate each other so all the components are going to trust that CA and then whenever they they're presented with a certificate they actually can trust it because they've been signed by by the CA ok so now that we have the cluster CA we can actually start securing some of our interactions so the first the most important interaction is the API server as we all know the API server is the main entry point to the entire cluster so we want to make sure that we're exposing this endpoint or the server over HTTPS and to do that we need a servant certificate and the corresponding key and as you as I mentioned before the certificate is going to be signed by the cluster CA which is going to allow all the components that are using or that are communicating with the API server to be able to authenticate it so there is an important gotcha here with with certificates for the master and that is that when you're setting up multiple API servers when you want to have an H a cluster you actually want to make sure that the load balancers IP address and DNS name is part of that certificate because otherwise whenever a client tries to talk to an API server through your load balancer the client is actually gonna try to validate that certificate and it's going to complain it's gonna say you know what the common name that's on the certificate is actually not what I'm trying to talk to so it's going to complain so just whenever you're setting up multiple masters just make sure that the certificates have the load balancers DNS name and IP address this is usually taken care of by your favorite installation tool they usually have a configuration parameter to just set extra extra extra names in the master search so there's actually another API which is really an internal API is just an internal private API as of now and that is the qubits API which you also want to expose over HTTP and then again to do this we need a servant certificate and the corresponding key and this API is mainly consumed by the API server when it needs to get logs or when it needs to get metrics from a specific container and also when it when it needs to exact into a pod so again this cert is signed by the cluster CA and that's how the API server can actually authenticate the cubelet and and make sure that it's talking to the the qubit similar to the API server the cubits API is also protected by authentication and authorization so this means that clients of this API also have to present client certificates so whenever the API server has to talk to a cubelet it actually needs a client certificate and that's why the API server actually has a client cert 4 we're talking to the cubelet so as you know as you'll notice I have little links at the bottom so if you want to look into it more there's some more ducks on it most of the slides having coolant so we talked about how to how a client can authenticate a server so the next piece is how can a server actually authenticate a client so that's where the client cert authentication strategy comes into play so the API server and the cubelet actually can authenticate clients using this strategy so this is mainly used by kubernetes components although you could use it for user off although there's some important caveats but the way this works is that any request that presents the client certificate that has been signed by the cluster CA is going to be considered authenticated part of that authentication process is to obtain the user information and the group membership of that user and that information is actually obtained from the certificate itself so the user is obtained from the common name field of the certificate and the groups are obtained from the the organization field of the certificate so as I mentioned before this authentication strategy is mainly used for authenticating kubernetes core components so each one of them is going to have its own client certificate because each one of them is going to have different access levels to the cluster so you want to have each one of these components to have their own identity and the way to do that is to have certificates with different common names so for example the scheduler has its own certificate with the common name set to system cube scheduler you'll notice that the cubed cert is actually special and the main difference there other than it belongs to an organization is that the host name where the cubit is running is actually part of the certificate so that is very important because you want to make sure that each cubelet on your cluster has its own identity why is that important so beef so that so the reason for this is that it allows you to enable a couple features in kubernetes which are called the node authorizer and the node restriction admission plugin so what these do is that they actually limit access to the API server from the cubelets perspective so without these features enabled the cubelet is actually capable of seeing and writing and modifying every single resource on the API so you ideally you don't want this you want to follow the principle of least privilege so instead what you want to do is by having each cubelet have its own identity you can enable this feature that will actually limit the access to the API to resources that are actually related to that node or to that cubelet so for example instead of the cubit being able to modify every single node resource in the cluster it can only modify its own resource in the cluster similar with secrets instead of being able to access every single secret in the cluster it can only access secrets that belong to pods that are bound to that node so we just went through all the certificates that we need and how do we actually produce these certificates so we can either manually create them so an admin can go ahead and produce all the certificates issue all the certificates that we need or there's actually an API in kubernetes that allows us to generate certificates so as of recently kubernetes offers an API that allows you to request certificates through just you know using the regular serger Brunetti api's so the way this works is that a client creates csr or a certificate signing request and shoots is all shoots it off to the API server once that csr is created it actually is going to remain in a pending state until someone manually approves it so the idea here is that you don't want to actually randomly approve and generate certificates because these certificates are going to be signed by the cluster CA so as we talked about before every single component in the Buster trusts that CA which means that is going to trust any certificate generated through this API so the CSR actually has to be approved for the certificate to be issued so once that CSR is approved the certificate is issued and the user can actually download it so I actually want to show you how this works so yeah let's dive into a quick demo and I have a kubernetes cluster running here on my machine it's a 1 8 4 cluster I think yeah 1 8 4 and the first thing I need to do is to generate a CSR so I've actually generated that previously just in in the interest of time and you can see here that I have my certificate request and what I have to do is to actually send this over to the API server so I actually have to wrap this in a CSR resource in the kubernetes api so the kubernetes api has a csr resource and you know similar to a pod or any other resource in kubernetes you add some metadata so I can call this CSR and set the name to cube Khan and then I have my spec and the most important piece here is the request field there and that's where the CSR is going to be included in this resource so that the API server can actually store it so I'm gonna go ahead and copy this yeah yeah I'll take questions in here and I can just post this to the API server and you'll notice here I'm using cat but it'll just read the CSR and base64 encoded and then just include it in the resource and then there we go so I've just created my CSR and I can actually get these this resource similar to how I can get any other resource and the API so I can go and do cube CT I'll get CSR and you'll notice that it's in a pending state the other thing to key to keep in mind and to and to learn from this is that there requester of the CSR is actually stored as part of the resource and this is important for what we'll talk about next which is how the Cuba can use this API through request certificates so I've created my CSR and again I have to approve the CSR so I'm wearing an admin hat right now and I can go and approve the CSR so the CSR has been approved and if I get the CSR again the condition has changed to approve and the certificate has been issued so as a user I can go ahead and download the CSR so if I actually describe or get this resource in yamo you'll notice that the certificate is now part of the resource so I can actually extract this the certificate using adjacent path so that's in the status field a certificate field yep I gotta get my coupon cert CSR and that is basics before encoded so I can actually base64 the code and there's my certificate so I can actually start using this certificate so you can imagine building workloads or building controllers that live on top of this API and that's one of the interesting things of kubernetes that you can actually use all these API is to build stuff on top so that's the quick demo of how this process works and now we can talk about a specific use of this API and that's how the cubelet can actually use this API to bootstrap itself and to rotate certificates when when they're it certs are nearing expiration so as we as we talked about before the cubelet needs a client cert to talk to the API server and it also needs a serving certificate for its own API so instead of having the admin having to generate all of each of these certificates one for each cubelet and all the other serving certs the qulet is actually capable of requesting certificates when it starts up so this is again this is built on top of the certificates API and also the bootstrap token Authenticator which I won't go too much into but it's basically a different Authenticator that that authenticates clients based on a short token and these tokens are basically used during the cluster bootstrapping process so this is how this process works so let's assume that we're bootstrapping a cluster and there's an API server and the controller manager it's all there all the way running so the first worker node is starting up and the qubit is registering with the API server and it's going to realize that it doesn't have a client cert so the first thing it's going to do is it's going to create that CSR resource that we just learned about and it's going to use the bootstrap token that is used for bootstrapping clusters the importance about the bootstrap token is that's is that it's going to tie the request back to a cubelet so as we saw before in the demo part of the csr resource is who requested the csr so in this scenario the the requester of the csr is going to be a cubelet which is going to be important in this flow so once that csr is created the API server is actually going to issue a watch event to one of the controllers that what that's watching the CSR's and there's a controller that's going to check whether the csr was requested by a cubelet given that in this case it was the it's going to automatically approve the csr so as we saw before CSRs have to be approved so there's a controller that automatically approve CSRs that come from cubelets and then there's another controller that is watching CSRs that once there's a csr that's been approved it's actually going to go and sign the csr to issue the certificate and it's going to go ahead and upload it to the api server once the certificate is up in the API the cubelet can actually go ahead and download that certificate so it can go and as we saw in the demo we can go and download the the certificate using the API and it's going to after after that it's going to be able to use it for further API access so just to summarize the most important steps the first thing is that the Keele it creates the csr using that bootstrap token then there's a special controller called the csr approving controller that is going to approve the csr automatically because the request came from a cubelet and then the csr signer controller is going to go ahead and sign the csr which will then allow the cubelet to download the certificate and start using it to talk to the api server so that's how the qubit gets the first certificate or that's how the cubelet bootstrap bootstraps its own certificate but what about rotation so you don't want to get paged at two o'clock in the morning if when your cluster dies because all your cubelets fail to talk to the API server because the certificate expired so instead of you know getting paged the as of kubernetes api as of kubernetes 1:8 the cubelets can actually request a new client certificate when the one they're using is nearing their expiration date so similarly the cubelet can also rotate the serving certificate and that's actually an alpha right now there's a couple of kinks to work out but yeah if you want to share if you want to follow the progress on that issue I like the issues down there so that's basically how we can create a secure kubernetes cluster deployment that's how certificates are used at the kubernetes component layer but as you might imagine certificates are also used for different things on the cluster once you have your your running cluster so one of those those things which is actually which is actually a missing as something that's missing is the ability to revoke certificates using certificate revocation lists so just something to keep in mind if you're using certs for user off you currently cannot revoke certificates so they're always going to be they're always going to be considered valid until they actually expire so if you want to learn more about this or if you want to join the discussion or if you need if you think you need this feature for whatever reason definitely check out this issue on on github so today we talked about certificates at the kubernetes component level so how the components themselves use certificates to authenticate each other but what about workloads so if you want to expose workloads over TLS you can actually do so using ingress so ingress has the ability to expose services that use TLS and the way to do this is you first have to define a secret that includes a certificate and a private key and then you have to reference that secret in your ingress resource so what's going to happen there is that once your ingress controller realizes that there's a new ingress resource defined it's going to go and use that secret to configure that TLS listener so if you if you don't want to do this manually there's actually an interesting project called cube Lego which can actually automatically generate all these certificates using the let's encrypt API so if you want to check that out that's pretty cool and then similarly at the workload layer there is actually a working group that is working towards allowing containers to prove their identity outside of the kubernetes clusters so inside the cluster each workload has a service account which they can use to talk to stuff that's running on the cluster but what about accessing external services so there's actually a working group called the kubernetes container identity working group which is working towards this idea of being able to use some mechanism that is going to allow containers running on the cluster to be able to access external systems that are running off the cluster the other potential use case although there's a bit of overlap with it something like Sto is potentially kubernetes could do service to service you know mutual TLS everything out of the box so if you want to learn more about this these are the notes of the working group and they have weekly meetings so I definitely encourage you to check those out so just to quickly summarize what we talked about today we went through how we can produce a secure kubernetes cluster and how certificates are used to create a secure cluster so certificates are key to the function of the functioning of a secure cluster and the main reason for this is that kubernetes again is a distributed system and you want to make sure that all the components are talking to each other in a secure manner and that's exactly where certificates come into play so certificates allow all of the components to authenticate each other and to establish mutual TLS and that is what's going to produce the the secure cluster we also talked about the API that kubernetes offers for generating and requesting certificates so we we saw a specific use case of this which is the cubelet and how the cubit is capable of requesting and generating and we and rotating those certificates and again this is an API that can be consumed by anything other other than just just a cute look so this is just a quick table that gives you an idea of all the certificates that are in use in a kubernetes cluster obviously at the component layer so I mean we talked about all the the API server the controller manager scheduler cubelet queue proxy and I hope this gives you an idea of how important certificates are and how careful you have to be with certificates so again you don't want to get paged at 2:00 a.m. in the morning when one of these certificates just expires and your entire cluster is down so I hope that gave you an idea again of how certificates work how they're used in kubernetes and if there's any questions I'm happy I'll be around and how did it take them Thanks no so that's a great question so the actual component that signs the certificates is the controller manager so the controller manager does need access to to the private key of the CA yes it's fairly sensitive so it really depends on how you're setting up your cluster so some installation tools run the controller manager as a workload on the cluster so you could use a secret for that or something like that yeah yeah yes yeah so the the cubed API yeah so the Cuba needs a client search to talk to the API server but then if you want to talk to the cubelets API you also need a client cert yeah so that's a great question so all the issues that I've seen around the cubelet API the cubed API is really an internal API right now it's undocumented so it's not really maintained or yeah an API that's going to remain stable so the recommendation is to not really depend on it but I think there are people that are that are using it yeah yeah yeah correct right so when you you're on your bootstrapping your see it when you're sending your first csr I think yeah so you're using a bootstrap talking at that point but I don't think it'd be encrypted that's a great question I'm not sure I can definitely look it up and come back to that yeah yep so the the majority of clusters that at least that I've worked with have a single CA so there is one CA so the CA is a single CA in the entire cluster the the yeah I guess so you're talking about signing the certificates when they're requested yeah in technically the controller manager but yes the the kubernetes api and the yeah exactly yeah make sense yeah yep the CA continues to be the same CA so there is one CA for the whole cluster yeah yeah yeah a way to automate the generation of keys the private keys so that really so for example in the community there's cube ATM that takes care of all of this so if you're thinking of creating a cluster cube ATM is a good way kiss Matic our tool also gives you that capability and it gives you a couple other things so we take care of that we take care of all of that for you so you're gonna have to do that so yeah yeah I think so I'm not super familiar with that right now but I believe there is an effort to integrate with external key management software I believe there is some integration with vault there's I there actually is you can actually use vault to store secrets today I'm not super familiar with that so I yeah oh there's the next talk there we go yeah yes Yeah right for the Masters so the master certs are actually not generated using the certificate API so you actually have to create them beforehand right yeah so either manually or the installation tool actually takes care of that for you so for example kiss Matic will take care of that or cube a the M will take care of that and then you also write yeah you set the Sun for the LV exactly yes sir that is a good question I don't know yeah I'm not sure yeah yeah yeah I would expect them to be after after they're created yeah I would expect them to be clean and clean up yep yes sir so it really depends so we we actually issue in cos Matic our CA is I believe two or three years and the certificates that are produced by that CA are one year no so as of right now the only one that gets rotated automatically is the the cubelet one yeah yeah yeah every whenever they they're close to their expiration they they the cubit is gonna rotate them yes sir yeah yep not necessarily so you can actually use your own CA you can provide your own CA to kubernetes and your installation tool so for example in cosmetic you can actually bring your own CA and we will use that CA to produce all the certificates that are that are needed for the cluster yes yeah so yeah there's some implications there definitely yeah - to upgrade the CA that is a great question I believe from what I've seen you can only use one CA so you would incur downtime there I think yeah I believe that should work yeah that sounds like a potential workaround there yep sorry so he was suggesting adding the CA to the machines trusted route list so that the the the binary just trusted because the Machine trusted so in the multiple master scenario each master is going to have its own certificate and then the the consideration there is to just make sure that the load balancers IP and DNS name are part of that search so that your clients you know can validate that certificate yeah yeah so you can bring your own CA I'm not sure if you're talking about like integrating with another system yeah I'm not sure I don't think so I think you actually have to provide like a file-based CA so yeah at this point you have to do that yeah cool thank you so much and enjoy

Channel: CNCF [Cloud Native Computing Foundation]

Views: 42,659

Keywords:

Id: gXz4cq3PKdg

Length: 35min 56sec (2156 seconds)

Published: Fri Dec 15 2017