Cert-Manager Beyond Ingress – Exploring the Variety of Use Cases - Matthew Bates, Jetstack

hi everybody thanks for joining my talk it's great to be able to talk to you about the cert manager project this year it's virtual but hopefully in the not too distant future we'll all be able to meet up again soon and meet in person i'm going to talk to you today about all the various ways in which you can use set manager beyond ingress i'm matt bates i'm from jet stack with the original creators of the project and together with now 280 contributors in the community we've got set manager to where it is today sub manager is effectively a kubernetes add-on or extension but it extends the kubernetes api and adds support for certificates and certificate authorities it means that you can manage x509 tls certificates that's the complete life cycle the issuance and renewal and use those certificates in your applications it has wide adoption and lots of contribution from the community and as of last year we donated the project to the cncf so it's now part of the sandbox it's got integration with a variety of different public and private pki providers both in the core of the project but also a so-called external issuers and we're going to look at that in the talk today briefly how does certain manager work as i said search manager is an extension or an add-on it provides a number of additional resources so called custom resources crds and provides them to the key on top of the kubernetes api these represent certificates and certificate authorities we can refer to that as an issuer this provides first-class support for those in the kubernetes api and you get all of the advantages of being able to manage those resources declaratively much like you manage pods and deployments set manager has a controller that manages the lifecycle of those resources and importantly is able to provide the automation of those certificates automating the fetching of those certificates and importantly the renewal of those as well is most commonly used to secure ingresses and you can see here it's really quick and easy to add a tls certificate to an ingress it's important by adding in this case the annotation which references where you want the certificate to be obtained from so in this case it references the let's encrypt staging issuer that i have in my cluster and it also requires you to specify the secret name there as well so the secret will store the certificates obtained from let's encrypt in my ingress so really really simple how does this work under the hood well set manager watches for ingresses using its ingress shim component as part of the cert manager controller and is then able to create a certificate which is backed by a point in time certificate request which encodes the certificate signing request which it then submits to your ca of choice so in my particular case it was the let's encrypt staging endpoint and that's the point in which cert manager is able to automate the acme flow there's additional resources that you can see there the order and the challenge but effectively all of that is automated for you and it results in a signed certificate from the certificate authority which is placed into a secret and then consumed by your application in the case of an ingress of course that's most likely to be an ingress controller something like nginx for instance and it's then able to serve your application securely you can also use these resources more directly so the ingress shim automatically provides that automation just based on some annotations that you add to the resource but you can indeed tap into the resources that certain manager provides that is that of the certificate and the certificate request so the certificate effectively is a more human readable like representation of the certificate that's the one that you can see and use and use the api for indeed use the shell this difficult request is like a point in time request that's made for an x509 certificate from an issuer and this is typically consumed by as i said here by managed by controllers or other systems you see the certain controller there is responsible for reconciliating those certificate and certificate request resources but of course there are lots of other certificates to managing kubernetes it's not just the ingress certificates there are many stuff goods across a cluster typically that you may wish to automate using cert manager and i've just highlighted some of these and we're going to talk about a number of these different use cases in this talk ingress as i mentioned pod to pod which might be without we have or indeed without a surface mesh and we're going to look at both in this talk today you may also wish to secure a cluster and the control plane and the data plane there are also a number of web hooks that you may have that are used for admission you wish to secure we'll look at how serve manager can be used in almost all of these cases we often get asked in the community how you can use cert manager in order to uh secure you know put to pod communications effect effectively that kind of east-west traffic uh in in a cluster without a service mesh um at this stage and this is particularly useful if actually what you've got is a really small handful of micro services um and you know you can you can really just use you need certificates you know you need those to be automated and you want to be able to consume those in your application and this you know would be preferable to the operational complexity of having um that kind of full uh mesh so here's an example of how actually how you can use a certain manager certificate resource um directly basically you can go and create this in source control of course and apply it you know directly via ci cd um to your cluster and cert manager will automate the issuance of the certificate and put it into a secret that you can then consume in in an application in a pod and so a couple of uh things to point out here um making this uh this particular example is going to be very short-lived i'm going to put it into a secret um that i can specify much like we did with the ingress and you can then consume that that secret in my in my application you know it really is a set of files in a volume mount and that i can then sort of put into my application and start serving tls and some dns sans here and i'm also making reference to mice yet this is going to be like a private ca so this would obviously not be a public ca if you just really want something local to the cluster and we're going to see how this um how you can sort of set this up in some slides to come here's a bit more yammer that shows how you can effectively take the secret and then make it available in a volume and a volume mount and the files would have will be available to this ping service or this ping pong service at the mount path etsy ssl privates there you go there's the the key pairs and the ca certificate as well there's a really really simple way of being able to use the underlying if you like resources and certain manager programmatically we've got a number of end users that do exactly this at scale you may also want to have certificates obtained from a private ca so rather than just a public ca such as let's encrypt if you're trying to get certificates for that use case where you're trying to secure pods that are communicating within the cluster you may wish to use a private ca for that purpose and there's actually two issues that that are built into set manager that are really useful and convenient for this purpose they're the self-signed issuer or so-called self-signed issuer and the ca issuer and you can see here what i've taken the number of snippets of yaml will show you how you can combine these resources in order to create what effectively is a self-signed ca there on the right-hand side you can see there is a certificate called myca i've used the sca flag to denote that this is going to be aca certificate it's got a 90-day duration specified the secret the common names and subjects the number of other properties that you can specify and an issue a ref in this case i'm referring to that sort of self-signed key that i generated there on the left hand side this is just sort of an out the box relatively simple if you like private ca that you can configure using search manager and it's a little known people don't know that you can do this you can also of course plug in and if you like a more robust and more secure and private pki if you wish and so we've got a number of options here both built into the core of the project but and also those that are external to the project as well so called external issuers so you can use vol you can use small step they've got acme an acme ca server um you can also use the cloud providers so we've got support for google's certificate authority service and there's also an external issuer for aws is pca and if you're in a more of an enterprise environment you can also integrate it with venify and there's an issuer for tpp so in that previous example we were looking at how you could use the certificate resource insert manager in order to obtain a serving certificate but how about if you wanted to use cert manager to do interpod pod to pod mutual tls and you know obtained by a by serving certificate but also a client certificate so i've got an example here of a really really simple example of a ping service and a pong service and um we open source this lab if you want to go and find it for you follow the link there on the slide and it sort of hooks up uh you know this ping pong service um simple go binary that you can also use in some of your your own examples in this example we have is the two services i said ping context pong verifies itself for clients uh um pong replies that's secured using the servicer um and yeah it actually replies back to the browser so you can see the the contents of the pong certificate and importantly here both ping pong they're only going to trust certificates that are signed by the same ca so how does this look um how does this look um in in yaml well we have uh as before we have the we use this certificate resource and we've now got um ones you've got basically two stiff kits got both the ping and the pong and you can see here um in the certificate resource we're able to specify the the key usages um that get passed down you know when creating the certificate and so you know you can sort of use here both client or sirthroth um and set those appropriately and both of the certificates here both for both ping and pong use the same issuer so they're pointing at that private ca that i created um on the previous slide or show on the previous slide so using the certificate resource in this way um you can you configure it to create certificates um in order to do things like for instance um your authentication to a database and so we've got this blog post and the one of our team put together uh in which we used we talked through how you can use cert manager to create certificates for client authentication with mysql and there's been lots of examples we've also seen the community uh of where users have used it exactly um for this type of purpose rather than manually having to create and manage those certificate resources for applications in the set manager project we have developed a csi driver in order to make it um even more seamless very neat thing about the csi driver is that it means the private keys can remain node local it's rather than being put into a kubernetes secret as is the case with some of the integrations that we've already seen in this particular case you can keep the private key to the node where it's generated by the csi driver it means you can provide unique key and certificates for each of your applications so if you're using a replica as part of a deployment you've got many pods each of those pods can have its own unique identity and that identity that x509 identity can be obtained at the point of application runtime it also means that these certificates are renewed much like the other use cases that manager will know when to renew the certificate and provide a means to do that for you so this just makes it really super simple in order to get those certificates to each of your applications how does the csi driver work with the csi driver resides on each of the nodes in the cluster it's deployed as a daemon set and just stepping through how it works first of all the pod is scheduled to a node the keyblade of course is responsible for coordinating the the runtime it works with the node driver this the csi driver by calling the node publish volume at that point the driver kicks in and it generates a private key and it generates a certificate request that's a certain manager certificate request which it effectively encodes a csr that's that certificate request is submitted to the api server on your behalf the cr is then reconciled and a certificate is obtained from your kind of ca of choice and that certificate is then together the private key is written to the node's file system and then that's bind mounted into each each of the pods obviously in teaching the relevant pods the driver will keep track of all of the certificates and they will also be responsible for it for news and if at any time or if and and when of course um the pod is terminated and then it will also send out a call to the driver the cubelet will send out a call to the driver in order to make sure that the certificate and the key are deleted so this is a really seamless um automated way of getting those identities you can also use cert manager to secure cuban these web hooks web hooks are used for a sort of dynamic admission control used for a variety of purpose for for instance mutating resources or validating resources um as they are admitted to the api server and typically use for instance for applying default values or enforcing policy using you know the likes of opa coverno and also for ensuring that resources that are submitted are semantically valid and in fact certain manager uses this um itself you'll notice there's a component called the search merger webhook that does exactly this now if you're doing using this as a as an extension point you want to be sure that when you are submitting these resources from the api server to those web hooks that it's secure and then also you can be sure of uh the uh the destination to make sure that there's nothing that various is happening um with it being mutated or or validated in in a way that you would not expect so you can use certain manager in order to secure those end points and secure those provide the means provide the certificates for those endpoints and there's a couple of annotations or few annotations should i say and where you can do this one is inject ca from so my dry o inject ca from you can reference a certificate and you can also inject it from a secret and you can also inject the api server ca certificate as well this uses a component insert merge called the ca injector responsible for watching in on these annotations and then providing the means to be able to automate automate it for you one example of where this is used um is in cluster api so this is an example of a command that i ran recently when spinning up a local cluster api cluster um you're using cluster ctl as you can see and one of the very first things that it does it fetches the various different providers and then it installs cert manager and that might look a little curious if you but uh if you dig a little bit deeper once the cluster api is up and working then you've got cube ctl get certificates and specifically you look in the capi webhook system namespace you'll see there are a number of certificates that are used as serving certificates for those web hooks now service message becoming increasingly popular and a number of users in the community are asking how they can integrate cert manager and also with all its various different ca integrations into the mesh so that those workload search the control plane certain data inserts are provided from their sort of provider of choice now if you think of what service mesh provides it's a really really rich capability to have consistent observability uh security and so various reliability features that are built into the platform so rather than developers having to build this themselves it's transparent and it's provided for them this can be you know highly convenient rather than services contacting each other directly i've got an example here service a and service b and these microservices or services communicate via a proxy and the proxy can be dynamically programmed based on resources that reside in the control plane and so what's great about this is that you can provide all of that capability in the proxy rather than having to be implemented in the application itself and quite often most of these meshes so most of which are based of course on envoy and have the ability to automatically provide mtls between proxies in the mesh so you effectively transparent mtls between all of your services which is highly convenient a number of service meshes out there in the open source have integrations with set manager so linkid of course a service mesh that's in the cncf the ability to integrate some measure for some time you can provided a trust tank and an issue a certificate and private key and this can all be automated using cert manager um and thereafter linkid has a component built into it responsible for uh effectively uh automating the provisioning of the lease certificates and the renewal of those sleeve tickets with you know relatively short short lifetimes so you get a full automation of that both the bootstrapped certificates um that are used the trust anchor and also the lease certificates uh as well so there's a really good combination putting the two together um istio has had the ability to plug in custom ca certificates into its into a citadel for some time you can reference a ca key pair using a kubernetes secret but we've win working with the seo folks in order to more fully integrate certain managers so that you can actually have the workload certificates obtained from cert manager itself because that's advantageous because you can then start to take advantage of all the different issuers that set merger has in its community and what this involves is effectively replacing the citadel that's that registration authority with cert manager we can create certificate requests and have those fulfilled set by those different providers there's also some integration which is much more experimental and it's been more recently released that enables istio to use the kubernetes certificates api and we're working with them on that integration as well and finally the third example i have here is the open service mesh project osm that we've been working with to plug in servermanager and they've had plugable certificate management from the get-go and they've got their own built-in component i believe support for also volt and also azure's key vault um but yeah if you want to use so much there is an integration that enables you to uh plug it in um and it will that component will be responsible for creating and managing certificate requests for the workloads that run within the mesh so variety of different integrations that enable you to plug into certain manager and in turn of course all of its integrations that it has with different issuers but those that are built into the core of the project but also those that are external to the project as well cert manager can also be used to secure the kubernetes control plane and the nodes um that form former cluster for now for anyone that's tried setting up pki and kubernetes they'll know that there are a lot of certs um client search services of all different flavors and uh it can get quite complex trying to manage this i know from doing this in the very early days of provisioning a kubernetes cluster well now things have changed we can use the magic that is cubed m or cube adam is here to the rescue really it provides auto generated water renewed control plane inserts and by default this uses like a self-signed certificate so if you're happy to accept that it's uh not not rooted if you like in your in your existing chain of trust and then you can really just use this sort of as as is um but if you're in an enterprise where you need those certificates to be rooted in some kind of existing pki system then there are a couple of ways of doing this at least one way is using the external ca mode and this enables keyboardium basically to generate the private keys and then generate the csrs and then you fulfill effectively the certificate signing requests yourself and so we've built a jet stacking integration um between cuba dm and venify just to demonstrate that you can use this and since we actually did this this command cert generate csr has has actually begun ga so how about provisioning of certificates for the the gubernates nodes on the previous slide we saw how you can use qbdm to provision certificate for the control plane and how you can customize that and use you know an external external ca provided certificates but how about the nodes themselves well keyboard em uses the kubernetes certificates api and um that one uh that's actually been built into kubernetes for some time uh it's only recently i think 118 and that kind of g8 there's now like a v1 of that api so through a process of bootstrap uh the keyblade makes a certificate signing request uh that's this resource here to the api server and that certificate signing request as you can see here on the side has a has assigned a name and those that's not a required field um as of as of uh v1 and those signers are actually built into the keep controller managers that one there the cube api server client kubler signer that's built into the controller manager and that will automatically basically approve um yeah the signing of those certificates but you can also configure it in fact to sort of have like a manual approval and that you may want to provide i'm using a something like cube ctl so having a sign a name means that you can plug in also other signing mechanisms uh in that you know and cert in the same manager project we've actually worked on some experimental signs you can see a link to a couple of them uh there one is the sinus ca so that's effectively a curve local ca and also an integration um with venifi using its uh kind of open source visa library in the next release of certain manage we're actually going to be adding uh support or the next releases should i say open valuing support for the community certificates api so that means you're actually going to use all the various cert manager issuers um in order to sign a certificate signing quest just like this um and that'll open uh up you know open the project up to being able to support you know kind of many more uh use cases so we'll support this typical request that's brought into the project and certificate signing requests that's in the core of the project core of the open source kubernetes project and over time we'll look to sort of more fully embrace that core type in the project so to summarize there's lots of ways in which you can use set manager beyond ingress there are many use cases that stretch far and wide really across the ecosystem and you can use it for lots of different types of workloads including even the cluster itself so you can use the certificate and certificate request resources directly and programmatically you can use the csi driver if you want to be able to have pretty seamless pod identities created and have those renewed and how important you have the private key remain node local you can also integrate search manager with a variety of different service meshes we've got integration today with link d istio and open service mesh more to come the web hooks and also the control plane and node certificates as well as said in 1.3 or as of 1.3 we'll be supporting the kubernetes certificates api and we're looking for many more use cases so if you have some ideas or you want to become getting involved please come and join us set manager dev cert manager slack channels come and join us i'm always really willing to hear of new ideas feedback and also have contributors come join us help build it so thanks to everyone we're going to be available here now for some live q a after this session it's been a pleasure being able to tell you a bit more about the project and the various different use cases that we're building set manager for look for your questions stay safe everyone thanks bye-bye

Channel: CNCF [Cloud Native Computing Foundation]

Views: 4,791

Keywords:

Id: wEW2kVKxgss

Length: 29min 37sec (1777 seconds)

Published: Fri May 14 2021