Cert Manager Kubernetes Tutorial (Let's Encrypt & Nginx Ingress & ACME | 5 Examples | YAML & HELM)

search manager adds certificates and certificate issues as a custom resource types in the kubernetes cluster it simplifies the process of obtaining renewing and using those certificates search manager can issue certificates from a variety of supported sources including let's encrypt hershey corp fault vinafi as well as private pki the typical workflow will look like this you create a certificate easier for example let's encrypt when you create an ingress for your service you will specify in the annotation that you want to use that let's encrypt easier to secure your ingress when you apply the search manager will detect that annotation and issue a certificate from let's encrypt and store it in the kubernetes secret that ingress can use search manager provides custom resource issuer that namespace specific and must be used to obtain certificates in the same name space where it was created and cluster issuer which can be used in any namespace similar to airbag role versus cluster role also in this video we will monitor certificates and desert manager with prometheus and grafana i'll show you how to define grafana data source and a dashboard in the code rather than do it manually from the ui you will be able to see all the certificates and expiration dates in the dashboard you can use alertmanager to set up alerts if those certificates are close to expiration and weren't renewed in this video we will go over five examples in the first one we simply create a self-signed certificate the primary use case for self-signed certificates is to use as certificate authority to use you other certificates in other words to bootstrap your pki public key infrastructure in the second example we will use that self sign ca to create another type of the third manager issuer which is ca for the third example we would need to deploy nginx in grass controller and a grafana sometimes we want to have private host names and the only way to get certificates for them is to use your own ca and issue certificates you may think that it's secure enough to keep them with plain hdp on a private subnet but in this example i'll show you what can happen in the case of men in the middle attack we will use wireshark to capture raw tcp packets between the grafana and us we will create a capture filter to only watch for post requests to grafana and update the credentials username and password after that we will secure grafana in grass with a certificate that will be issued from the same ca that we created previously finally in the fourth example we will create a let's encrypt issuer and obtain the certificate using the staging environment first for this example we will use the http 01 challenge the most important lesson here is that i'll show you how to debug if you face all sorts of issues our fifth and final example we'll use the let's encrypt issuer that will use the dns-01 challenge here i'll show you how to transfer the subdomain to route 53 in my case the monitoring sub-domain to follow along you would need a kubernetes cluster you can easily create one using eksctl tool and myconfig by the way all the commands that i run in the video and the source code are available in my github repository first let's check if we can connect to our cluster it's a standard check that returns the kubernetes api service in the default namespace i'm trying to make all my videos self-contained that you would have all the components needed for the following examples most of the cloud native services such as short manager enginex ingress and other xposed metrics in the parameters format where i can i try to use those metrics in my videos and create grafana dashboards for them this video is no exception and we will need to deploy the primitives to our cluster if you want to learn more about primitives you should watch another video how to install primitives on kubernetes cluster let's quickly deploy parameters we have crowds folder with custom resource definitions then we have primitives operator that will monitor those crowds and create corresponding objects in kubernetes and finally we have parameters folder we have airbag policies and the main parameters object to create primitives instance here are few important configurations primitives will watch service monitor objects with the label primitives equal to lesson83 by default parameters will only detect those series monitor objects in its own namespace in this case monitoring we can instruct parameters to select series monitor objects in all namespaces with the label monitoring equal to primitives we need it since the assert manager helm chart will create a series monitor object in the search manager namespace now let's go to the terminal and apply those folders one by one first let's create custom resource definitions for prometheus operator then let's deploy prime ethos apparature with its airbag policies and finally the parameters itself parameters operator will convert primitive's custom resource into stateful set and a config map now let's make sure that primitives is ready before we can continue the next step is to deploy assert manager whenever you deploy services in kubernetes with yamo or helm it's a good starting point to take open source home chart and adapt it to your environment then you can generate yaml files and apply them or adjust overwrite default helm variables and install it using home let's see what default variables are in the search manager home chart you can find it in the jet stack search manager github repo we're going to override a few of them [Music] i assume you already have helm cli installed let's add that helm chart replace today let's search to see what versions are available right now this is one of those tutorials where you want to install the same versions first we have a lot of moving parts engine linux and grass kubernetes versions and search manager let's use the 153 version which is the latest one at this moment create search manager values.yaml file to customize the helm chart deployment first we need to enable kratz helm chart will create the following custom resource definitions each year cluster issuer certificate certificate request order and a challenge then let's enable primethose matrix and create a service monitor object this label has to match to the one on the parameters object this helm chart will take the primitives instance key and convert it to simply premethos we're going to add a few more later when we get to example 5. now we can optionally generate eml files by using the helm template command provide release name chart name namespace where you want to deploy it version of the chart the path to the values that you want to override and the output directory let's see what we got all the files will be placed in the helm generated yaml folder for example the cluster resource namespace flag will specify the namespace where the kubernetes secrets will be created when you use cluster easier instead of issuer [Music] when you deploy the helm chart to the namespace that doesn't exist yet you can use the create namespace flock for this example we want to create a namespace ahead of time we need assert manager namespace to have monitoring equal to the primitives label in order to prometheus select the service monitor object go ahead and create a namespace first now let's use the helm install instead of the template and provide similar flux this will deploy crowds and assert manager pods when you use helm3 to install the chart it is a namespace code let's make sure that all the ports are ready we have three different services the main one is the third manager then the ca injector which helps to configure the ica certificates for mutating webhooks validating webhooks and conversation web hooks and a webhook one of the functionality is to ensure that when search manager resources are created or updated they conform to the rules of the api simply that you provide only valid arguments now let's verify that the parameters config was updated and we can monitor the assert manager let's port forward primitives operated service and go to the target section all right we have the third manager target later on we will deploy grafana with a dashboard let's create our first example with a self-signed certificate to bootstrap our ca the first step is to define cluster easier it's going to be the simplest one you don't need to specify the namespace since it's a cluster scope and only one parameter is self-signed it's not the certificate in just a resource that we will use to generate one now create a ca certificate kind is a certificate then you need to specify the namespace where the certificate and the key pair will be stored let's make it as ca which means we can use it to issue and sign other child certificates the typical duration for the certificate authority is 5 years then the common name it doesn't really matter what you will put there it doesn't have to match with existing domains then the kubernetes secret name the third manager will generate a private key and a certificate and place them into the secret in the search manager namespace then you can choose what algorithm you want the final step is to specify what cluster issuer you want to use this should match the one we just created let's go ahead and apply the 1 folder to get the certificate run qctl get certificate in the 3rd manager namespace this object does not contain the certificate and a private key itself instead it has a reference to the kubernetes secret now let's get that secret the certificate and a private key are stored in the divorce by example io key pair secret let's get it from there you can use the output yama flag to get base64 in coded values this secret contains the private key stored in the tls key secret key then the certificate in the tls search since it's a self-signed certificate tls search is equal to ca shirt let's get the value and decode it using base64 tool dash d2d code and all to provide the file name where we want to store it now to make it human readable run the openness cell command and provide the path to the certificate you can see that it's a cea certificate it will be expired in 5 years the next example will use this certificate to issue a new one you don't have to use ca that we generated if you have an existing certificate you just need to import it to the kubernetes using the secret create a new example 2 folder and the cluster issuer to indicate that we want to create a certificate authority type easier you just need to specify the ca key and provide the path to the kubernetes secret since it's a cluster issuer that secret must be located in the same namespace where we deployed search manager this can be modified with a cluster resource namespace plug the secret device by example io key pair was generated in the first example here i want to create a certificate in another namespace let's create a new one and call it staging then a certificate itself for the blog devops by example.io give it a name and a namespace this will place a certificate and a secret in the staging namespace then set ca to false choose a duration let's encrypt usual certificates for 90 days so let's use that magic number here then you can define when the search has to be renewed the common name on the certificate and a list of dns names that you want this certificate for this field will be translated to subject alternative name on the certificate kubernetes secret to store key pair algorithm for the certificate finally cluster issuer which is based on osce let's apply all of them list certificates in the staging namespace if the status is ready means the key pair was generated and stored in the kubernetes secret the secret name is block devops by example io key pair let's print that certificate and decode it with opensl as we did it for ca the difference here now tls3 and ca third is different let's start in the file as well here you can see two hostnames blog and www blog they came from the dns names key on the certificate object also it's valid not for 5 years but for 90 days next we need to deploy the nginx in grass controller in this video we will quickly override few variables and deploy using helm if you want to learn more about nginx ingress you should watch nginx in grass controller for kubernetes tutorial to start add the ingress engine x repository and run helm update [Music] for this video we will use the 401 version let's create nginx ingress values and override few default variables first one we need to give to ingress class name then let's disable the webhook for now just to avoid additional complexity it does not affect the ingress it only verifies if you ingress config is wallet before updating the nginx also in adws when you use the service of type load balancer adws will provision a classic lb let's upgrade it to the nlb network load balancer using this annotation now this is important part that needs to be configured to pass the http 01 challenge by default the nginx ingress controller doesn't watch anymore ingresses with the ingress class annotation it will only select ingresses with a new ingress class name key since third manager still creates ingresses with depreciated annotation to pass the http01 challenge we need to enable watch ingress without class and give it a name we need to pass the ingress class flag as an extra argument to the controller that's it for now let's go and deploy it provide helm release name chart name namespace version values and enable create namespace since it doesn't exist and we're not going to monitor engine x ingress in this video as i did in the previous one let's verify that ingress controller is ready before proceeding as i mentioned the primary mechanism of specifying in grass class is a new type called ingress class now let's deploy grafana it will be used both to illustrate how to create ingressors with sort manager as well as to monitor certificates and expiration dates first let's create a kubernetes secret with username and password encoded in base64 format the user is admin and the password is devops123 then create a config map with a datasource.yaml key that will be used as a file name when we mount to the port you can create a data source from the ui or you can predefine it in the yaml since prometheus is deployed in the same monitoring namespace we can just use the kubernetes service name parameter separated and the port also we're going to put our assert manager grafana dashboard to the config map and mount it as a file as well we need to provide the path for grafana to discover our dashboard this is our search manager dashboard defined as a json object finally in the deployment you can see that we create volumes from all of those config maps kubernetes dashboards data sources and dashboards then we mount them to the pod we need a service to create ingress later now let's go to the terminal and apply the grafana folder we're going to use the grafana service in the ingress definition with the port 3000 for the third example ingress objects must be created in the same name space where you have your service now let's use the ingress class for nginx and grass controller define the domain graphana devops by example.io and the service first we create ingress without tls section and circ manager later we will come back to it and secure it with https alright we have ingress for grafana now we need to create a cname record that will point to aws public load balancer hostname my domain is registered with google domains but it doesn't matter you just need to create dns record and point to that lb let's check if you can access grafana for now use http it works let's enter admin for the username and devops123 for the password we have a data source that points to the parameters out of the box also we have a dashboard currently we have only two certificates and the soonest expiration is almost 90 days now we need to sign out since we want to test if you can get credentials in plain text when someone logs in in this example i want to use wireshark and simulate man in a middle attack wireshark is a widely used network protocol analyzer used for troubleshooting analysis software and communications protocol development let me install it with homebrew i'm not going to use ui just a command line interface and instead of display filter i'm going to use a capture filter they have different syntaxes to start we need to specify the network interface that you want to attach with wireshark you can list network interfaces with ifconfig or ip other command on linux let's use en1 as our primary interface with the private ip address first you need to provide the network interface en1 x4 output of hex and ascii dump packet bytes then let's create a capture filter for wireshark we can use boolean expressions and or etc first we want to monitor a traffic that comes to the grafana devos by example host then we know that it is on standard port 80 since ingresses only serve hdp on port 80 and https on port 443 then the most interesting we want to capture only post requests let's open ascii table let's convert post to the ascii hexadecimal values capital p translates to 50 or to 4f as to 53 t254 then when the tcp packet arrives we check if it contains post finally let's redirect output to the file post [Music] to test go back to the grafana and enter your credentials we can see that we captured one tcp packet that satisfies our filter let's open it alright we got the username and the password in the plain text here now let's secure it with tls and test it if you can read tcp packets and get the password again we want to use the cluster issuer that we created in the previous example number two then we can optionally define the duration and the renew field also we need to add a tls section hosts will be used as subject alternative names and name of the secret where you want to store the certificate and a private key let's see if the certificate was successfully issued by the assert manager ca easier it is in a ready state so we can continue if you list ingresses you can see additional port 443 that was added to the grafana ingress now you need to use https it works but since it was issued by the self-signed certificate it's not trusted let's add ca to the keychain on a mac [Music] [Music] let's select the ca we created this file in the first example when we generated a self sign server make it trusted now if you refresh the page the error should disappear this certificate is valid let's use wireshark to capture some packets on port 403 we can use host and post request since those packets are encrypted and we don't have access to the content of the tcp packet let's see now all the packets that we get with wireshark are encrypted and we can't get a username and password as well as any other useful information example number four finally we're going to start issuing certificates from let's encrypt in this example we will use let's encrypt staging environment and in the fifth example we will use let's encrypt production environment now instead of cluster easier let's use issuer it's not a big difference it just you can use that issuer only in the namespace where you created it here we're going to use the acme key acme stands for automatic certificate management environment protocol always use staging environment first when testing let's encrypt automation then provide the real email in case if let's encrypt fails to renew the certificate you will get an email with a warning then the secret name for solvers you can use either http 01 challenge or dns-01 on line 16 we specify what in grass class will be used to resolve that challenge search manager will create additional temporary ingress using that class to prove to let's encrypt that we control our domain and a server then almost identical but the issuer is for production let's encrypt environment if you use this one you will get a wallet certificate the difference here is only the server url well and a different secret name and issuer name let's apply folder example4 before using those issues check if they in ready state if you describe the issue you should see the message that the acme account was registered now let's create ingress for prometheus i mentioned in the previous video in general it's a bad idea to expose internal services to the internet but here we have only one single public ingress so let's do it in the annotation use issuer and not cluster issuer that points to the let's encrypt then the pretty standard fields for the ingress point to the primitive kubernetes service now i think the most important part of the video i'll show you how to debug the issuer if the assert manager fails to obtain the certificate the flow when using acme with sort manager in the following first it will create a certificate then it will check if there is a valid one already if not then it will create certificate request then the certificate request will create an order and finally a challenge let's see if the certificate is ready now nope it's false [Music] let's describe the certificate to get more information about the problem you can see certificate request was created it was approved but it's not ready let's describe it as well order was created and the certificate object waits till it's done let's move to the orders then we have only one order for the parameters certificate let's describe all right order created the challenge let's list and describe the challenge you see it's in painting state finally we get a meaningful message parameters divorce by example that i owe no such host well it's expected since we never created cname for our parameters ingress also when the cert manager creates ingress for the http01 challenge it will also spin up the port you need to make sure that that port is in readystate after you pass http challenge servbot will delete that pod you can see that it's running also we have acme grass now here if you're not going to specify the nginx ingress watching grass without class that ingress will be ignored and you will not get the load balancer hostname let me print out the ingress you're not going to find the ingress class field here only the annotation in grass class equal to external nginx by default nginx ingress will not watch those annotations anymore before i create cname let me split the screen in the first one i will run get certificates and in the second one get challenges that's actually a reason why i use dns 01 challenges in my production environments it allows you to create ingress and obtain the valid certificate without creating a cname and redirecting traffic to the service that is not ready yet and does not have the valid certificate now let's create a cname for parameters ingress it's going to be the same load balancer hostname as for grafana challenge was completed and it disappeared and the state for the certificate change to ready if you check the parameters in the browser now you get a certificate but it's going to be from the staging environment you can see that the certificate is not trusted to get a real one just use let's encrypt http 01 pro easier it's time for the last example first we're going to delegate a sub-domain to route 53 and create dns-01 let's encrypt easier you can delegate the whole domain to route 53 but just for this example i decided to delegate only the monitoring subdomain that means that all sub-domains such as grafana monitoring devos by example that io will be resolved using route 53 and not google domains first we need to create a public hosted zone in route 53 let's call it monitoring devos by example.io and keep the public hosted done in the future videos i'll show you how to create private hosted zones and use openvpn to push dns name servers to your development host that you can resolve private host names delegate subdomain is very easy you just need to create ns record for your subdomain and point to route 53 name servers let's do it now now let's test the subdomain create a record and try to resolve it locally with a dig ip address can be anything just to test dns [Music] all right we successfully delegated monitoring subdomain to route 53 since we will use imrols for kubernetes service accounts we need to create open id connect first if you maintain your own kubernetes for example with cops you need to create a similar imroll and attach to the kubernetes workers we need to get an open id connect url from the eks cluster then use it to create identity provider now we need to create iron policy and grant access to create dns records in route 53 the way the dns-01 challenge works you need to prove to listen crypt that you control domain by creating a specific txt record with a token that lets encrypt gives you when let's encrypt verifies that you have a txt record they will issue a certificate you need to do it every time whenever you want to renew your certificate approximately every 60 days you can see why without automation it's not a viable option to renew search by yourself the first statement is to be able to get the current state of the request to find out if dns record changes have propagated to all route 53 dns servers the second one to update dns records such as txt for acme challenge we need to replace the id with the hosted zone id [Music] [Music] let's give the name sort manager route 53 access and click create policy now we need to create imroll and associated with the kubernetes service account you need to select the open id connect provider under web identity let's call it cert manager acme later we will use this irn in the kubernetes service account to allow only our shared manager kubernetes account to assume this role we need to update trust relationships let's find out the name of the service account attached to the third manager deployment it's called cert 083 search manager here update the audience to sub and sts amazon aws.com to the service account search manager is a namespace for that service account and the last argument is the name of the series account itself we completed the im configuration on the adws site now we need to update a couple of kubernetes objects to specify the item role to use by the series account you need to add an annotation by default it will use the one that is attached to the kubernetes workers this amazon resource name you can get from the adws console [Music] that has to be your account id we need to update deployment as well to include additional flag to the circ manager you need to add issuer ambient credentials to be able to use that i am role if you are using cluster issuer you need to add or replace this one with cluster issuer ambient credentials when you make this change kubernetes will recreate a pot by the way the sort manager helm chart allows you to specify those configuration options before you install it in the helm values you can add service account annotation with the irn of the i am role and extra arguments as well let's create the final example with a dns resolver it's going to be similar to the first one for the staging environment the difference here instead of http use dns then specify the region and the route 53 hosted zone id the second one is for production environment to get real certificates we will use this one for the last certificate the difference is in the server url let's create the last grafana ingress and use that dns issuer under easier specify let's encrypt dns-01 prod the rest of it is pretty standard hosts secret and a service let's split and watch those certificates and challenges and apply example 5 folder with the issues and grafana ingress let's check if the assert manager was able to create a txt record yep it's here magic acme challenge record challenge state transition to wallet and in a second window certificate will be ready [Music] if you have any issues the best way to find the error is insert manager logs [Music] now we need to create a cname record for grafana in route 53 since we delegated the monitoring subname in there it's a valid certificate issued for 90 days by let's encrypt if you login to grafana you will find more certificates there thank you for watching and i'll see you in the next video [Music]

Channel: Anton Putra

Views: 13,212

Keywords: Cert Manager Kubernetes Tutorial, letsencrypt, cert-manager kubernetes, kubernetes ingress controller, lets encrypt, cert manager, kubernetes ingress, ingress kubernetes, kubernetes tutorial, cert manager kubernetes, cloud native, kubernetes ingress controller nginx, nginx ingress controller, kubernetes nginx ingress, nginx ingress letsencrypt, nginx ingress controller eks, nginx ingress kubernetes, devops, anton putra, sre, kubernetes, prometheus, grafana, tls, cert-manager, aws, eks

Id: 7m4_kZOObzw

Length: 42min 59sec (2579 seconds)

Published: Mon Sep 06 2021