Monitor TLS Certificates With Prometheus And Grafana

unless you're using an automated method obtaining TLS certificates for your devices then you need to manage and monitor them and the reason for that is because a certificate will have an expiry date or at least it should if you're concerned about security so you need to be proactive and replace certificates before they expire otherwise a browser will either complain when the certificate expires and so all the users but we're still a browser will just refuse to connect to a device now an interesting open source combination of monitoring tools that's available for free is Prometheus and grafana but how do you configure Prometheus and grafana to monitor DLS certificates well if that's something that you're interested in finding out and stick around and watch this video as that's what we'll be going over now because this video is specifically about monitoring TLS certificates I'm going to assume that you already have Prometheus and graphara installed or you know how to set these up not I do have another video which shows you how to install and configure these in docker now what we're going to be using to actually monitor our TLS certificate says the Black Box exporter so first thing that we need to do is to install that now I'm using Docker can pause so I'm going to use Nano to edit the docker Dash compose yml file this is already being set up so all I need to do is go to the end of the file here then I'm going to copy and paste in the details for this actual container and what this contains is basically the same information you'll find on the GitHub web page I've just adapted it for Docker compose so we've got details of the image we want to use the name of the container just to make it a bit easier to identify default Port is 9115 or restart I do want the actual container to actually restart if the computer reboots for example but if I want to carry out maintenance then if I actually manually stop this container I don't want it to automatically start up again so that's why I'm going to set to unless stop instead of always now the actual export it does need a configuration file from the perspective of the actual container it's got a folder called slash config what we're going to do is we're actually going to create a folder called black box within the current working directory where Docker compose.yml exists that's why I've got the dot and then the slash so that just makes it easy because you need a configuration file that's got to exist outside of the container itself so that's what that mapping is for and then you've got a command there just to actually let the actual exporter know what the configuration file is but again this is from the perspective of the container its perspective is it's a folder called slash config and in there as an actual file called blackbox.yml that file is actually going to exist within our actual real folder black box so now that we've defined the container I'm just going to exit out and save this actual file now the next thing to do is to set up a configuration file for this export but before I can even do that I've got to create the folder where we'll be storing it so this folder is getting called black box that's the same name that gets referenced within the actual contains config file then we'll create the file itself and this is going to get called blackbox.yml then in there I'm going to copy and paste in the configuration now this is being taken from an example on the GitHub web page and all I've done is to add in this external module so it's entirely up to you whether you want to include all these other modules or not but the useful examples for other types of connections you can test as far as this video goes we've got a module called https underscore 2xx you can call it something else if you like I've then got a setting of Bill underscore if underscore not underscore SSL in other words I want to make sure that I'm on a secure connection because by default is a set of false sum deliberately setting it the true for that reason the whole reason I want this module is to actually test these TLS certificates for method we're using get so just like a web browser it'll connect to the server and then use a get command to pull information down and then I've got setting here of follow underscore redirects if you look at older examples this is referred to as nor underscore follow underscore redirects so this newer version makes a lot more sense it's a lot easier to understand because it's the older version was basically reverse logic so this is a lot easier to um to understand because it's just basically seen as are you going to allow the actual server to redirect your session well I don't want that because there's the risk of an actual server directing me from an https session to an HTTP session and that's not going to help if I'm trying to monitor um these TLS certificates so I've set that defaults for the preferred IP protocol by default it's IPv6 so because I'm using ipv4 I do have to specify that and because I'm using my own certificate Authority the actual computer is not going to trust any certificates that it sees unless I actually tell it about the root certificate so if you're using a public certificate Authority you're fine if you're using your own certificate Authority then you do have to point it to your root certificate file the only thing that actually point out here is that we're looking at things from the container's perspective so yes I will be putting this file that we've got here for my routes here into this black box folder but from the container's perspective the fold is called config so I just need to bear in mind and then after that we've got the type of proverb and then we've got time out of five seconds so I'll just save that config and then I need to copy across my actual certificate file for my root certificate uh thoroughly and then what we can do is actually start up this container so because I'm using Docker compose I'm going to do Docker compose up then Dash D in other words it's going to bring up all containers that are referenced in that yaml file unless they're already running but it's going to do it in the background because that Dash D option so I'll hit return so they should be quick because I've already got the image downloaded and then what I can do is just dot PS and I'll do Dash l as an option because it'll just tell me about the last container so it's up I'm not seeing it any warnings about a restart so that looks to be fine anything to point out is we're running on that default Port of 9115 but we've now got the actual its border up and running now the next thing to do is to configure Prometheus to actually scrape the metrics from this exporter so what I need to do is to edit a configuration file called prometheus.yml and for me that's in a folder called Prometheus and then I need a copy and paste in the actual job for this export so it's called the end of the file here now this actual job is based on the exam you get from the GitHub web page key things to point out is that I'm using a module here called https underscore 2xx in other words that's the actual module that we defined within the exporters configuration file so you need to make sure that's correct then when it comes to the targets I've got it pointing at two Apache web servers as a test now if you notice here I'm using the fully qualified domain name as well as https here and the reason being is typically a TLS certificate it's going to be based on a fully qualified domain name now because these are just your basic web server I can just basically get away with pointing at the root fold and it won't matter but if you're dealing with an actual server that's going to do a redirect to some other landing page make sure that that URL is the actual file landing page so as an example true nets for example if you were just point in your web browser at the actual root folder you get redirected to the login page so it would really be the login page that we'd want to put here because the way I've got this black box exporter setup is that it doesn't tolerate redirects so firstly do be careful exactly what URL you put in here it's best to test it in a web browser first to make sure that you actually know what the final landing page is now what we do within this job because the exporter isn't running as an agent on the server is to do some re-labeling and then finally you actually point to the IP address and Port but the actual Black Box server is listening on now if this had been a bare metal computer or a virtual machine and I had all of this running as Individual Services then I could have just used localhost for example but because I'm using this all on a Docker platform I can't use that it wouldn't work I've got to use the actual IP address of the computer to get this to work so we'll save our configuration file and the next thing to do is to actually reload Prometheus because it needs to actually get that configuration file updated and loaded into memory now one option is to actually reload the container in my case but um the way I've got this set up is I can actually get it to reload the actual configuration file and get it actually loaded on the Fly and I can do that using the curl command we're just basically post an actual command to it and tell it to reload in this case I can use localhost because as far as the computer is concerned there's a service on here listening on Port 1990 which is Prometheus so I'm just going to run that and now what I've got to do is just basically wait for this the whole kick into action basically because although I've updated the configuration for Prometheus and I actually got it to lower that config in the memory I've still got to wait for the timer before it actually starts doing anything I've left this a while to actually start Gathering metrics and no one can actually check them on Prometheus itself for instance so if we go to status and then to targets you can see I've got a reference here called Black Box and it's seeing two of two up and if I click on show more for instance you can see we're actually seeing a state for these two uh actual targets as up but that doesn't really tell us much especially about the actual TLS certificates to get that information we have to go and query the actual Explorer itself and for me that's going to be on Port 9115 this same computer and that's what I get when I point the web browser to the actual computer and Port 9115. if I click on the actual refresh button there you can see how it's regularly pooling these actual um Apache servers that I've set up so I've got our Apache one and Apache 2. now one of them is showing a natural result of success and another one that's actually showing a result of failure now that's to be expected in my case because I actually have deliberately set this up so that the actual certificate for Server 2 has actually expired you can find more information if you click on the logs for instance here scroll through this and it's just saying build to actually verify the certificate the certificate is expiry expired or is not valid and say in my case it has actually expired so checking on Prometheus itself is a useful way to test at least if the connectivity to the actual servers there in other words the server is still listening and active whereas if we go to the actual exporter we can find out more details about problems specifically about things like you know certificates expiring is to set up a dashboard in grafana and although you can actually do that yourself fortunately there are people actually creating dashboards and making them available for free so when it comes to this exporter we're going to take advantage of this dashboard called blackbox exporter HTTP program and what we're going to do is click on the option copy ID the clipboard then go over to grafana select the menu and then select dashboards and then click on the option new and then select import and then what I'll do is just copy and paste in the actual ID and then click on load you can change the name if you like but do need to make sure you select the actual data source which is Prometheus and then click on import and there you go we've now actually got a much better way of actually monitoring our actual servers as well as the actual certificates on them so I've actually got two Targets set up for this exporter not surprisingly I've got one showing a state sub up the other ones showing states down but you can get some really useful information out of these so for example it's coming back for this one showing a call of 200 which is what you'd expect at the TLs side of it or SSL as they're referring to it works you can get the TLs version it's used as well as natural expiry so this is extremely useful it lets you know how long you've actually got left on that actual um certificate in terms of how long it's going to be before it expires so this is a great way to actually monitor your certificates now ideally what you want to do is stay ahead of the curve here and be more proactive you don't want to reach a state or sit down because the actual certificate has expired so we do want to keep an eye on these actual expiry days but extremely useful way to be able to monitor these TLS certificates now if you find this video to be useful then do consider subscribing to the channel does that really mean a lot to me but it's also a good indicator to let me know how a video is like this are helpful to people such as yourselves that are watching in which case thank you on the other hand if you're not ready for that level of commitment then I'd really appreciate it if you could press the like button is that where that will help me get the video out to other people that might find it useful as well

Channel: Tech Tutorials - David McKone

Views: 1,355

Keywords: monitor certificate expiration grafana, prometheus monitor certificate expiration, prometheus monitoring tls certificates, monitor tls certificates, monitor tls certifcate expiry with prometheus and grafana, monitor tls certificate expiry

Id: 4UGB9pNjSvM

Length: 14min 9sec (849 seconds)

Published: Fri Sep 01 2023