Monitoring Proxmox VE With Prometheus And Grafana

monitoring computers is very important and even more sort of hypervisors like proxbox ve because if something goes wrong it will likely affect all of the virtual computers that have been run by that one physical computer now an interesting open source combination of monitoring tools that's available for free is Prometheus and grafana but how do you configure Prometheus and grafana to monitor proximox V if you're running them in the docker for instance well if that's something that you're interested in finding out then stick around and watch this video as that's what we'll be going over now because this video is specifically about monitoring proxbox ve I'm going to assume that you already have Prometheus and grafana installed or you know how to set these up if not then I do have another video which shows you how to install and configure these in docker now all Prometheus is relying on an exporter to get its metrics we're not going to be installing any additional software on approxbox VE nodes instead we're going to rely on proxmox's API in which case the first thing we need to do is to set up a user account and an API token that our Explorer can use so to do that we go to date sender and then underneath permissions we click on users then we'll click on ADD you can call this whatever it is you like I'm going to call this one mikis but I would suggest using something less obvious than that name and then you can fill in the rest of the details if you like but then click add next thing we need to do is to assign permissions so we click permissions and then click on the add drop down menu and select user permission for the path we'll select the forward slash because we're monitoring everything basically the user we'll just click on that drop down menu and select the user account that we created then for the rule we'll click on that drop down menu we'll scroll down until we find pde auditor and we'll select that and then we'll click on ADD next thing to do is to set up the API token that we're going to use because you might have noticed when we create that account we weren't asked to assign natural password anywhere so we'll select the API tokens then click on ADD we need to change the user from root to the account that we've just created you can put what you like into the token ID but I'm just going to call this one exporter now one key thing to point out is this privileged separation which by default is enabled you do want to disable that because if you enable that it means you've got separate permissions for the actual user account and separate permissions for the actual token we just want to keep things simple we've already signed permissions for the actual user account so by disabling this it means you get the same permissions regardless and when he's not going to make any difference because nobody's going to actually log in to this actual um computer system using that account so I don't need separate permissions anywhere now from a security perspective it does make sense to actually set an expiry date and to do that you can click on the calendar option here typically when you set up tokens they usually expire after maybe 30 days or roughly a month so I'm just going to jump forward a month and I'm going to pick to the end of this month in September for example the only drawback of that is it means you do have to actually maintain the actual token going forward I mean what you can do is you can actually just keep extending the expiry date if you like which is quite useful the only thing you actually need to bear in mind though is that although that means you don't have to actually then update the actual export or with a new token it does mean that if there's ever a breach you're not aware of it and someone's got access to that talk and well still be able to get access to the system so there's different ways of managing this but I would suggest from an actual security perspective at least selling some expiry date that's practical in any case what we then need to do is click on ADD then what that does is to give you what the token ID is so it's basically the name Palm then an exclamation mark followed by the actual token ID it's easy to get that back again and recreate it the only problem is the secret that it actually shows you this is extremely important you don't want anybody seeing this and this is a one-off view that you're going to see you need to copy this down and store it somewhere safe going forward because that's it as soon as I click on that close button here it's gone I won't be able to find out what this is anymore so you can either just copy and paste it like that or click on the option to copy the secret value to the clipboard but again just just remember that is a very private um actual secret key there I don't mind showing it on a video because I can easily recreate this even before the video goes back out anyway so it's going to be different regardless and it does only an actual test lab anywhere but once we've got these we can then use those to actually set up our export when we configure it now it's usually not a good idea to be installing third-party software on a dedicated platform like a hypervisor especially if you're relying on technical support for instance unfortunately this actual layer exporter we're going to be using can be installed in a Docker container and then it can connect remotely to our actual nodes so because I'm going to use Docker controls here I'm going to edit the yaml file so I'm going to use Nano to edit Docker Dash compose.yml so this is already set up to run other containers so I don't need to put in the version of a Services block there but I'm going right to the end I'm just going to hit return and then copy and paste in configuration for this exporter we're going to run so I've given a name of pve-ex4 and I'm planning it to the image to use to make things a bit easier to identify I'm giving it a container name of PVE Dash exporter then I'm just going to stick with the default Port here which is 9221 now I do want this running 24x7 but rather than setting restart to always I've got it set to unless Dash stopped the reason being is that I can actually manually turn the actual container off or stop it and it won't automatically start it back up on your hand if the actual computer were to reboot then the actual container would be started up and then finally I've got to point the actual configuration file out to an external file so to container sees this file as being slash Etsy pv.yml but we're going to create our own actual file which is going to be basically in the actual folder where we've got Docker Dash compose.yml then there's I'm going to be setting up a subfolder called PVE and then I'll keep the configuration file of pve.yml in there so I'm just going to hit return I'm going to save and exit out of here and then that actually gets a container setup now as part of the setup of this exporter we need to create a configuration file for it and because I keep all of my config files separate what I need to do is to actually create a separate folder for this so I'm just going to use make dir to create a new folder called PVE and then what we'll do is to actually create a config file so this is called pde.yml so this is the same path and file name that we've referenced with our Docker compose file so I'm going to hit return then I'm going to copy and paste in the actual configuration details so right at the top we've got default and then we need to Define our actual user account now I call this user Prometheus so you have to put in whatever username you use but it's got app at the end so overall username is Prometheus um now we're not using a username and a password to actually connect in we're using tokens so I've got to define the token name so that's the token ID that we referenced when we actually created the token within proxmoxv but know that it's in actual chords because it's a string here and then I have to tell it what the token value is so this is the actual secret that we've got back when we actually created that token again that's in quotes but you have to put in here whatever it is that you you got back Now by default proxbox ve just like all the systems is using self-sign certificate so unless you've actually changed that and created your own certificates and you've actually got your system to actually trust them then when it comes to this particular setup you want to put in verify underscore SSL and set that to false in other words we just bypass that actual check so same exit out of that file and then what I want to do is to actually start up that container so I'm just going to run Docker compose up Dash D so this will basically start up any container finding that yaml file if it isn't already started but it's going to do this in the background so I'll hit return now it should be quick because I've already got the image downloaded anyway to save time so it says that it's actually started but I'm going to run Docker PS just to make sure it hasn't come back with any warning so we've got our Prometheus PVE Dash X4 right at the top there so I'm just gonna keep checking yep it looks as though it's okay it certainly hasn't been restarting so at least that's actually now got the exporter up and running now the next thing to do is to configure Prometheus itself and give it a job to actually script metrics from this exporter so I need to edit the configuration file for Prometheus which for me it's called prometheus.yml and it's in a folder called Prometheus I'm going to go to the end here and I'm going to copy and paste in the actual configuration for this new job now this configuration is based on the example that you actually get from the developer of the actual exporter itself so it's got a job name of PVE and then here in this static config section we actually Define the nodes that we actually want it to connect to and we've got a metrics path some parameters here and then right at the end we've got the actual IP address and Port that we connect to to actually talk to the exporter the key point about all this bit though is that we're doing re-labeling reason being is that we've got Prometheus actually talking to an exporter which is on the same computer as Prometheus itself and we're then connecting out to some external computers so we need to do a bit of re-labeling otherwise things will get a bit confusing so I'm just going to hit return and save this actual file now because I've changed the config file I need to actually get Prometheus to update itself you could actually restart the container but I've actually configured Prometheus to allow me to actually get it to do a Reload of that config on the flag so I can just use the curl Dash X command here so I'm going to hit return and that gets Prometheus to reload in its config file without having to actually restart the container but either way I'm gonna have to give this a bit of time now the actual system to collect metrics from our actual proxbox nodes well I've waited a while for the actual X4 to start collecting information and I'm now on Prometheus and if I go to status and then to targets I've got a Target here called PVE and it's showing three of three and up which is good to see if I click on sure more it's actually showing me the actual details of those three actual uh nodes that I've got it talking to and what it can do is you can actually get details of those actual metrics but one thing to point out is that you can't just point it at the actual IP address and the actual Port four the actual computer if I put that in and just hit return you see it actually tells you to point to a specific Target so if I change that URL and actually get it to reference one of the actual um nodes that we've got set up here hit return you can see there's a whole bunch of metrics that it's actually collecting from the actual node so that's good to see boxmox is talking to the actual exporter and the exporters now actually gathering information from our actual nodes well now that Prometheus is gathering metrics from our proxbox ve nodes the last thing to do is to set up grafana and to create a dashboard that presents all of these actual metrics in a much more easy to understand format now you can create your own dashboard in grafana if you like to but there are actual dashboards that other people have created that you can take advantage of this one here proxmox via Prometheus is one that's actually suggested by the actual developer of the Explorer so we're going to be using this so what you do is click on copy ID to clipboard then go over to rafana we'll click on the menu option it's like dashboards and then on the new drop down menu we'll select import and then I'm going to paste in the actual ID and then click on the load you can change the name if you like but either way we've got to select the data source which is Prometheus and then click on import and then what that gives us is much more easier way to understand all of the metrics that are being pulled in and there's there's quite a lot of good information in here I mean you've got Network information disk IO and so on we've got our CPU history and memory utilization that's going on as a whole bear in mind what what we're seeing here is information cluster-wide as opposed to individual nodes even though we've got three nodes to choose from the result is going to be the same no matter what you pick because to be as efficient as possible that's what's actually going on is we're connecting in and gathering information from the cluster as a whole but I'm getting a breakdown here it's actually telling you about a virtual machines that you're running how they're doing and so on so this to me is a a much much easier way to actually monitor something like proximox ve and it presented in a pretty good format I must admit it's straight away it's right there showing you exactly what your utilizations details are not only just like for the whole cluster but you can go right down to the actual individual virtual machine if you're running containers you can get that information as well so definitely a useful way to actually monitor your proxy cluster well hopefully as you'll see setting up this monitoring for proxbox V is fairly straightforward at the time recording the exporter is still being maintained which is also very good to know but it provides us with a way to monitor a proxbox ve cluster now aside from centralizing all of your devices under one monitoring solution with some additional work you'll be able to set up alerts within Prometheus to notify your problems occur so all in all this is a great monitoring solution to have for proximox4e now if you find this video to be useful then do consider subscribing to the channel as that really mean a lot to me but it's also a good indicator to let me know how videos like this are helpful to people such as yourselves that are watching in which case thank you on the other hand if you're not ready for that level of commitment then I'd really appreciate it if you could press the like button is that where that will help you get the video out to other people that might find it useful as well

Channel: Tech Tutorials - David McKone

Views: 5,557

Keywords: monitor proxmox with prometheus, monitor proxmox with grafana, monitor proxmox with prometheus and grafana, monitoring proxmox, monitor pve, proxmox prometheus grafana dashboard, proxmox monitoring prometheus, proxmox monitoring grafana, proxmox monitoring api, proxmox server monitoring, prometheus, grafana, monitoring

Id: PtsdThgnZqs

Length: 16min 8sec (968 seconds)

Published: Mon Aug 21 2023