Best Server Monitoring with Prometheus and Grafana using Node Exporter and cAdvisor

well if you guys watch my 2023 home lab goals video that I just posted a few days ago you will note that one of the goals that I had in the home lab for 2023 is to get my hands on more free and open source Monitoring Solutions there are so many that I want to try out and get a better feel for observability in the home lab however a proper solution that is a fantastic solution for open source monitoring of your servers in your home lab is a solution built off of well-known free and open source utilities and those are Prometheus node exporter grafana and C advisor each of those components fit together perfectly to make a solution that is phenomenal when it comes to getting metrics from your Linux servers Windows servers and your Docker containers stick around as we create perhaps the best free and open Source server monitoring solution for Linux windows and Docker containers let's talk about how the pieces to this monitoring puzzle fit together it sounds complicated but it's actually not too bad Prometheus is a really great monitoring solution that allows you to scrape logs and other statistics from remote servers node exporter is a complementary piece of software and is an application that you install on your remote server and then it exposes all of those key performance indicators and metrics for Prometheus to grab those and pull those in for monitoring purposes then we can take all of those metrics and we can feed those into the grafana solution it's an excellent solution that allows us to create beautiful dashboards based off of that monitoring and metrics information so we can essentially create great looking dashboards that allow us to visualize all of those metrics that we're pulling from those remote hosts now we mentioned a piece of software called node exporter node exporter is written for Linux environment so you can't load that on a Windows Server however this project has been ported over for Windows servers and that is known as Windows Explorer so using Windows Explorer we can expose all of those same metrics and many others in the realm of Microsoft services and have those exposed for Prometheus to grab all of those metrics and feed those into our grafana dashboards the last piece of the puzzle our Docker containers we can use a solution called C advisor now this is a Google project that is free and open source and it essentially does the same thing as node exporter except it exposes all of the key metrics for our Docker container environment we can create a Docker compose file that will spin up all of the required containers and I'm want to step you guys through this step by step and you can even copy and paste the code to your environment and easily stand up the required containers as you can see I am remoted into the Linux host that I'm going to be loading the monitoring solution onto now this host is a simple Ubuntu 2204 virtual machine and it has a clean installation all the updates have been ran that is the platform that I'm going to be using I have already installed Docker onto the host so if you don't have a Docker host that's going to be a requirement so the next thing we're going to do is we're going to install Docker compose we're going to use Docker compose to spin up the docker containers that we need for the monitoring solution so I'm going to install this from the app repository and we're just simply going to install Docker compose so now that we have Docker compose installed we can verify that by just simply issue issuing a Docker compose dash dash help everything looks good there we have installed Docker compose so now we are ready to begin the installation one other thing I want to call out is that I have a Linux user created that is a sudo User it's a Linux admin as you can see the ID of the user is one zero zero one and that's going to be important as we look at the docker compose file we need to make a quick change in our Prometheus configuration if you're using a non-root user even a sudo user this user is a sudo user again we've added it to sudo users group this is the ID and we're going to move forward with this user being our Docker user that is not root so now what we need to do is create our local folder structure to house our local volume mounts for the monitoring solution so I'm going to create folders for Prometheus and grafana for the docker containers to have those persistent volumes mounted one of the easy ways that I have to do this consistently especially in the home lab environment is I simply have the commands chained together in a very crude bash script that will allow me to create the directory structure that I need so I'm going to just paste this in and show you guys what I'm doing here so in my home directory I am creating the parent folder and I'm just kind of abbreviating all of the Technologies or the three major Technologies here and then I'm creating a Prometheus folder I'm also creating a grafana folder and a child folder called provisioning that is used in our Docker compose file then what I'm doing is I'm creating the shell of the docker composed yaml file as of yet it doesn't have any contents but we're simply once again we're getting the file created also the Prometheus dot yaml file which we need to configure to have our scrape configurations for our monitoring solution going to do is just run the command and no worries about the command not found that we see there in the output we do a LS we can see that the directory has been created and we can go into the directory we can see the subdirectories have been created and go back out and check our Prometheus directory we can see the prometheus.yaml file has been created so we see that our directory structure is indeed looking good we've got the directory structure that we are going to reference and use in our Docker compose file which we're going to take a look at next now you may be wondering how do we install and configure all of those different solutions because we named Prometheus node exporter rafana and then also C advisor so how do we get all of those things spun up and talking to one another correctly well that is a simple process using Docker compose don't worry I've got you guys covered on that front we just simply create a Docker compose file that spins up all of the related containers and then we essentially configure Prometheus with our Prometheus configuration file to point that at all of the remote hosts that we want to monitor so let's step through this process of configuring the containers configuring Prometheus and do that step by step along with creating the directory structure as you remember we created the docker compose.yaml file so that is simply in the root of the folder that we created for the monitoring solution let's edit this file which should be blank at the moment so we just have a blank Docker Dash compose.yaml file what I've done is just simply paste it in the configuration for our Docker compose file and let's just step through this and see what the configuration looks like for all of the containers first of all we are creating the networking mode in bridge mode we have volumes for Prometheus data in our services configuration here is where we actually Define the docker containers that are going to be part of the monitoring solution here we have the node exporter container and this again the node exporter container is the solution that exposes the metrics for our local Docker container host we can install node exporter on our remote host as well however this container will simply expose the metrics for our local Docker container host Hardware so that is the purpose of spinning up the node exporter container and here of course we are exposing the networking ports we are using the monitoring Docker Network that we we are creating with this Docker compose file next we have Prometheus we're pulling the Prometheus image if you recall I showed you guys the user that I was using for the configuration of Docker and the user that I'm currently logged in as it was important to find the ID of that particular user as when I tried this initially when I did not specify the user Prometheus had permissions issues when trying to create the data inside the container from the mounted directory that we created that is what the stanzas here are for user and then I have the environment stanza underneath the Prometheus configuration for the user ID as well as the group ID container name and then here is where we are mounting the directories that we created using the very crude bash script to just get those in place so that we don't have to create those manually and as we can see part of that is exposed using this prometheus.yaml file to the container and I'm going to show you guys what's in that configuration file as well we scroll on down we can see the networking ports that we are exposing now we get to grafana same thing for the user account here I'm also designating the ports mounting the volume that we created as well to the Etsy grafana directory inside the container and the same thing here we have it connected to the docker monitoring Network that we created next we have the C advisor container you'll notice one Oddity here I have 8081 translating to 8080 and that was simply because I had another Docker container running on 8080 so you can leave this 8080 or change this to 8080 or any other port for that matter to translate into Port 8080. as you can see we're exposing some volumes see advisor depends on the redis database container so I've got redis container just simply copied and pasted this configuration from the official documentation for C advisor so we're just simply pulling the latest redis container again exposing those ports connected to the monitoring Docker Network so now that we have this configuration what I'm simply going to do is save the configuration for the docker compose yaml file I'm going to go back in make sure those changes saved which they did so now we have that directory structure configured with the docker compose dot yaml file okay now that we have our Docker compose file ready to go we can simply issue the command Docker compose up Dash D and it will read the configuration from our Docker compose yaml file in the same directory where we are running the docker compose up command so let's run our Command as we can see it's creating the network the volumes and it's also pulling all of the relevant containers contained in our configuration file okay everything shows to be done so what we can do is issue a Docker compose PS command and we can take a look at all of the containers from the docker compose creation process as we can see C advisor is showing health of starting everything is showing up so we can continue to take a look at the containers we can also do a Docker PS and make sure we don't have anything restarting which it doesn't look like we do that's one thing to just make a note of if something is wrong with your Prometheus configuration or your grafana configuration you will see those containers restarting if you do see an issue with one of your Docker containers you can use your Docker logs command and look at a particular container and you should see the specific error message relates to that container again we're going to look at the docker compose PS and we have a healthy designation here so that's awesome all of our containers look to be functioning correctly so now we can proceed forward with the configuration of Prometheus okay awesome we have our containerized solution of all of those monitoring components working together in a seamless way so how do we get these metrics into grafana to then be able to query that information as well as create those beautiful dashboards that all of us are looking for now that we have provisioned all of the required containers we need to configure Prometheus for monitoring the solutions that we want to monitor as you recall we created the Prometheus directory and inside the Prometheus directory we have a Prometheus dot yaml file the prometheus.yaml file contains the configuration that tells Prometheus which nodes that we are going to monitor and as you can see the configuration is currently blank so let's actually add some configuration to our Prometheus this yaml file it's just a simple configuration file that defines the scrape interval as well as under the scrape config stanza you are telling Prometheus essentially which nodes that you want to monitor here we have our local node exporter container that we're going to be pulling metrics from as well as the C advisor container save our changes and now we want to stop and restart our Prometheus container so we're going to start our Prometheus container we're going to make sure that it's not crashing for whatever reason which everything looks good at this point so now we have properly configured our Prometheus insulation with the configuration needed to stream those metrics into grafana which is our next step we should now be able to browse out to Port 3000 on our Docker container host and see the grafana login so the default login is admin admin going to ask us to change the password and we are now successfully logged into grafana for the first time to point grafana to our Prometheus data source we navigate to the cogor settings icon in the lower left hand corner of the grafana interface we're going to choose data sources here we're going to choose add data source and the first one available is Prometheus so let's click Prometheus now that we've clicked Prometheus we see the example of the address that we need to place into the URL configuration here I'm going to populate that with the IP address of my Prometheus installation so I have that be sure to add your port and scroll all the way down and we should be able to save and test and have the data source show to be successful once we have our data source configured we can browse to the explore dashboard icon so if we click explore as you can see it's already pulling all of these metrics in the query Builder so we could essentially go out and we could build individual queries to pull information from these particular metrics but I'm going to show you guys one of the things that we're going to do is we're going to Simply import one of the community dashboards if you click the dashboards icon you have the option to import and on the import dashboard dialog box you can see that we can either upload a Json file or we can paste in the ID of a community dashboard now where do we get the Json file or the ID you can download those from the grafana community dashboards site if you type in the search dashboards field node exporter we're going to be able to find many different Community dashboards that have already been created this is a great one the node exporter full and then there was another one that I really liked as well it was node exporter full with node name so if you click on that dashboard or any of the others you will notice that you can either download the Json or you can copy the ID to your clipboard which is what I'm going to do it's just the easiest way to do it we hop back over to our tab for drafana we're going to paste in the ID of the community grafana dashboard and we're going to click the load button when we click the load button you can you can customize the name of the dashboard choose a folder if you want to house this particular dashboard under a particular folder be sure to choose your Prometheus instance so I'm going to click the drop down I'm going to click the search and choose the Prometheus default and then we're simply going to click import immediately you automatically see the really great looking dashboard that's being pulled from the node exporter metrics that are being pulled on the Fly and as you can see we can choose the window of time that we want to take a look at if I choose this to 15 minutes it will give you more granular resolution of those metrics and as you can see when you scroll down there is a wealth of information that you can see as part of the metrics pulled from the node exporter so I've shown you guys how to do this in a Linux environment so now let's install the windows exporter to allow us to then pull those key metrics from our Windows host and get those into Prometheus as well as into grafana if you Google windows exporter and then you navigate to the releases link you can scroll down to to find the latest release under assets you want to grab the MSI file so I'm going to download that next we just need to copy the MSI file out to our Windows host which I'm doing now now let's simply run the MSI file Windows exporter requires no configuration so if we just verify if we look at the services console we should see the windows exporter service now installed and running so if we scroll down we can see Windows exporter is running it's set to automatic if we go back out to our container host we need to edit our Prometheus configuration and we need to add a job name to the Prometheus configuration for the Windows host I'm going to paste in the configuration needed for the Windows host where we just installed the windows exporter service so as you can see I've got the network address it defaults on the Windows exporter to Port 9182 that can be customized during the installation of the windows exporter package so we've got our changes made let's save and exit and we need to restart Prometheus foreign now we can find a Windows exporter dashboard so if we go back to our community dashboards and search for Windows exporter we're going to see many options presented to us I want to try out this Windows exporter node it's got a 5 out of five for two ratings so let's check it out so we can copy the ID to the clipboard let's go back to grafana and we're going to go back to our dashboards and then we're going to import a new dashboard foreign ER for the name we're leaving the default folder let's choose Prometheus and import automatically we see that now from the Windows host where we just installed the windows exporter we're already getting some really awesome information CPU usage we can see how long the server has been up we can see the addition of Windows the hostname the ram CPU and if we scroll down we can see all kinds of information from Windows processes Network information CPU usage and other information about our Windows hosts including our detailed look at our disk space so that is awesome so now we have both Linux and windows monitored using Prometheus and grafana now let's look at monitoring our Docker container hosts and all of the relevant Docker container information using Prometheus and displaying that in grafana for monitoring Docker we really already have everything in place and the reason for that is when we initially installed the docker compose stack we deployed C advisor C advisor is short for container advisor and it's the solution that you want to deploy if you want to monitor relevant Docker statistics so we already have deployed that the only remaining step is to navigate out to the community dashboards page and search for a Docker dashboard that you want to use and this was one of the free dashboards that I have just downloaded using the same process that we have already looked at with the other two solutions so as you can see it shows running containers Network traffic memory CPU and many other statistics so awesome information so guys what do you think about this awesome free and open source monitoring solution hopefully you guys have enjoyed this walkthrough and this is something that you guys can just simply clone down my git repository for this free and open source monitoring solution that you can literally clone into your home lab change a couple of directories as we mentioned in the docker compose file and then you're up and running with little to no configuration and the thing I like about this is all of these components are free and open source so if you're looking to steer away from any kind of Enterprise or paid solution this is a great way to do that well Brandon Lee I hope you guys have enjoyed this video and the free and open source monitoring solution that we've covered please do like the video subscribe to the channel keep on home labbing stay safe out there guys and I hope to see you all very soon [Music] foreign [Music]

Channel: VirtualizationHowto

Views: 49,132

Keywords:

Id: RAqMP_NnGec

Length: 23min 56sec (1436 seconds)

Published: Fri Jan 27 2023