InfluxDB + Grafana | Monitor Proxmox Cluster in EASY MODE ! | Proxmox home server series | Home Lab

hello everyone and thank you much for watching this is Mr P and welcome back to another episode in a prox moog's Home Server Series in this video will show you how to set up inflex DB and grafana to monitor your proxos cluster so let's begin before jumping into the tutorial video I would like to go over exactly what I'm going to use for this video as you can see in front of you is a simple proxmox cluster containing three notes I have note one two and three each node runs LXE container difference is that the node one runs lxc container that has Docker contain Docker engine and Docker compose installed where node 2 and node 3 just running a blank lxc container just to show you that they will be picked up by the grafana or inflex DB and inflex DB setup and it will be charted and presented inside the graphs inside the graph the influx DB is the um selfhosted service that you can run self host it or you can go and basically host on in the cloud in in the data centers and the influx DB allows you to capture a lot of data as you can see ingest query store and analyze all types um all types of time serious data metrics events traces in real time with aboundant C cardinality in a single platform and what that means basically influx DB will capture all the telemetrics all the metrics everything from prox Mox and it will display inside the graphs you can set up the graphs inside influx DB to display everything or you can push the graphs push the tetric from influx DB straight into a grafana and grafana will do the rest grafana will present you something like this as you can see in this video in this screenshot a charts and everything showing you what's going on with your pro MOX server another thing I would like to mention that this setup can be done or you can set up for single node for two nodes for three nodes proximal cluster for 5 6 15 20 doesn't matter once you have everything set up you and you add additional node the all the setup all the influx DB plus grafana setup will detect that you have extra node and will capture that data from extra node because and that will happen because the way the setup will have will be done is the influence DB will monitor data Center level it will monitor at the top level regardless how many nodes are available underneath it can be three 30 300 influx DB will capture data from all of them so let's start setting everything up first thing I need to make sure that I have a Docker running inside this Alexi container to check I just need to type Docker and if I get help output it means Docker is running but to be 100% sure that the docker is running I can run a command which is docker space run space hello- world and what that command does it goes and pulls very small and simple Docker image from a hub do from a hub. do.com and run this output as you can see I have a hello from Docker that means that um everything is working Docker engine is running and to check that Docker compos is running I can run Docker compose and press enter and if I get help output it means Docker compose is run so that is all good first thing I need to make start or what what I need to do next is to set up the user that user will be creating the folders and do compos files to run grafana and influx DB to create a user you need to type user add sorry add user not user add add user and give the username in my case it's going to be Mr P press enter as you can see adding user Mr P automatically assigns the group ID and the user ID and now it says new password for this user I'm going to type 1 2 3 uh something simple 1 2 3 4 1 2 3 4 give it all the names and everything press yes and that's it user is being created next thing I need to make sure that user will be able to run pseudo commands to check if a user allowed to run a pseudo command you can type ID space followed by the username in my case as you can see I have uid 1,000 GID 1000 and a groups 1,000 there is no pseudo group assigned to this user to fix that I need to type user mod space- a lowercase a capital G pseudo Mr P enter and now that's been changed to double check if this already take took effect and it's all saved I can run ID Mr P and now as you can see this is the extra number that is been added that we were missing before that means the user belongs to pseudo Group which allows user to run pseudo commands to change to the user I need to type Su space username in my case it's Mr P enter and I get the line saying to run a command as administrator user root use pseudo followed by a command C man sudo root for more details so right now I am logged in as a user to clear the screen by the way to clear the screen inside inside the terminal Linux C you need to press contrl L and now I type CD and Tilda that will take me back to my home directory and in here I will start setting up directories for inflex DB and grafana to set up the folder I need to type MK dur space and give it a name in this case it's going to be influx DB like that enter and I navigate into this folder by running command CD space inflex DB CD stands for change directory enter so I am right now inside influx DB directory in here we need to set up two sub folders first folder will be data so so to create a folder again MK dur data and MK dur config to check if the folder exists I can run l l enter and it gives me this kind of table layout information is showing that inside here there is two folders config and data next we need to create a Docker compos file to create a file I can type Nano space docker-compose.yml enter and in here I will paste this I will paste the output or do compost uh commands that we will use for this video all the commands that I'm using in this video you'll find the link in the description below which will take you to my git repo to that specific place where it's going to be all these do compos files and everything added in for you just to Simply copy and paste what you see in front view is my actual Docker compost file from my main Docker container from my let's say production proxmox so it's a bit different so what I need to do I just need to delete this part let's go like this here we go and delete this and delete this so right now inside the home folder Mr P folder influx DB folder there is a folder called data and it will be mapped to VAR lib inflex db2 on the toer container site and on the host influx config will be mapped to Etc influx db2 I want that to always restart regardless if I'll stop or not I want make sure that this is always running and I leave the ports same as 8086 in case if you already have a container running with the host Port 8086 in use you just change this number on the left do not change anything on the right hand side and everything else stays the same container name I leave it I will leave as monitor influ DB you can change that to anything you want everything is okay I will press contrl X to close and say Yes I want to save this and press enter to confirm that the docker compos file needs to be overwritten enter so that's been done so if I type cat Docker compost file I'll get output of the content of that file and I can see everything is fine to start the docker compon a file and initiate Docker container creation you need to run P sudo Docker compose up space- D that's why we gave a user Mr P uh super user uh group or assigned to Pudo group because it needs to have a pseudo in front to run a do compost file as a user and we're saying that pseudo dock compost check the folder and find this file in there and once you found it run these instructions that's what that means it means I wanted this to be up running and- d means means it detached it means it runs in the background and press enter give it a password for the user that you created enter and right now Docker goes and checks if inflex DB container exist if it doesn't it goes and pulls the container for me and gets everything set up so I'll be back when this is done Docker create a container for us and the last line says creating monitor influx DB done if I type pseudo space actually let me clear the screen by pressing crl L and now I can continue so pseudo Docker space PS I can see that there is a it's badly formatted this way because it's I increased the font size and everything on here for you to clearly to see but pretty much what it does does this is container ID this is the images we're using this is the command that is initiated created 25 seconds ago well it's changed by now state is 24 seconds ago then Port is 8 86 and 886 and then we have a name which is container name monitor influx DB so that is running if I run a Docker PS Docker P sudo Docker PS command again as you can see it says about a minute now so right now we can access this container to access this container I need to type IP space a to find out what IP address this lxc container received from DHCP and my case is this is IP address if you can see your IP address in the list if you type IP space a then vertical line grab and then type first two free numbers of your local IP address in my case is every local IP address starts with 192 so if I press add it will tell me that this is what is found and I can quickly identify that this is my IP address so I need to go to this IP address followed by the port numbers 886 so let's do that I punch the IP address followed by 886 and if I go to the look address I will be greeted with a welcome to influence DB starting page I will press get started username I will say Mr P password needs to be eight characters or more so let's do that and again repeat the password initial organization name you can put an organization name if you want I will do that I will write Mr P and initial bucket I will say for example let's write prox MOX YT is going to be my initial bucket bucket in the Pro influx DB setup means let's say like a categ like a root folder bucket is basically imagine you have um a lot of buckets in the warehouse and each bucket contains a certain things let's say one bucket bucket is for your database another bucket is for your website another bucket is for your I don't know passwords another bucket is for your Docker containers so bucket is more like a place where you dump all the data so I'll press continue and this is next page which is very very important this long line you need to make sure you copy that and keep it somewhere safe and do not show that to anyone if your inflex DB is exposed to internet if you're only using this locally and there is no way to access this then still I suggest this to be hidden from anyone I don't mind to show you this because as with everything that create for this channel for the proxmox Home Server series everything gets wiped after the video is done so you more than welcome to copy this anyway this is still going to be destroyed after video is done so I copied this all off screen into my notes and that's it we can right now go and start setting everything up if I click quick start it's just going to jump straight into a setup process and I can start setting up for example monitor python nodejs go Arduino if I go to the bottom and then click on this icon it expands all of bit so right now let's say low data and this is where all the stuff you can go and fetch to get everything running let's say you can monitor uh Google cloud services you can monitor Linux stuff you can monitor MailChimp you can monitor engine X end stats where what is something else like a wire God so this is the stuff that inflex DB can capture data from so what that means that a free BSD can push data to influx DB and you can start putting graphs and knowing exactly what's have you can ping the stuff it's a lot and a lot of stuff you can do here if I go all the way to a top and type proxmox you might say oh there you go there's a proxo you just go and set this up there is much much easier way to do this than just going and setting user the username password the tokens for the using Etc this is the similar thing that I was showing you when I was giving you a demo about zabic proxmox cluster monitoring where you need to create a user give him a permissions Etc there is much much much faster way if I go to proxmox Cluster setup on my proxmox single node again regardless if you have a proxmox single node or cluster or cluster with 10 nodes this will work I need to click on a data center scroll all the way down and look for metric server I click on that and if I click on ADD as you can see one of the options it says graphite another one is influx DB if I click on influx DB it will tell me to fill this information so let's begin name let's call this influx YT influx DB YT server is the IP address of your inflex DB um container in my case is 192 168 17864 Port is 80 89 by default but as you know ours is 8.86 so I need to go and change that if you set up the container on a different port on the host side you need to make sure that this port is actually here as well and then I will set up HTTP if you have SSL certificates going inside your proxo setup inside your home lab you need to choose https I will use HTTP just for Simplicity of this tutorial organization I need to type Mr P and it needs to be uppercase exactly how I filled when I was filling up the greeting or welcome screen form from inflex DB and a bucket I need to type prox MOX YT making sure that this is match and now there is a token and this is the token that I told you to make sure that you keep it safe and do not show it to anyone this is where you need to paste that token in when everything is in and this looks okay I press create and right now metric server has been created which pushes data from Data Center and everything was underneath which is means in my case it's all free nodes to inflex DB YT name named uh influx DB server located this IP address under this port if I go to my setup here and I just refresh the screen now I should see a bit more stuff showing up here so CPU as you can see pv1 2 and three let's say what's this node it's still data coming in that's fine let's refresh it a bit more it's going to take some time for inflex DB to get all the data in but as you can see I already can drill down to a specific node to see exactly what's going on and get this one going and start showing graphs to set up graphs in influx DB yes you can you can go and start customizing but when you look online for anything that's self host and monitoring recommends to use grafana and that's what we will set up next I'll go back to my proxmox cluster click on the lxc container where I'm running DB go back to my home directory by typing CD space Tilda and right now in this directory I have a folder called inflex DB I will create another folder called grafana navigate into this grafana by typing CD space grafana and inside this folder I need to create I do believe one folder I will quickly Double Check Yes I need to create one folder here which we will call it data enter so inside grafana we will have a folder called data and next we will need to create a dock compos file for the grafana while I'm still in grafana folder I need to type Nano space do- compost. yml enter and I'll paste the docker compost instructions for my main procm setup and we will do a bit of changing here first thing what I need to change is straight away this I need to delete all this and I need to make sure that after grafana I have slash data and everything else pretty much can stay as it is in my inside my main setup so as you can see we're going to run version 3.8 services grafana this is the image we're going to use container name I leave it as monitor grafana you can change to whatever you want if a port 3,000 on a host is already being taken you need to change the number here volumes this is where the grafana will store all his data user 1,00 1,000 you need to specify what user you will use and this is why we created user in the first place or one of the reasons we create a user missp is to make sure that this contain this Docker compost file will run successfully next we're going to have inventory uh environments restart always and then this is just uh from my main setup I like to set up the networks this is not very important for you to keep you can delete these lines if you want I'll keep them here but you're more than welcome to remove them everything looks okay so let's save this by pressing contrl X then y then enter and if I press contrl L to clear the screen and type cat Docker compose yml enter I get an output and everything looks fine to me to initiate this container it's the same as we've done with influx DP I need to type pseudo Docker Das compose up not IP but up- D enter and again Docker checks if a graph image grafana container image exists if it doesn't it goes pulls everything down extracts and gets the container running for me grafana container is running as I get green output saying done everything is fine now I can double check by writing running pseudo Space Docker space PS enter actually let's clear the screen and run the command again and I do see that I have a grafana container running with this ID number and I do have influx DB container running with this ID number so grafana is ending with Port 3 3,000 in my case I can go back to the the influx DB tab just right click and say duplicate the tab then delete everything after the port and just type or after P address just type Port 3000 enter and this is the grafana welcome page the default username and a password will be admin and admin username is the one we specified inside the docker container as you can see here security admin password is admin a username is admin sorry always but the password I chose to be admin as initial setup don't worry once you log in it will ask you to change the password so I'll change the password to something else than just being admin that's done I click submit and that's it we are inside the grafana dashboard there a bunch of stuff you can do here what I will do I just jump straight away into the connections under connections I need to look for influx influx DB I press on influx DB and it tells me how to what influx DB is and how to set it up I'll need to click add new data source in here you can give it a name to this data source inflex DB by default is fine query language I need to change to flux once I change to flux I can continue filling the information your url is going to be HTTP again if you have SSL certificates going inside your home lab you need to set up a https and then provide the certificates for Simplicity of this video I'm just going to use HTTP and I'll enter the full IP address followed by the correct port to allow connection from grafana to inflex DB and I think I yeah I need organization I need to type Mr P and same cap capital letters the same as I set up token this is the one you copied from the welcome screen which I told you that token needs to be always kept secure and the default bucket will be BRX MOX YT SA a test as you can see if I press save and test data source is working free buckets found we can double check if I click on this click on the low data buckets and if I just zoom out a bit I have one two and three that's it graph detected that the influ DB connection is successful and it's found free buckets click again just to make sure everything is in and that's it right now graph pulling the data from inflex DB and inflex DB pulling data from portina not portina sorry proxmox I click on this hamburger menu I can actually click on this and always be open I click on a dashboard and this is where we need to create a new dashboard the way you can create dashboards is there's a three different ways you can create a new dashboard create sorry there is up two different ways you can create a new dashboard by importing or setting up manually or importing straight away which is going to say import and actually let's go back I'll show you what's going to happen if you click new dashboard you can start adding visuals yourself if you created a visual or graphs or charts that you want to use for another project another monitoring service you can add this from a library or you can import dashboard so this button does exactly the same what happens if you click on this and right now you can import the Json model for the dashboard that you want to use or you need to get the dashboard URL oid if you go to gra Labs you will find in here A bunch and a bunch of stuff for the dashboard I'll show you just quickly I'll click on this link all dashboards and this is a bunch of dashboards that you can use for various things you want to monitor for example you want to monitor AWS there's a dashboards for that somebody other people created the dashboards and they uploaded the ad Json code into graph labs and they provided unique ID number which you just copy and paste and you can use those dashboards in a matter of seconds for example AWS metrics this is you want to look like this or maybe you want to look like this or want to look like this we will use proxmox flux I'll leave a link to this inside the description as you can see says proxmox flux cluster but it will detect one node and it will show you fine no problems so I'm going to click on copy ID to clipboard click on this right click paste load and it's detected that is being pulled down the latest update to this was as you can see updated last year more than a year ago that's fine it still works name instead of saying prox mlux cluster I will name this prox roxmo dyt folder is going to be default folder dashboard that's fine and your data source I need to click on this and need to make sure that influx DB is being selected select that and click import and right now let's wait for a second or so to start everything going first thing if you leave this like that and you're going to say okay nothing is happening something is wrong at the top you need to make sure that filtering is being selected correctly data source influx DB that's fine bucket I need to make sure Pro MOX YT is selected and server I need to make sure that everything is selected as you can see now stuff's happening on the screen and that's it the data is being pulled from inflex DB and being presented in graphs charts bar charts inside the grafana and as you can see across my proximal cluster local directory 64% used Z Pool ZFS pool is 1% used in total I'm using 8 GB of memory total memory will show up after some time just leave that running for a minute or two and you will see here hi it's Mr P from a future one thing I forgot to mention this total memory problem this will always say no data because if you chose the inflex DB bucket to be anything else than proxmox this will always say no data to fix this you need to Mouse over this block click on the three dots and choose edit and as you can see here from bucket proxmox you need to make sure that this is matches the exactly bucket name that you gave to inflex DB in my case it's proxmox YT and now if I click apply the total memory is working right Mr P is back to you in total across my prox Mo cluster of free noes I have 12 logical course swap total I can see the um iops I can see memory who's using the most memory at this time it was pv1 out of all the My prox MOX noes this PVE one was using the most because this is the one that doing a heavy lifting as other the two proxmox proxmox knows just running container with no activities on there what else we can see here so if I go on here I can see that monitor wayt out of Ubuntu alx1 Ubuntu lxc two and monitor YT uses the most amount of memory as you can see jumping in where these two just chilling with 5% it's still showing no data here we just can refresh and the data is going to be populated a bit more at the top you can select between 5 minutes to show the last 5 years or more I will use for example let's do last 5 minutes like that under here I can choose how often this dashboard refreshes every 5 seconds so every 5 minutes oh sorry every 5 seconds I want this dashboard to refresh and the dashboard will show the last 5 minutes if I'm happy with this setup I can click on this dis to say this current setup and I click save which means that every time I open this by default it's going to be 5 minutes and it's going to be 5 seconds if I click on this star it's going to be added into the most favorite dashboards if I click on a hamburger menu expanded this a bit say click home as you can see on the start start I can see prox MOX YT I press on that and right now I can see this one straight away showing if you're running let's say a ky I have a a separate computer Raspberry Pi doesn't matter what with the attached display you can run grafana in a kiosk mode which means like you can go to that let's say Raspberry Pi with a tached screen you enter this URL on that on that Raspberry Pi you click on this triangle going up then on the computer and that's it this is right now in a kiosk mode and it will run constantly as long as the browser is working and there's a connection of to that Raspberry Pi you will see all the graph showing up I'm just going to press escape to exit the kiosk mode and that's it this is this setup is done I can see okay monitor YT is really going up CPU usage going up a bit the memory uses as you can see is using half you can go straight away to prox Mo and let's see how he doing okay so the actual monitor YT how how is doing so it's using half of the memory I gave only 512 megabytes so it uses just a half I can see proxmox is being busy let's say click on a Data Center and summary let's scroll down a bit let's see as you can see these two just using just over 60% of memory where pv1 uses 821 that is been reflected inside my graph setup and again all these as you can see around okay BMS is now running so obviously that's why it's not going to show anything lxc traffic in lxc traffic out and Etc and once you let's say you have this running for a week or so there's a couple of nifty things like say for example if I drag a mouse and select it's going to jump straight into that location as you can see here it was I selected between 5 minutes past 700 p.m. and 6 minutes past 7 p.m. and it's showing this I can go back to five minutes or if I'm for example there's a bunch of stuff happening on the screen I'm very interested in exactly what pb1 is doing I click on that other two is been deselected and showing only this I click on next showing this I can press control click to include two or all of them or just one and that's it very is simple to set up influx DB and grafana monitoring system to monitor your proxo server if you found this video very helpful I would appreciate if you click subscribe or just please click like that will be fine as well and like always I will see you in the next video goodbye

Channel: MRP

Views: 2,669

Keywords: grafana, grafana docker compose, how to setup grafana, how to setup influxdb, influxdb, influxdb docker compose, monitor proxmox cluster, mrp, proxmox home lab, proxmox home server series, selfhost grafana, selfhost influxdb

Id: 5dEO1QxZJiI

Length: 30min 38sec (1838 seconds)

Published: Mon Jun 03 2024