Getting started with Prometheus Grafana and Node exporter - Part 1

hello there this is rahul and in today's session we are gonna talk about prometheus and grafana well this is the first part of the series and where we will focus on how to install prometheus server grafana server and node exporter if you are new to the prometheus and grafana setup and you are just getting started with the prometheus and grafana then i would highly recommend to watch this session where we will start on virtual machine where we haven't installed anything before so we will first install prometheus server then we are going to install node exporter and then we are going to install grafana server so these are the basic steps which we will be targeting and after that we are also going to see like how to configure a grafana dashboard by creating a prometheus data source so this is going to be our first session where we will be focusing on these points i'll also put the time stamp into the description section so if you are interested in some specific topic then just click on those time stamp let's switch over to our desktop and start with our prometheus and grafana setup to start with on the left hand side you can see this is the terminal and this is my virtual instance where i will be installing the prometheus server along with the node exporter as well as the grafana and on the right hand side this is the guide which i have prepared and you will find all the instructions along with the command into this guide so i'll put the link of this guide into the description section so just feel free to use those command while you're performing or you're trying to set up your prometheus server along with the grafana let's start from the step number one and the step number one is installation of a promethean so if you follow the instruction then here there is a download link and this is the official download page from the prometheus and here we need to first download the prometheus binary since i'm using the linux operating system that's why i'm just gonna use the linux distribution or the linux binary for installing my prometheus so here just copy the link from here go to your terminal and here first of all i'm just going to create a directory and go to in that directory and simply run the command w get and paste the url and it should download the prometheus binary onto your local system okay now that's been done in case if you are using windows operating system then you can download this windows zip you can extract the windows zip and then you can install the prometheus onto your windows machine so just use this zip file for installing onto the windows operating system but for this tutorial we are going to use the linux or the ubuntu as our base operating system let's extract the file and the command which we are going to use is tar xv fsat and after that we are going to enter the name of the file and simply hit enter and i'll clear the screen we can verify by using the command ls lrt so that we can verify the content and this is the extracted directory i'll go into that directory and simply again we can run the command ll or ls lirt and here you can see these are the content which we got after extracting the prometheus zip file let's run the prometheus server and the command for running the prometheus server is down and then forward slash and after that you just simply need to type the prometheus so this is the binary which we need to execute to run the prometheus server and simply hit enter and this is the command which i have mentioned over here in the guide so you can just copy this command to start your prometheus server and by default this prometheus server will start on the port 9090 and since i am using the virtual machine so i need to provide the virtual machine ip so here you can see this is the address where our prometheus server has started and this is the virtual machine which i am which i am using so the virtual machine ip is 192 168 56.1 in case you are installing without any virtual machine and you are installing on to your local system then instead of using this ip you should use localhost but let's go back to the browser paste the ip address and after that you can enter the port number and here you can see this is the first ui which you will get after installing the prometheus server let's have a look on the ui of a prometheus server so this is the default ui which you get after installing the prometheus server and here you can see this is the expression bar where you can enter the prom ql which is the prometheus query language and to give an example if you simply type up over here and click execute then it will let you know all the available or the active host on which the prometheus is running so right now we have installed it on a local host that's why it is showing localhost 1990. in case if it is connected with a different remote server then it will show you all the list of those remote server over here so this is just an example how you are going to use the this prom ql structured query language over here the another thing over here if you go into the status section and if you click on the targets then it is also going to show you like how many uh nodes or the how many remote server prometheus server is scrapping so as of now we have installed the prometheus on the localhost that's why we are showing it is getting showing us only one host and if you are going to add further more resources which we are going to do into the later section so here the list will keep on increasing so this is the very basic ui where you can interact with your prometheus server if you want to see the matrix then you can simply change the url over here instead of graph you can simply type the matrix and hit enter then here you can see all the metrics which is like a scrap via prometheus server now after installing the prometheus server the next thing which we are going to install is the node exporter but before we install the node exporter let's try to understand what is node exporter is so this diagram shows a relationship between the prometheus server on the left hand side you can see this is my prometheus server and on the right hand side these are my remote server and where i have installed or i am going to install the node exporter so prometheus server is just like a collector it collects all the data which is like send by a node exporter so till now if you look at here then on the left hand side we have just installed the prometheus server and that is up and running now what we are going to do we are going to install the node exporter so the first of all we need to have an another host or another server where we can install the node exporter so uh for this tutorial we are going to install node exporter onto server the first server would be a prometheus itself so first of all we have installed prometheus server and along with that we are going to install the node exporter on this system also and secondly we are going to start one more virtual machine and on that virtual machine we are gonna install the node exporter so this prometheus server will collect the data from its own host that is localhost because we have start we have installed the node exporter and then it is also going to collect from the remote machines so there is no limit on installing or to connecting prometheus server with other node exporters so you can have any number of instances running into your infrastructure so here you can see there are three machines so you need to install the node exporter on three machines and then you need to tell prometheus server to collect the data from these three hosts so that we are going to see into the next step all right so what i have did now on the this terminal as you can see this is the process which is running so this is the process for starting our prometheus server and that is up and running so you can see over here if i refresh this page then it's showing us the metrics and on the bottom i have again opened a new terminal and this terminal is still connected to the same prometheus server as you can see over here and that you can verify from here also so here you can see so both the terminal are connecting to the same server that is prometheus server and now i'm gonna install the node exporter and for that what you need to do is uh go to the step number 2.1 and here there is a download link for node exporter so click on it open it and here you will find the installation packages or binaries which you need to download since i'm using the linux operating system so i'm just going to use this linux amd64 as a file copy the link from here and i'm just gonna check okay i'm just need to create one more directory so i'm just gonna create a directory mkdir node exporter switch to that directory clear the screen uh type the command w get paste the url and hit enter all right so now we have downloaded the node exporter onto our prometheus server let's extract the zip file i'll clear the screen i'm just gonna paste the command tar xv fc and after that we need to type in the zip file and extract it and again we can verify the content by running the command ls and here we can see this is the extracted directory go into that directory i'll clear the screen and i can run the command again ls and here you can see this is the binary of node exporter which we need to run to start our node exporter onto prometheus server let's start the node exporter and the command which i'm going to use it dot forward slash and then the node exporter and simply hit enter and it should start the node exporter to verify it uh it should be started on the port 9100 and you should use the host ip since i'm using the virtual machine that's why i'm just gonna use the virtual machine ip address so this is my virtual machine ip address and in case you are installing it on your local machine without using any virtual machine then you simply need to type localhost and after that you need to specify the port that is 9100 but i'm using the virtual machine that's why i'm just i'm just not gonna use the local host but instead i'm just gonna use the actual ip address of my virtual machine which we have assigned so this is the ui which you're gonna see and if you click on the matrix then this here you can see all the matrix which is like a scrapped from our prometheus server using the node exporter so this is the first verification which you need to do after installing the node exporter so till now we have accomplished two tasks one is we have installed the prometheus server which you can see over here which is running on a 9090 port and if you replace it with the graph then you can see this is the graph and this is also running on the same host as i told you and this is my uh node exporter matrix ui so here you can see all the matrix which is like a scrap from our prometheus server there is one important concept which i would like to mention over here is like here on the left hand side you can see this is our prometheus server which we have installed and we have also installed the node exporter on the same prometheus server but this node exporter and this prometheus server still don't know each other so we need to make a communication or the link between the node exporter and the prometheus server and that's why we are going to create exporter config.yml and this is a configuration yml which tells like which node exporter our prometheus server is going to use so in this exporter config.yml we are gonna mention uh like a local host because we just want to use the node exporter on the prometheus server so we are gonna add the entry of our local host into our exporter config.yml so that our prometheus server knows like this is the node exporter i am gonna use it so in case if you wanna add one more server so let's say this is our another server with the ip address 192.168.56.2 so we need to add this entry into the exporter config.yml also so and this server will also have a node exporter installed on it so what is happening over here is we are sending the matrix of our localhost prometheus server as well as we are sending the matrix of our remote machine that is uh located on 192.168.56.2 and these entries you will find into exporter config.yml i'm just going to show you the exporter config.yml in a short so this is the exporterconfig.yml which i was talking about and here the first parameter is the scrap interval that is 15 second so this scrap intervals tells prometheus server to scrap a remote machine every after 15 second the next thing is the scrap config and the scrap config there are one more flag that is targets in the targets we need to mention the remote machine hostname or the ip address since we are doing this for a tutorial purpose that's why i have used the localhost 9100 that means we just want to scrap our own prometheus server that's why i'm using the local host and also we are gonna scrap a remote machine uh which is located on the ip192 168 56.2 and the port is 9100 because we have installed the node exporter on that remote machine also now we are going to do that step later into this session but this is just an uh just to give you an idea so here in the target section you need to replace uh the host name or the ip address based on your requirement and the situation so this is how exporter config ml looks like all right let's create the exporter config.yml and here on the left hand side you can see on the top we are running the prometheus server and on the bottom we are running the node exporter so what i'll do i'll just stop the prometheus server and after that i'll clear the screen and let's create the file first so i'm just going to use the touch command and the name of the file is exporter config dot yml and we can verify using the command lslart and this is the file which we have created let's do one thing this path is too long so i'm just gonna reduce it by using the command so let's put dollar sign over here and hit enter okay i'll clear it so now i have reduce i have just removed the complete path it was getting too long anyway so now we have the exporter config.yml created but the file is empty so i'm just gonna copy this configuration from here open the exporter config in the vi mode and go into the edit mode paste the content and one thing which you need to do is we need to change the ip address because we are not running on 100.0.1 so i'll just go over here and i'll just change the ip or i can remove the ip because we have not started the remote machine yet so i'm just gonna simply remove it i'm just on i'm just gonna simply run it on a local host because we have not started the virtual machine yet so save and quit okay so now what you have seen over here we have created the exporter config.yml and we can verify the content of it once again and yeah and it is scrapping the target localhost that's fine i'll clear it and if you go to the guide now what we need to do we need to tell the prometheus to use this config file and to do that what you need to do is you need to start the prometheus server the way we did previously apart from that you need to specify the flag that is config file and then the file name so i'm just going to copy this command over here paste it over here and here i'm just gonna specify this export config.yml so that our promethy server knows like which host it needs to scrap and simply hit enter and this is how it is going to read that remote configuration of a node exporter and here you can see this is the file name which is which our prometheus server is using for scrapping those host later on we are gonna start a one more machine and in that we are gonna install the node exporter and then we are gonna edit or add the entry of that remote machine ip address over here so that we are gonna do a little bit later let's verify our scrap configuration if you go to our prometheus server this graph url i'll just remove once again and hit enter and here you need to go on to the targets and if you it might be possible that it is not showing up so just click on show more and here you can see this is the host which we are trying to scrap that is our local host the status is up and here you will find the information like scrap duration that is 15 second that we have configured into our scrap configuration yml and the last configuration and last scrap which has happened that is 18 seconds ago so this is the uh like a ui of our server prometheus server which tells like how the scrapping is going on and when was the last time we have scrapped our host machine so from here you can verify our scrap configuration which we have applied now what we are going to do we are going to start a one more machine so we are going to start one more remote machine instead of local host and we are going to add a node configuration for that remote machine also so we are going to see one more entry over here okay so this is my prometheus client remote machine and if i can show you the ip address so this is the ip address of my remote machine 192 168 56.2 so i have just started another virtual machine with a different ip address and this is uh here you can see the multiple uh terminals so on this terminal i'm running my prometheus server and this is my node exporter running over here and this is my prometheus server with my exporter config.yml so here you can see this is the exporter configure yml which i have started and here i have started a one more virtual machine and that is our prometheus client and on this machine i'm just gonna install the node exporter first this is a very simple virtual machine right now and i have not installed anything so what i'm gonna do i'm just gonna go back over here uh yep here click on the node exporter copy the url and i'm just going to create a directory node exporter cd node exporter wget and paste the command clear it and then we need to extract it so i'm just gonna copy the command paste it over here and node i'm just gonna extract it go to the directory yep and i'm just gonna start the node exporter over here also and hit enter so now we have started the node exporter on our remote machine and we can verify uh using the ip so i'm just gonna simply copy this ip open a new tab put two and then the port two zero zero and here you can see our matrix of our node exporter so now we have installed the node exporter on our remote machine but what we need to do we need to tell our prometheus server so i will go back to my first terminal so this is my prometheus server over here which is running i'm just gonna stop it i'll clear the screen and verify our exporter config so here you can see we have only added a local host we have not added a remote entry of our remote machine so if you go back to our url 192.168.9090 oh i have start i have stopped my prometheus server but let's start it again okay so i have started it back again and if you go into the targets then here you can see the there is only one host which is showing up i have just started so that's why the status is unknown but once i'll add that entry uh i mean to say when once i add the entry of my remote machine then there will be another host showing up over here and here it has gone to up okay so what i'm gonna do i'm just gonna kill the prometheus server again and i'll clear the screen and i'm just gonna open into vi mode the exporter config and go into edit mode and add the entry and i'll just copy the address from here from here copy it go over here and paste it over here all right after that i'm just gonna save and quit now one thing which i realize over here while editing my external config i don't need to put http i just need to put the ip address or the hostname and the port number so i'm just gonna remove the http and i'm just gonna save and quit and after that i'll clear the screen and just restart our prometheus server and now we have started our prometheus server i'm just going to refresh my prometheus url and starting up i'll go to status go to target i just wait for a second or so and here you can see now as i mentioned previously we have not added any remote host that's why there were only one entry which is of localhost but now we have added one more host and now we can see the entry over here so we have just started the server that's why the status is unknown i'll just refresh it once or more and here you can see the status is up for both the host and now we are scrapping both localhost as well as our remote machine and here the instances here you can see the verify the ip address over and also the last scrap and the scrap duration so this is how you configure a remote host like here this is the remote host which is running and where we have installed the node scrapper and here we are having the prometheus which is running with our node export exporter config.yml which contains the entry of both the host so this is how you are gonna configure the remote node exporter or the remote host node exporter onto your prometheus server now if you look back at this picture then we have set up a prometheus server we have set up a node exporter we have set up the exporter config.yml the missing piece into this diagram is the grafana so now we have set up the whole infrastructure ready for our grafana dashboard and we have the data which is being scrapped from our local host that is prometheus server as well as from a remote machine and now we are scrapping all those uh data into our prometheus server now the next task is we need to visualize the information in the form of a dashboard and for that we are gonna use the grafana so the next thing what we are gonna do over here is we need to install the grafana and this grafana is going to use the data source the data source which we are going to create it later and that data source is going to be a prometheus data source so here you can see this is our prometheus server and later on we are going to create a data source which will connect this prometheus server with our grafana dashboard okay to install the grafana this is our step number third and here we have the instructions for installing the grafana on the left hand side i have opened the one more terminal and i'm again connected to the same prometheus server so here you can see there are a couple of terminals which i have opened previously so this is our prometheus server which is up and running you can see over here this is our node exporter which is also running on the same prometheus server and if we go to the second terminal then this is our node exporter of our remote machine that is also running and then i have opened one more terminal just to keep the thing simple so now i am connected on a prometheus server and here i'm just gonna install my grafana uh server also so that we can visualize the uh visualize everything onto the dashboard so the first instruction which we are gonna follow we are just gonna update the package info so copy this command from here go to terminal and paste it over here [Music] and also if you want to follow the official link so these instructions i have copied from this official link of grafana so just click on this link if you if none of the instructions are working so you can get the exact copy of the same instruction from the official guide i'm just gonna copy the second one i'll clear the screen i'll just paste the command okay that's been done i'll run the fast final command i'll clear it paste it over here okay that's been done now i will add the stable repository of a grafana into my ubuntu and for that i'm just gonna copy it i'll clear the screen paste it over here hit enter and that's been done and now what i need to do is i need to update my repository information because i have recently added a grafana repository so i'll just do sudo apt-gate update paste it over here hit enter and that's been done let's install the grafana copy this command go back to terminal clear the screen and paste the installation command type y for installation okay so now the grafana server has been installed after installing the grafana server we are going to start the grafana service and for that we are going to use this command to start our grafana server so we will start from the first command i'll clear the screen over here i have already copied the command i'm just gonna paste it over here after that we are gonna start the grafana server paste it over here and then i'm just gonna check the status of my grafana server service and paste the command over here and here you can see our grafana server has been started and the status is active and running one more thing if you wanna start the grafana server on a boot time that means once you restart your ubuntu or linux machine and you want the grafana service to be up and running then you can simply execute this command so whenever you are going to restart your machine then you don't need to manually start your grafana server it will start automatically so simply run this command and that will take care of a grafana start during the boot time i'll clear the screen the next question is how to access the grafana dashboard so as you can see over here i have mentioned the url to be localhost 3000 but since i'm working on a virtual machine that's why i'm not going to use the localhost bus instead i'm just going to use the ip address of my virtual machine and in case if you have installed the grafana on your local machine without using any virtual machine then simply you just go you can use this url localhost 3000 and it should open you the login page of grafana so let's open the new tab over here and paste the url so this is the ip of my virtual machine and remember this is the same ip where i have installed the prometheus server also but here i'm just going to use the port 3000 because that's where i'm i will be able to access my grafana dashboard so simply hit enter and here you can see this is the default or this is the login page of my grafana so this is the first time i am using the grafana that's why i'm just going to use the default username and password and that is admin admin and since it's the first time we are using it then it's gonna ask for new password and confirm new password so i'm just gonna type the new password and then submit i'm just not gonna save it i'm just gonna click over here so as you can see this is the default uh dashboard of our grafana so that means if you are able to see this dashboard then that means you have successfully installed grafana onto your linux machine or windows machine whatever operating system you are using moving back to the guide now we have set up the grafana server and we are able to access the grafana dashboard and we have changed the default admin username and password the next thing is we need to set up the dashboard using the prometheus data source so the next thing which we are gonna do we are gonna create a prometheus data source and as i have mentioned earlier prometheus server is different and graphana server is different there is no communication happening between those two server and to make that happen we need to create a data source so data source is the communication link between the prometheus server and the grafana dashboard so for that purpose we will go we'll go on to the grafana dashboard click on this data this settings icon and here you will find the option to create a data source click on it and once you click on this data source i have already created one that's why it is showing up here but you will find this option a blue color button that is aid data source click on it and here you will find the different sources with which you can create a data sources since we are working with the prometheus so i'm just gonna choose prometheus and here one thing you should remember uh you need to specify the host url of your prometheus server and since this whole session in this whole session we have installed the prometheus server as well as the grafana server on the same virtual machine that means our both the instances of server that is prometheus and grafana running on a same linux or ubuntu system that's why i can use the same url that is http localhost 1990 so i don't need to change the url over here because my local the prometheus server and the graphana server are running on the same machine but in case if the if you have installed the prometheus and grafana on a different machine then you need to specify the correct ip or the host name of your prometheus server so that's the only thing which you need to remember while creating a data source all right now i have created or i have mentioned the url and then click on save and test to verify it and here you can see our data source has been updated and our data source is working so now what you can do you can just uh click on back and yeah now we have created our data sources let's move back to the guide and see what are the next instructions are so here you can see all of these instructions we have already performed we have created a data source for prometheus we have mentioned the url and we have saved and tested and created our data source the next thing is we need to create a dashboard now we have the data source and now our grafana server is communicating with our prometheus server and now we are going to create a dashboard so then we can visualize some of the things okay so for that uh there is one cool feature which is provided by grafana labs and if you head over to this link this grafana lab link which i have mentioned in the blog so i'll open this link over here then here you will find the predefined or already created dashboard by other developers so in case if you are having a similar requirement and you can find similar dashboard over here then you can use the dashboard as it is by importing into your grafana server so you don't need to create the whole dashboard from the scratch and for the same thing what i'm going to do i'm just going to use since i'm using the linux machines and i just want to visualize those matrices like gpu or the ram or the cpu so for that i'm just gonna search with the keyword linux memory into our grafana labs and try to find some similar dashboards so just type in over here linux memory and here you can see there is one dashboard which is like created and it has been downloaded like a 1.81 k times so it's quite a good dashboard i would say click on it and here you will find some of the information the key thing which over here is this id 2747 because this is the important id which we need for importing our importing this dashboard into our grafana server so here you will find some screenshot and here you can see like what this dashboard looks like and how if it's relevant to you then you can use it okay so this is the screenshot of this dashboard and i think i can use it since i'm doing it for the demo purpose but there are some really good uh dashboards available on to this link so you can just search around all right so i'm just cop i'll just copy this code from here i'll go back to my dashboard this is grafana i have already created the data source i'll click on this plus icon for importing the dashboard go to import [Music] enter the number that is 2747 click on load and then you need to select the data source so here you can see it has successfully imported the dashboard and this dashboard is maintained by a guy fritis hoglund that's okay and this dashboard has been last updated on 2018 so it's an old dashboard but since we are working for a demo so that's fine for us okay so here the one thing which you need to be a little bit careful here is the dashboard or sorry not dashboard the data source so here you need to carefully select the data source so here i have created the data source prometheus one previously and so i'm just gonna select the data source because this is from where it is gonna fetch the relevant data and to show it on to our dashboard so i'm just gonna select prometheus one click on import and here you can see uh this is the dashboard uh which is like uh created by that user and i can see all my data over here so here some of the statistics of the linux memory distribution swap usage and yeah it's a very very very basic dashboard i would say okay so now we have done the final step like importing a dashboard into our grafana server or the grafana ui and i think this is the basic steps which you need to do for configuring your prometheus server node exporter and the grafana server to visualize anything so this is the last step for this session and the next thing which i'm planning to do is the alert manager so now we have seen the node exporter but we haven't seen the alert manager yet but we have seen the crucial components involved in this setup so in the next session we are gonna see how to set up the alert manager so that we can trigger some alert based on certain conditions such as like a cpu usage memory usage or if some of the node is going down so we are gonna see the uh alert manager also into the next session if you like the today's session then please consider subscribing to this channel where i keep on posting similar devops content on a weekly basis and in the upcoming session we are gonna look on the alert manager and also how to set up prometheus and grafana for our kubernetes cluster running on aws see you into the next session of devops till then take care and bye

Channel: Rahul Wagh

Views: 29,604

Keywords: prometheus, grafana, node exporter

Id: peH95b16hNI

Length: 36min 42sec (2202 seconds)

Published: Mon Mar 07 2022