Prometheus Alertmanager Tutorial | Setting Up Alert Manager And Configuring Alert Rules Target down

[Music] hello friends welcome back to my channel and in this tutorial we are going to see how we can configure alert manager and prometheus so if you have seen my previous tutorial on prometheus we have seen how to set up prometheus how we have configure node exporters right and how we can integrate grafana we we have seen all these things in my previous tutorial okay so what we will doing is in this lecture is like we'll be going through the steps on installing a led manager so we'll be already using some machines where we have set up prometheus and also the node explorer okay so if you don't have that setup you can follow my previous tutorial how to set those things okay then we will see how we can configure the prometheus.yaml file for the led manager to work and also we'll be seeing how we can create some rules uh for the alerts okay so what we will not be doing in this part or you know in this section of tutorial is like we will not be creating the alert no yaml file uh to send the alerts to gmail or slack or you know any other setup which you want to okay that i will do it as part of next tutorial but in this tutorial we will go through what uh is the ammo files which we need to edit and but we'll be also seeing the how to create the rules and how to set up the prometheus.com file okay so before i start uh i would request you to subscribe to my channel if you haven't done so like the video share and comment okay so we are going to start with the centos machine so if you have seen my previous tutorial we have a prometheus install and also the node exporter okay so this machine we have both prometheus and node export rings installed and i also have another machine where i have only the node exporter is installed okay so this is how it will need the setup so there will be one prometheus setup and two on our exporters so i'll just show you more about this so if i go to prometheus file and if i see my prometheus.ml file so if you see the prometheus ml file you can see my configuration i have a the node exporter rights job name is not exporter which i have one from the local host another one from this ip address okay and one normal target for the prometheus is the local host so let me clear out the screen so let me start the prometheus first okay so so i'm just starting the prometheus okay and i'm also want to start the node exporter okay so i have start of the novel exporter on the prometheus machine and let me start the note exporter on this machine okay so i have all those things running okay so if i access the prometheus with ip address and the port 9090 you can see the prometheus page is coming up right and if i go to the targets you can see one endpoint is for your localhost metrics from the prometheus and you have a two endpoint for the node exporters right one for localhost another one is for the other node exporter machine okay so that is how the setup is currently now so now what we have to do is we have to configure the uh alert manager right so what we need to do is first we need to download the alert manager so if i go to this prometheus.io download and you can check for the alert manager so you can see some download options right so since i'm using linux i will copy this link okay and i will go to the machine from each machine i'll download it using duplicate okay so if i run w get let's wait for this download to be completed so we'll wait for this download to complete and we have to extract it so we are almost done okay so let's wait for a few seconds okay so we are done so let me clear out the screen and you can see we have a download for alert manager right so let me untie it so so let me just use this manager and let me record okay so we have a new folder called unless manager okay so let me log into that now you can see you have a alert manager uh file and we also have a manager.tml so if i read through this alertmanager.tml file you can see it's mentioned like some wrote right and what intervals and what is the receiver let's talk about webhook and what the webhook does right so it's using something to use repo so if we are using like some email or slack or some other kind of method to you know send our alert we need to create some receiver for that and we need to define that here but that i will show you in the upcoming tutorial okay so first i need to configure the outlet manager here so that the alert is getting created and we can see that the electron getting generated okay then the next phase would be how to send those alert into our communication channel okay so let me start the alert okay so if i see alert manager okay okay so let me show you how we can access left manager so you can access the alert manager using the port 9093 okay so the ip address of the machine a 9093 so in our case so in our case uh it's not reachable because the port nine zero nine three is uh uh not blocked so i will go to this machine okay let me exit uh from the left manager so let me add it to the firewall exception so let me run this command firewall command to add the 9093 as an exception and let me run the command to reload the firewall okay so don't worry i'll give all these comments in the video description and i will also link the github locations from where you can copy all the commands okay if i go back to the browser and if i run this machine okay okay first i need to start the led manager so okay let me start the alert manager and if i go back so now you can see i have a alert manager running right so we don't have any alerts created at this point okay so nothing is showing up here but if you see the status you can see how the alert manager is configured okay so the led manager is running now so now what we need to do is we need to integrate alert manager with the prometheus okay so let me go back to the prometheus machine and let me exit out from the running from ets okay so if i go back if i read this prometheus dot ml file okay you can see there is a section for a lab manager right so now what we have to do is we need to configure the alert manager over here so it can map to the right ip address okay and also we need to define an alert rule so that you know it can see on what rule that manager has to be configuring that okay so uh we will configure these two things in the yaml file first then we will go to create uh direct rules okay so let me edit the file first okay so if you go to my github location i will link this so i have create given this uh from ethicist.tml configuration how it should be okay so what we need to do is this section so we need to set the rule file okay so i'm going to name the rule file as i let dot rules.dml and the target should be localhost okay 9093. so since uh i'm using localhost so it should be a localhost so if you know what you're using someone's ip address or some other machine you should map it to that okay so let me remove this okay and this should be mapped to localhost okay so that will take care of the lead manager where the elect manager is set up okay now we need to change the rule file okay so what i'm going to do is i'm going to name this file as log dot pools.tml okay i will keep the second one because it's uh commented out so anyhow it's not going to work okay so if you don't want you can just remove this so it's not a problem okay so what we did is we change the target to localhost 9093 and the rule file as i let dot rules.dml okay so this name you can give anything but only thing you need to remember is we need to create a file in the same name format otherwise it will not work okay so let me save this file okay so next thing what i'm going to do is i'm going to create a file called alert dot tools dot ml okay and the content of this you can check my github location okay so i have a file called alert.tools.yml so i have given a template here so what it does is it's checking for instance downs alert okay and it will check when there is a an instant is done the node exporter is done it will send you an alert and also it says out of memory and out of this cpu load uh some configuration okay so you can use this template or if you have some other template you can use that as well so let me just copy this okay i'll paste this one here so this will be the alert okay so this would be the electrode so if you see the name alert dot rule so that has to be set correct okay so let me save this file so now we have the all files uh ready so let me start the prometheus again okay now if you go to the prometheus page again to go to this rules you can see there is a rules defined now right what we have added direct showing up here so you see the state okay right and all those things you can see and also you can check the configuration it will show you like when we have added the alert manager and also the tools file and if you go to this alerts tab you can see now we have there are four alerts right uh high cpu load out of the space instant down right so what i'm going to do is i'm going to shut down one of the node exporter okay so i'm just going to exit from this if i go back to my prometheus let me refresh it so you can see in prometheus now there is a instant down active right so what does means like now it has identified one of the instances down so that's why it has created that this is the reason why i left ins inside the section of instant now right so now this is what the alert is getting captured okay now what we need to do is next phrase is we need to send this alert into the slack or gmail or you know any way how you want to communicate for that we need to configure in the alert manager so that i will show you in the next tutorial but uh let me just recap what we have done here okay so what we have done is uh we have installed net manager so if you see i have a net manager installed so you can download with wcat and we have a running alert manager using this file okay i'm not creating as a service we'll we will show all these things in the next sections but you know simply you can run this file okay and we have alertmanager.yaml which we have not touched this in this section where we have to create a receiver okay to send the notification in the case of uh sorry so in the case of snow exporter and prometheus if you go to the prometheus we have a prometheus.yaml file which we have edited right so the prometheus.aml there we have added the target for let manager which is localhost and also we have added a rule file right and you see like i have created a file called tools or camel here so that is inside the prometheus folder okay it's not in the alert manager so if you read through this uh we have given the alert set for checking out the instance down out of memory right out of this space icp load right so these informations are given here so based on that i know we have uh seen how the alert is getting generated in the prometheus so based on that you see that the instance down a lot right because we have one more exporter if i like it this one as well maybe let's see now since i have stopped two node exporter you can see there are two active instance right so there are two uh you know node exporters are down so if i show you the targets you see the two node exporters are down right so that's why that alerts you showing uh to our exporter so if you have a case like a high cpu load or search space uh that also will come up here so next tutorial what we will see is how these alert can be forwarded to your uh gmail or uh probably slack in another session or you know i will also see what other options we have it for you know sending the alert to some uh form of method we want to receive it okay so that is all uh with this tutorial i hope uh this tutorial is informative for you and please watch my upcoming part of session so that you can continue having more knowledge on the prometheus the topic okay we'll also see future how these alerts can be integrated to the profile so kindly subscribe to my channel like this video share and comment

Channel: Thetips4you

Views: 28,982

Keywords: prometheus alertmanager, prometheus alerting, prometheus alert rules, prometheus alertmanager tutorial, prometheus alert rules tutorial, prometheus alert config, prometheus alert server down, prometheus alert setup, prometheus alerts, prometheus alert rules configuration, prometheus alert on target down, prometheus alert low disk space, prometheus alert disk space, prometheus alerts tutorial, prometheus explained, prometheus introduction, prometheus, how to, tutorial, alert

Id: bPHOI29rwB0

Length: 16min 47sec (1007 seconds)

Published: Thu Nov 19 2020