Prometheus Tutorial | Prometheus Email Alert | Prometheus Alert Manager

hey everyone i hope all of you are doing great this is going to be fifth lesson of this prometheus tutorial and in this tutorial we are going to continue learning how to send an email alert whenever one of the note goes down so if you have not watched my previous video please watch that video because this is just a subsequent part of that video if you don't understand that video well in that case it is just not going to be easy for you to understand this video so please go ahead and watch the previous video and then come here again so in previous lesson i have shown you how we have set up alerts in prometheus and we created the alerting rule that if any of the server goes down we wanted to an alert to be triggered and we haven't really installed alert manager so far so in this video i'm going to show you how to install alert manager and how to send an email alert so so far you have seen that we can see an alert here which is saying server is down and right now we do not have any server which is down that's why we see everything is showing as inactive now as soon as one of the server goes down we are going to see that alert is going to be moving to pending state and if the server remains down for more than one minute in that case you are going to see that state is going to be changing to firing state so that's what we have learned in previous video now let's go ahead and start configuring email alerts by using prometheus alert manager so just a quick introduction of the you know complete step which we need to do in order to send an email alert is that first we need to setup and configure alert manager so alert manager is a separate component which is uh which you can download from this website here is the alert manager and you can install alert manager either on the same server where you have installed a prometheus or you may installed alert manager on a separate server as well okay so once you install alert manager and you configure alert manager after that you need to tell prometheus where you have installed alert manager okay so basically you're gonna go to a prometheus configuration file and there you are going to tell where is your alert manager located so once you do those two things uh you your prometheus server and alert manager is completely connected and then you can start creating all the alerting rules which is going to uh get fired once the condition meets and which is going to result in a triggering of the alerts in alert manager okay so that's how everything is going to work now now i have already downloaded this alert manager so i'm just going to go to this server this is the same server where i have where i have installed prometheus on lesson number one so just go ahead and watch previous videos as well which is just going to be more clarity but if you already have been following all the previous tutorials so this is the server where we installed prometheus and now we have downloaded alert manager so i have downloaded alert manager and extracted alert manager so that's why i see a alert manager folder here so i'm gonna move to alert manager folder and then we can do ls and here you can see we have alert manager alert manager is a utility and there is also a configuration file which is called alertmanager.yml now let's open this alertmanager.yml file now by default you can see this alertmanager.yml file is having some sort of configuration okay so all you need to know here is that root you can leave the root as it is and in the root you see receiver has been configured okay so there are many receivers which you can configure so receiver is basically a kind of alert which you want to configure okay so there can be web hook alerts there can be email alerts and there can be several other types of alerts as well so all this information you are going to get from prometheus configuration documentation alert manager configuration documentation so once you go to this documentation you can just scroll down and then you can go to receiver section here you can see receiver and under the receiver you can see we have several types of receivers available so in this lesson we are going to configure email config okay so that's the receiver which we are going to use and i'm just going to go back to alert manager server and now you can see this is the configuration file this is the default one which is available i'm just going to create a new file i'm not going to change this file and i've already created a new configuration file so you can also go ahead and create a new configuration file or if you want to modify the existing one you can do that as well but in my case i just created a new configuration file which is called email.yml and let's open this file and in this file you can see this is the settings which i have set okay so the only thing which i change from here to here is one thing you can see receiver name previously it was webhook so i created a new configuration file where i change the receiver name to send underscore email it can be any logical name you just need to make sure this name matches with the name which you have given under receiver so i'm telling receiver is going to be send email and here also i'm telling that my receiver is going to be send email and this is going to be the configuration of that receiver so here the important thing is that you need to use email underscore configs and you can see this here as well in this documentation you can see that we are using email underscore config stanza and then under this we need to use all the all the settings which have been given here okay so we need to use two field where we are going to specify where we want to send an email alert to and then we have from field where we need to specify from which email account the email is going to be sent to and then we have some other settings which is like smtp authorization username and password okay so from your organization from your project uh whichever project you are working on you are going to receive all of these smtp details from the project and you need to put all of these details here now notice one thing here uh since i'm using hostinger so i have put hosting or smtp details here in your case uh it is going to be different and also in my case from email id is email.rtpanther.com and 2e is prometheus underscore demo at mailinator.com and similarly i have given username and password here as well uh i'm showing you password here because i'm just going to change the password afterwards once i complete this tutorial so if you are just make sure whatever password you give here that is confidential and you do not share with anyone okay so after that there is inhibit rules we are going to discuss about inhibit rules and several other things but that is going to be in the next part of the tutorial in this one we are just going to set up this configuration which is which i am showing you right now on the screen and now we are going to start alert manager con alert manager okay now just momentless let me go go here back on this documentation and here you can see there is already a syntax given on how you can start the configuration how you can start alert manager and you can pass the configuration file so by default here you can see in this command line they are passing alertmanager.yml configuration file which is the default one but in my case i have created a new configuration file and i'm going to use that one so here i'm going to use that i'm going to start alert manager by using this configuration file which is going to be called email.yml okay and hit enter and then it is going to start okay so i'm just going to open one more instance of the same server and then i'm going to show you what is the changes now we need to make in uh prometheus dot configuration file okay so i'm just gonna go back to documentation again just to quickly show you uh that we were here and we have set up and configured alert manager now we need to configure prometheus to talk to the alert manager so i'm gonna go to this i'm gonna go to this prometheus server again and let's do ls and we need to go to prometheus and here i'm going to open prometheus.yml and now here by default you will notice that this alerting in the alerting this target is going to be disabled so here you just if you see hash prefixed here you can just remove that hash and then you can give the ip address or the host name of wherever you have installed alert manager so in this case i am giving localhost because i have installed and we have installed alert manager and alert manager is running on the same server okay so if you are running it on a different server you can give the ip address or the host name of that server and by default alert manager port is 9093 so that is what i have given here okay so we are going to just save it and once you make changes in this prometheus configuration file make sure you always restart prometheus services i have already restarted it so i do not need to do it again now let's go ahead and we are going to we are good now we have already set up a prometheus we have already set up alert manager and we have also configured prometheus to talk to alert manager and we have also created alerting rules which was created in previous lesson and that's the reason we can see here that it is we can see this alerting rule here which is called server is down okay so now i'm going to shut down one of the node exporter and this one we are going to see that it is going to show us that one of the node is going to pending state and then it is going to change to firing state so let me just shut down services on one of the nodes so here we have node exporter running so i'm just going to stop node exporter services and we are going to see here that alert is going to be changing to pending state okay and this is the malinator mailbox which i have and this was just the older alert so i'm just going to remove this one we don't need this one uh we are going to wait for the new alert which is going which we are going to receive pretty soon okay so let's go back to prometheus and let's refresh this page and shortly as you can see right now we have one alert which has been which is being shown here in the pending state so one of the node has gone down which is 10 1 10.128.0.3 so uh it is going to wait in this state for one minute and if this server remains down for one minute in that case it is going to be changing from pending to firing state and once it changes to firing state that is when we are going to receive the email alert here okay so uh i'm just going to pause this video for a moment and once this changes to firing mode i'm gonna i'm gonna resume the video okay so right now you can see we have now uh alert been shown here in the filing state so we should be receiving an email alert pretty soon so here you can see we have already received an email alert which is telling that server is down so that's all in this video see you again in the next video

Channel: Vikas Jha

Views: 18,194

Keywords: Alerts in Prometheus, Prometheus Alerts, Prometheus Email Alerts, Prometheus Alerts Manager, Prometheus Server Down Alerts, Prometheus architecture, Prometheus for beginners, Learn prometheus, Prometheus tutorial, prometheus monitoring, grafana prometheus, prometheus architecture explained, prometheus monitoring tutorial, what is prometheus monitoring, what is prometheus monitoring tool, what is prometheus, prometheus monitoring tutorial for beginners, prometheus setup

Id: 0qCqXxf97Gk

Length: 11min 2sec (662 seconds)

Published: Fri Sep 10 2021