Prometheus email alerting | Prometheus alertmanager email configuration

[Music] hello friends welcome back to my channel so if you have seen my previous tutorial on prometheus we have spoke about how to install prometheus alert manager right and also we have seen how to set up our prometheus ml file to interact with the led manager and also we have seen how to create alert rules so what we have not done in that tutorial was like how to send this alert uh to any medium right like email or slack or any other method right so in this tutorial what we are going to do is we are going to see how we can do the prometheus email alerting so we will see you know as part of this tutorial we will see how in the prometheus and alert manager what configuration we have to do for the email to be sent to our email address so what we will be using is we'll be using the gmail configuration so if you have your own smtp server or if you have your own email setup please make use of that in this tutorial okay so before i start i would request you to subscribe to my channel kindly like this video share and comment so keep supporting me so that you know as a community we can learn more things and we can grow together so i have a machine centos machine in which the prometheus is installed and also the node exporter and also the alert manager okay so in the previous tutorial i have used two machine one for for the prometheus node node exporter and also another machine just as a node exporter okay so in this tutorial i will use only one machine because i just want to show you how we can generate the email right so if you see you know uh let me show you what we have done in the previous tutorial so you can those who have not uh seen my previous tutorial please go and have a check how we have set up the prometheus.yaml file okay so the first thing is like prometheus.tml file so if you have seen like this configuration we have in dialed manager configuration right we have set up the localhost 9093 because my allied manager is running on this machine right and i also we have set up the rule file from where which it has to go which is called alert dot rules.yaml right and these are uh the previous tutorial like where we have set up the node exporter right so i would request you to please go and have a look on those tutorials like if you have not set up prometheus before so do that first and then you can check this tutorial okay so now you see like there is a file called alert.rules.gaming right so if i read this file you can see the alert uh rules which we have configured right the name uh dot rules and in what cases we have to create the alert right so instant down how host out of memory hosts the cpu load right disk space so these are the four alert categories we have created okay so all these details you know i have it in my github location so you can have a look there okay so let me start the prometheus first okay so i'm going to start prometheus let me also start the node exporter [Music] okay so the node exporter is also started okay so let me it don't do i'm not going to start the alert manager now okay let me show you how we can configure the alertmanager.yaml so let me access the prometheus first on the web browser so if you access the prometheus using the ip address and port 9090 you have the alerts right so four alerts we have created as per the other node rules and you can see one is active because you know one of my node exporter is down right so if i go to target you can see one of my node exporter is restaurant okay so that's why the alert is uh high or active so now what we need to do is we need to send this alert to an email right so let me go back to the machine okay so let me go to the folder alert manager so inside outlet manager we have a file called alertmanager.yaml right so we have seen this in the previous tutorial but i have told like we are not going to edit in the previous tutorial right so now if you see this is how the default file look like right so go global uh setting for result timeout it five minutes right and also we have some root right so and also there is a receiver called web hook so it's a name of the receiver called web book it can be any name okay and but uh if you see below that it have a receivers and inside that you use the same name webhook and you are giving the configuration right so what i'm going to do is i'm going to edit this file okay and i'm going to create a receiver uh called email notification okay and i will use the same here and i'm going to give the email configs uh for that okay so let me clear out the screen okay so let me edit this file let manager.tml okay and i'm going to change this one let me change this to email underscore notifications okay and i'm going to call the same thing here okay so these things i i will delete because it's not a webhook config it's a email underscore config okay and let me delete these things as well so it's not neither because we need to enter some other configurations okay so before i do it here let me show you uh you know how to write this in a sublime text or text editor okay so this is the file okay which we are going to use here okay so you see the global uh time out you can make it less than five minute as well okay so if you want you can change this five minute to one minute so if i go there you can change that as well okay and this group buy and everything if you want you can keep it or if you don't want if you want to simplify you can do it in this way also okay you can directly go for a route with the receiver name and uh that is your name uh under here okay and the email config so under the email config what we need to do is we need to give a to email address okay and from email address okay and also we need to give the smtp address for since the gmail i'm going to use smtp.gmail.com okay and you need to authorize the email to send the email right so you need to have a username right and the identity and the password okay so the password is going to be you know the email password but you know what i would suggest you is like you rather than giving your email password you create a app password in gmail okay so let me show you how you do that now in gmail if you log into the gmail account and if you go to manage your google account you will get into a screen like this okay and you will have on the left side like home personal info security and option right so you need to click on the security option so there is that is the place where we can create the app password okay so once you click on this uh security option you will have an option for uh two-step uh verification and a password right so if you see i have one password already created so you can click over here it will ask you to log in again okay then you can create an app password so you may not have a prometheus application there so you can just give a name like prometheus alerting or something and you create a password so you will give get a password from there so you make a copy of it then you can use that into this uh script what we are doing okay so what you can do is you can use that app password here so that means uh you you are giving the only the password of the app which you have created okay so it's almost like you are allowing the app to access your gmail uh the full access right okay so don't uh openly give that password anywhere or don't share it with anyone okay so you keep it only for your usage okay so now now what we have to do is we need to copy this content okay and we need to put it in under the email okay so that is all we need to do so this is the whole uh file okay so i'll be sharing uh the whole file and again in my github location so you can copy it from there okay so this is the inhibit rules which i'm keeping here to give the cvrt status of the alert okay so if you change your emails with your emails which you want to receive the email also to send okay so and also the password which you create from the password so if you see like you know the full details we have given here so this is for the email config right so change with your details okay so you can copy this content from my github location so let me save this okay so we have the file saved okay so alert manager dot dml okay so this is the complete file so i think that's all we need then we can start the alert manager now okay so let me start the alert manager so you can see the alert manager started listening okay and let me check it here so we have also one instant down alert and it's firing right so i think it's already done now okay so i need to go and check my gmail so let's just give me a few minutes as you can see i started receiving the alerts so you can see it has already sent few alerts to me so you can see firing one is for the same ip address okay and the two is for the second node exporter which i have also closed the node exporter which is running so let me show you that so now if you see like i have uh two instance down right so i have uh close the other node exporter which was running so targets you see the two node exporters are down right so that's why i have two alerts uh coming one for the one uh the note exporter which was running on other machine this is the ip address right and this one is from localhost okay so you see that uh the first one shows the ip address one twenty nine nine one zero zero right and this one is the local host okay so this is how the content of the email look like right so the alert is name is called instant down right because we have set up dollar damage instance down for the node exporter right and it's a cvr it is critical right so it's giving you a message like which endpoint is done right so if it's for some high cpu sage or the memory usage or what we have configured other parameters if that is created you will be getting the alert for that okay so that is great right so we are getting the email notifications now okay so that's what we wanted to do so if you have seen like you know we configured the whole uh process now right dot manager.yaml to configure the receiver so this receiver what we are using the email config right and the from which is dot yaml to integrate with the alert manager and also we have created another room right so end to end if you relate all these tutorials like what we created the previous sections if you relate all those things you have a complete uh prometheus system right you have prometheus running then you have a node exporter to collect data from different metrics right then you have a alert manager now we have configured the alert manager to send email as well as notification right so that's great right so you have a full monitoring system ready so that is all i wanted to show in this tutorial so i will share my github location from where you can copy the contents okay for your setting up so i hope this tutorial is informative for you and thank you for watching so the only request from my end is kindly subscribe to my channel like this video share comment uh share this videos with all your friends colleague who want to learn more about technical topics okay so i appreciate your support

Channel: Thetips4you

Views: 18,515

Keywords: Prometheus email alerting, prometheus email alert, prometheus alertmanager email configuration, prometheus email alert template, prometheus alertmanager email, prometheus alertmanager, prometheus alerting, prometheus alert rules, prometheus alertmanager tutorial, prometheus alert rules tutorial, prometheus alert firing, prometheus alert setup, prometheus, alert, alertmanager, grafana, prometheus alert severity

Id: x2qTvTN8YKI

Length: 14min 15sec (855 seconds)

Published: Tue Nov 24 2020