How to configure Prometheus alertmanager to fire slack alert notification | Prometheus tutorial

[Music] hello friends welcome back to my channel so we are back with another tutorial on prometheus and alert manager so if you have seen my previous tutorial we have spoke about how to set up prometheus node exporter alert manager right and also we have seen how we can send alerts to our email right how we have done the configuration uh for sending email alerts uh using the prometheus alert manager but in this tutorial what we are going to do is we are going to see how we can setup prometheus alert manager to configure the slack so uh so what the end of this tutorial what we will be seeing is like we have alerts or notifications sent to slack from prometheus so if you have not seen my previous tutorial please go and have a look because we have spoke about how to set up the you know the alert rules also how to set up the prometheus.yaml file to interact with the you know alert manager i will i will have a just a recap on that also in this tutorial okay just a quick recap so that you know you can go back to my previous tutorial okay and uh also we'll be sharing all the details of the you know the rules and also the uh configuration which we are using in the github location which you can make use of it okay so before i get started into it i would request you to subscribe to my channel if you haven't done so yet kindly subscribe like the video share and comment so in order to start i have a centos machine so if i show you that you know in this endos machine i have prometheus node exporter and also i let manage running so in previous tutorial i have showed you one more machine where i have an old exporter running on that as well but in this tutorial i'm going to use only one machine because i just want to show you how to send an alert okay so let me show you a few details which we have set up first okay so let me go to prometheus folder and let me read this prometheus.yaml file now if you see this prometheus.yaml file we have an alertmanager configuration right which we are connecting to the localhost 909 report because the alert manager is running on the same machine and also we have a rule file called alert.rules right so based on that only the alerts are getting created okay so this configuration for job name uh the prometheus job name and also node exporter i already told you on my the first tutorial like how to set up prometheus and also how to set up node exporter so if you are very new to prometheus i would say you go to the first tutorial how to set up then nor exporter then lab manager so you can have the complete information okay now if you see there is one more file called alert.rules.yaml which we are calling inside the prometheus.cml right so let me just have a view on that so you can see that as well so this ala dot rules dot yaml file is a little bit different from the previous email alert so but we are doing the same uh letting for instance down out of memory but if you see the way how i have returned there will be a little bit of change because uh the labels and alert and you know the way the annotations are done there is a little bit change i have done so there are different ways where you can write it but i think this will be helpful for you to see what is the difference from the previous one and this one okay so you can have a look on that so here if you see we have four alert one for instant down out of memory high cpu load and also out of this space right so there are four alerts which we have created so now let me go and start the prometheus first okay i will also start the note exporter okay so now if you have seen we have the prometheus running right and if you see that there are four alerts which we have created as per rule right and one instant is down and it's uh pending because you know uh as i told you i had a two node exporter and one is down right so i'm not turning on this node exporter uh in this tutorial so anyhow i have to send alerts or we'll make use of this down a lot okay and uh if you see the rule file you can see there are four rules right one two three four one four inches and down out of memory cpu high and uh out of this space okay so that's all uh from the alert configuration right now we have to go to the alert manager [Music] so now if i go to alert manager and if i read the alert manager dot yaml file so this is the default yaml file okay where we have the global uh road and the receiver name is webhook right and the configuration for the web book here if you have seen my previous tutorial on email i have shown you how to change this to email right so this was the configuration for email if you have not seen like we have created a receiver for email notification right then we have to give our email id and the app password which we created from the gmail right so now let me go to the alert manager for slack so the difference in the slack is the receiver name will be the slack uh name it doesn't matter what name you give i just give a name like slack notification you can just you like slack abcd whatever name you want to cue but the name has to come here as well then the difference is like we have to give the slack configs and in the slack config we have to give an api url which is the web hook uh url from the slack okay and there should be a channel name okay so to do this uh first i need to create uh the api all right so if you have seen uh one of my tutorial how to integrate slack uh with jenkins uh you would have seen how i have shown how i have configured the jenkins right so if you seen we are getting some job alerts from jenkins now here what we need to do is uh we need to go to the configuration like settings and manage app so once you click on the manage app you get into a browser page where you can see the app store the jenkins ci app i have used it for the previous integration with slack and jenkins for this tutorial we need to search for webhook okay so let's search for webhook so you will see a lot of options there will be outgoing web book and incoming web book so what we require is an incoming web book so you just click on it so once you click on the incoming web book you can read through description configuration and all those things there's a button called add to slack right so just click on that so that will take you to the option where you can select a channel okay so in my case i will create a new channel actually so and i will call this as the prometheus letting okay okay so i'll tell prometheus left okay let's me let me create that yep so i think you can see uh the prometheus dot alert channel is created okay and uh just click on this button right add incoming web book integration so this button will be enabled only once you select uh the channel okay so now it will create a webhook url so you can you can copy this uh you can read through the contents if you want but we for us it's uh simple we don't need to do much you just need to save the setting okay and what we require is just go back to your uh configuration change this api url with the book url okay so don't use mine okay i'll be deleting my web book url because ij i'm just showing for uh you know your demo so if you are you have to create the book url for your uh slack okay and uh the channel name is prometheus alert right so we just need to copy this as well and just put the name here without space okay so it's that pretty simple okay so these sections is already available in the file so i can show you right so these sections are already available so let me clear out the screen let me edit let manager.yaml okay so what we require is we just need to change some of this we name this as uh black underscore notifications okay use the same here for the name okay then you need to change the web book to slack slack uh underscore configs okay and you need to change this url because we are not going to use the same method okay so i will do one thing i will copy those contents from my text editor okay so i will copy these contents and i'll paste it just make sure that the alignment is correct if you don't put it correct then the ml configuration will not work okay because uh yaml will look like a space and everything has to be correct okay so let me save this file [Music] so we have the ml file saved now let me run the alert manager okay so we have the outlet manager running now let's see we have to wait uh for a few minutes because i think we have given five minutes yup you can see there is an alert game right so i'll let manager firing instance down right so we already got that if you go back to the left manager okay let me just show you one more thing so that's your dot yaml so if you see the result timeout is 5 minute and also the repeat interval is 1 hour just for our testing case what i will do is i will change this to 1 minute and also the repeat interval as one minute okay so we can see the alerts coming at much faster okay so let me start the alert manager again okay so you can see there is one more alert game because we given the repeat interval as uh every one minute okay let me close the other node exporter as well so if you go to my prometheus you should see two instance down okay so i think it's just updating so you can see there are two instances down all right so because i have closed the other node exporter as well so you can see both are down right so let's wait for that manager now so now if you see we have uh two instances being created right so one for the uh 129 ip address another one for the local host so there are two alert has been created now the challenge here is we are not getting much information from this right it's just a link uh coming up where actually we in the email if you have seen we got some information like what is the criticality and uh some information right we can bring those information as well into the alert letting but we need to add some more content into this section okay so let me show you how to do that so i have already prepared that so if you see i have added some more content like you know to make it much uh better so you can go through this content okay so there is also a site uh called this where you can test this kind of uh letting how it will look like so you can create your own uh no alert configuration alert rules you can put it here and you can do a preview so it will show you how it will look like okay how the how it will look like in the uh slack so in our case if you see i'm just trying to sort uh things a little bit better and i just want to show you you know the content which we need to take it from the electing rule okay and i'm also putting an icon to look it much better okay so let me copy this content [Music] and i will go to the alert manager file okay dot ml file so let me paste this okay okay so this is how it will look like okay so i have added those additional content into this okay you can copy all this content from my github link i will give it into the video description so let me run the alert manager again okay okay so we have the alert manager running okay now let's see in the slack how the alerts are getting created okay now if you see i have a new alert uh created right if i click on show more if you see it it is giving me much more information right so it's giving me not only the heading it's also showing me what is the alert right and what is the cbr it's critical what is alert name what is the instance name right uh what is the job it's a node exporter right so it's giving me much more information so this is how i have enhanced the alert so that's the thing i know which i have used okay so you can also see how to enhance your alert you can use your own method okay it's not a problem but i just use my content what i need to show so if you would like to use you can use the same format for your purpose as well so now if you have seen like you know we are getting the complete information right and every one minute it's uh firing the alert so we have given the repeat as one minute right since it's uh 23 16 and this is 23 15 and it's 23 14 and if you see the icon i have used an icon of prometheus okay so this link is going to the prometheus icon so you can use a different icon as well okay so this will give you a complete information you can change the way how you want to do it uh depending upon your need okay this is only example i want to show how we can integrate uh prometheus alerting with the slack okay so now if you have seen like in this tutorial we have gone through all the steps right uh the previous tutorial we have just done a recap what is alert dot rules how we have set up also the prometheus.yaml file and in the led uh manager.tml file we have configured to connect to the slack using the webhook api right so you have seen how to create the api and how to make use of that right so you can go through my github link for the complete detail okay so i hope this tutorial is informative for you and i thank you for watching again i have a request for you kindly subscribe to my channel like the video share and comment [Music] you

Channel: Thetips4you

Views: 17,303

Keywords: prometheus alertmanager slack, prometheus slack notification, prometheus slack alert, prometheus alertmanager, prometheus alerting, prometheus alertmanager tutorial, prometheus alertmanager webhook, prometheus alert not firing, prometheus alert target down, prometheus alert when value changes, prometheus alert rules, prometheus email alert, prometheus alert grouping, prometheus, how to, tutorial, beginners, alerting, email, slack, education, tech, firing, alertmanager

Id: 6Jr6rntMI3A

Length: 18min 35sec (1115 seconds)

Published: Fri Nov 27 2020