How to trigger Alerts from Alert Manager (Prometheus) to Slack Channel #prometheus #sre #devops

hey guys what's up lovely good afternoon to everyone so today once again i come up with a very important topic and today's topic is how we can send or trigger the alert from alert manager to slack channel i'm going to finish this whole process into four step so in the step number one i am going to create the channel inside the slack in the step number two i am going to set up one incoming wavebook in the step number three i am going to configure the incoming wavehook as a receiver inside the alert manager and in the step number four which will be the testing stage where we will be shut down one of the services and we'll see that whether the alert is triggered into the slack channel or not hi guys welcome back my channel the learning destinations my name is avinash and if you haven't yet subscribed my channel then please subscribe my channel and press the bell icon so before i take you to the practical sessions let me tell you one of the things there are some pretty requested that you must be prepared so in my system there is already three services running node exporter alert manager and promisius that i am going to use under this tutorial so if you haven't yet uh ready with these services right and if you don't know then you must watch my this tutorial how to configure alert manager will promise yes where you will have idea how to start these services okay so which you can find under my channel agnes mark the learning destinations okay so you can assume this is the part one and the topic which i am covering now this is the part two okay so now let's start the practical so now i am at uh slack and you can create your own account it is pretty much simple all you need to just keep one email id handy right so i have created my account on the slack and now i am going to create one of the channel so i can use this channel to trigger my alert okay so i give it name like app monitoring i say create skip for now so you can see that i have created one channel app monitoring right then after that in order to create the incoming webhook you must visit the channel you must visit the url api.com okay here you need to click on create new app and then it's okay you take it from a scratch and you give the app name so i'm giving this app name like monitoring and here i need to select the workspace so my workspace in the slack is devops so i say create app so you can see that i am inside my app and inside my workspace then after that you need to click here incoming webhook it's still getting loaded i have successfully created my app you can see here monitoring then after that you see the option is incoming webhook you need to click here it's taking a bit longer time it's still getting loaded let me pause this video for as of now now so after uh clicking on incoming webhook then you must switch on this you must activate this options right so activate incoming wavebook yes now it is on now it gives us facility to create the web so this is just for demo purpose you can use this v url and can trigger into the specific channel like the channel which i made app monitoring right so in such a way you can test it whether this wavebook is working or not so but i need the new wave hook that's why i'm going to create one okay then it's still getting loaded so it gives us the facility to select the channel in which you want to trigger the alert so you can use this waybook to trigger the alert into that specific channel so after creating the app you can select here the channel where you want to trigger the alert so i have created this channel inside the slack if you remember so i am going to select this one and i say allow so you can see here i have already received the webhook url that now i need to configure it as a receiver inside the alert manager so step one and two is completed so step one i have created this channel app monitoring this one okay and then after that i have created this wavebook okay so now let's move to alert manager configuration okay so now i am going to configure the receiver into the alert manager and for that i need to edit the alert manager configuration file which is alertmanager.tml okay so here you can see that uh it is already given everything now what i need to do i simply need to change this webhook url okay so this is the script for like a filing alert and once uh once the alert is resolved then also in such condition it triggers the notifications okay so i will i'm going to post this script into the description sections you can take it from there so let me change the url so i can simply copy the url from here i say copy this configurations i have already because before tutorial i have already tested and verified everything that's why that you can see there is already one api url which was my previous url when i was performing the testing so testing was successful now i am performing for you so i have created everything in front of you so there is a new url so i need to just mention this one the url okay so here the time is you can see like group weight is 10 seconds so it will wait only 10 seconds right then interval is 10 second only then after that it will send the alert and the send resolve is true that means that once the alert is back to the normal state then it also in such condition it triggers the alert okay then after that you can restart the alert manager service so it can take effect okay let's check the status okay so now a step third it means that we have successfully configured the incoming wavebook into the alert manager so we have done with the step number three as well now step four is testing so this is my prometheus server so let's check that how many services as of now up and running so i execute up queries let me refresh and type it again don't think about too much about this message there is some mismatch in timing that's okay so i'm going to execute this up queries so as of now you can see that the three services are running node exporter alert manager and prometheus master okay so i'm going to shut down one of the service node exporter and let's see that how the message is processed into the alerts and then after that we will see that whether that message is triggered into this channel or not so as of now you can see that under the app hyphen monitoring there is no any notifications okay so let's shut down that node exporter so i say system ctl stop node exporter so it is a stop now let's check the status okay so it is down let's click here again the status execute the same queries and here you can see that it's a zero it means the services is down and let's check alert because there is some threshold value so you can see as of now the status is spending so guys these all things i have already explained under this video so you must watch this video then you will have better understanding what is the meaning of pending finding all those state inside the alert manager okay so once it will reach to the threshold value then it will go into firing mode so now after the threshold value it crossed then we can see that now the alert turned into a state fighting state it means that the the services node export services has been down now okay now let's check into the slack channel whether we received or learned or not okay yeah bingo so here you can see that we have successfully received the alert okay that this services node export is down and we have received the alert as well so here you can see that message the instance downs this instance the 9100 port that is what the node exporter services is running now let's start the services and check that whether we are receiving the alert for resolve state or not so let's start this services stop start let's check the status here so it's up running let's check in prometheus let's execute the up query first because there is some threshold value that's why it is just showing firing state execute now here you can see that the services is successfully banned okay now let's go to alert and here you can see that now it's come back came back into the normal state it's a zero active it means that none of the instance is down and let's check into the slack channel now so here you can see that we have received just now the notifications for resolve notification as well so when the services was down we received notification as a filing state and when it get back into the normal state when it was resolved then we received again the message with the resolve notification i think almost all the steps i have successfully covered and this is what i wanted to show you now you have better understanding that how exactly the incoming wave book is configured so you can receive the alert manager notification into the slack channel so if you have any queries don't uh forget to write me in the comment sections i will answer it there and guys if you haven't yet subscribed my channel then i really request you please subscribe my channel and press the bell icon and don't forget to share it with your friends and the colleague so write me if you want me to make any videos on any specific topic i will do it for you so that's all from this lecture guys i hope you really enjoyed it i will see you in my next tutorial till the time thank you so much have lots of fun on the weekend

Channel: Avinashkumar - The Learning Destination

Views: 3,708

Keywords:

Id: 2E6GH9rAUwU

Length: 12min 24sec (744 seconds)

Published: Sun Aug 07 2022