How to Install Prometheus on Kubernetes Cluster? (Prometheus Operator | Grafana | Alertmanager)

today we're going to deploy primitives to our brand new kubernetes cluster first we're going to start with the primitive sprouter then we're going to be using the custom resource definitions to create them prometheus and also we will set up the grafana and import existing dashboard to monitor primitives itself and finally we're gonna set up the alert manager and configure it to send notifications to the slack and at the end i'm gonna show you one of the examples how you can trigger it and send notification to the slot let me quickly show you how it's gonna look like so we have the prime ethos and under the status you can see that we have multiple targets and also we intentionally broke one of those uh target one of the target which is the primitive separator and it will trigger the alert and send notification to the slack also we gonna set up the grafana and add primitives as a data source and import the grafana dashboard for prometheus that's the one and finally that's the the alert manager that will send notification to the slack so when the target will go down for parameter separator it will send the message that target is down let's go ahead and create the folder called kratz and let me create the first file which will be primethose.yaml and let me place the content all right so it's pretty large file and it's the custom resource definition for prometheus so later on in this video we're going to be using those kinds objects in kubernetes so instead of defining stateful set or deployment we're going to say kind primitives and then the parameters apparature will be able to detect this and then parse the prime ethos object and well create or delete the parameters instance and let me do quickly create other graphs and then we can apply all together all right so we created all of them let me quickly go over so this is the alert manager config alert manager and then put so we're going to be using the alert manager in this video directly i would say and then we're going to use the promethos premiere tools rule and service monitor well for example the service monitor is used to add additional targets to four parameters to modify the parameters config and then it will reload the config also we have the thanos rules so in this video we're not gonna be deploying thanos but in the future videos i'll show you how to set up tunnels as well so those are just the custom research definitions which you can think of it as you would um the way how to store uh structured data in kubernetes and then your application for example primitive separator can query the data and perform certain actions for example maintain primitives cluster all right so let's go ahead and create all of those custom resource definitions in our kubernetes cluster let's go to terminal and well first of all let's check what custom resource definitions we already have keep still get crowds so it's a global object so you don't have to specify the namespace so right now we have only two well based on your kubernetes cluster you will have more or less than two all right so now let's go ahead and deploy those custom resources we're gonna use cubestyle apply command and we just simply provide the path to that folder and we're gonna apply all those files all together all right so we created those crowds let me see right yeah we do have them okay so the next step would be to deploy the parameter separator so let's go ahead and create the folder primitives router and let me also define few files and then we're going to apply them all together so the first one will be the namespace so we're going to use the dedicated namespace for all our monitoring tools so let me create this first so it's a kind namespace and the name is the monitoring the next object will be the series account all right so this is the series account for the which is called parameter separator then let's define some roles all right so this is the cluster roll which is uh cluster wide it will grant permissions across all the name spaces alright so this is the cluster role and now we need to create the cluster role binding all right this cluster roll binding will bind the cluster all which will which we just created which is called parameter separator and the subject which is the the series account that was created here let's go ahead and define the deployment for the parameter spiral now all right so this is the deployment object nothing special about it just one single replica some labels and this is the some arguments that we're gonna pass to this uh port and well we defined some resources and etc so and this is the series account that we just created and also let's go ahead and create the serious account well let's go ahead and create the service we're going to be using that service to scrape metrics later on all right so this is the service and this is the selector to select those pods and those has to match those those labels all right so now we have everything ready and let's go ahead and create the primitive router let's go to terminal and type cubestl apply and instead of crowd let's use the parameters operator folder and we're going to create all those objects right so let's get the pods in that monitoring space get ports and since we deployed it in the monitoring space we need to provide this additional flag to cubesat all right so now it's ready and well it's a best practice to always check the logs to see if there are any errors so let's do that keep still locks dash f to follow logs then the name of the port and don't forget the namespace monitoring all right so it looks good no errors well let me quickly search for errors yeah i don't have any errors all right that's good now well let's go ahead and deploy parameters first let's create the folder and define some objects and the first one of course gonna be serious account then we're also going to define the cluster all for that serious account now we're going to bind that role with that serious account and finally we're gonna define the custom resource switches primitives right okay let's take a look on the parameters right so the first block metadata will be well the same and then under the spec here comes the custom part so and here under the version you can provide the certain version of your promethos then you have the alerting block uh later on we will deploy the alert manager and this is just some definitions some parameters that helps to identify and select that alert manager for example the monitoring namespace and this will be the service name and this is the port later on i will show you exactly those are two very important parameters later on uh when we will create some serious monitors and prevent those rules those labels will be used to select those rules well later i'll show you what you need to match you know this to work then we're gonna provide that series account that we just created and then some resources also we have the disabled admin api storage under the storage well since we're going to be using the local disks for that parameters cluster well for the parameters instance we need to create the volume and attach that volume to that port so here this is the optional so if you're not gonna specify the storage class name it will use the default storage class for your kubernetes cluster for example let's go to the terminal and type cube still get storage class so for example if you're not going to find this key here so you're just going to look for default storage class and use that one then well we say 20 gigabytes for the local disk then the security stuff replica there is a way to deploy the high available primitives cluster but later on i'll show you and create a tutorial how to deploy them thanos and use highly available parameters cluster then we can define the retention i believe default retention is 14 days but we're gonna say seven days then this is the just a global parameter that you can actually uh well you can provide your scrap interval on each series monitor object later on i'll show you all right so that's pretty much it you you can have you can add additional parameters as well but for this video that will be sufficient all right let's go ahead and well what else do we have also before we gonna create this parameters cluster i wanted to show you that you can also create the ingress so since we are deploying parameters in brand new kubernetes cluster we don't have any ingressors just yet so i'm just gonna show you the definition of that ingress that you can use in case if you want to use some grass right so let me show you so and here the ingress here you need to update with your ingress class and also with your domain and that this particular part will stay the same so this is the service and this is the port and only it's just for you as an example if you want to use it and also i'm gonna create the series monitor which is the way to add targets to your primitives and here we're gonna add the parameters itself as a target so this is another custom resource definition which is called series monitor and like i said this is very important block under labels it has to have this label team devops by example well it may be your custom label but this particular label has to match with this uh series monitor selector those labels have have too much alright so now and then you have to select endpoints to scrape right so for example well this is just a regular owner labels which is just controls how parameters handles conflicts between labels that are already present and the server labels and here later on when we deploy this parameters i'll show you how to debug and get exactly the name of those parts and this is the the port of that endpoint and this is the label on that endpoint right now let's just deploy the whole thing let's go to the terminal cube still apply and it's going to be primitives folder and it's going to create all the subjects including service monitor all right so let's check cubestyle and getpods in monitoring namespace and we have the primitives demand which is the the port created by that custom resource which is created by perimeter spirator and it contains uh two containers the first one is the parameters itself and the second one is configure loader so when you for example create the series monitor it will update the the config for that parameters and reload it all right let's um run it again get puts and now it's ready let me show you that we have the service keep still get svc in monitoring space and that series for the ap router and this series for that parameter that we just created and let's since we're not going to be using the ingresses in this lesson let's just put forward it and use localhost okay so we're going to be using this service name and to do that we're going to use keep still in monitoring space and this is the command port forward then the svc and that's the name of the service then the first one is the port on your localhost and this port inside the port let's run it and now we can go to the browser and type http localhost and port 1990 right for the parameters and this is the parameters and under the status we can check the targets and we already have one target so this target parameters comes from this series monitor and later on we will create additional service monitors and i'll show you well actually let me show you right now how where we got this web for example so let's go to terminal and type cube still get endpoints in monitoring namespace and this is the endpoint basically that we need to select and let's describe it keep still describe endpoint and well this is the end point and also then namespace all right all right so this is actually the web here which has to be that's the port name has to match which is a map to the port 99 also this particular operated primitives equal to true comes from the endpoint label and this has to match that particular label so two parts uh this label and this port okay and that's how would you add additional series monitors all right so now well let me actually go ahead and add another series monitor for the parameters apparatura which also has the endpoint parameters endpoint that you can scrape okay let's go ahead and add that series monitor all right so this is the parameter separator and well maybe let me quickly show you again so that's pretty much this label has too much as you remember then the namespace monitoring and that's the name of that just that serious monitor object here the endpoints let me go back here and now well let me get all the endpoints and now let me get the apparature endpoint so it's going to be let me delete this and just copy the whole thing and this is the name of the port and this is the the label that matches to this on primitives appearance right yeah so now let's go ahead and create this series monitor for operator and to do that we can just reapply the whole folder and it will just create or update some objects for example in this case it just created this series monitor let's go back here let's refresh so it takes a little few seconds probably to detect and refresh the config the parameters config all right now it detected it added the end point and right now it's unknown state so when it will be able to scrape it it will be either up or down so let me refresh it again so it's usually like i said maybe 5-10 seconds let me see double check the deployment it should be should be right all right so now it's up okay now we have two targets all right the next step will be to deploy grafana let's go ahead and create the folder for grafana and of course the first one again the series account that's called graffana now let's define the config map with the grafana configuration so it's not the kubernetes configuration just the grafana file which will be stored as the config map and then we're gonna mount this to the grafana port for example where to put the logs and some paths just generic stuff also we want to persist data right so when we create the dashboard we don't want that dashboard we don't want to lose that dashboard when we restart the port so to avoid that we're going to create a resistant volume claim and we're going to mount this pvc to the pot and well let's just do 10 gigabyte now let's create the secret file where we're going to define the username and password for the grafana all right so um it has to be encoded in base 64. so well the first one is the admin user and this is the admin password let's go ahead and generate uh the encoded versions of that username and password to do that we're gonna use the echo command we're going to provide dashing this one is very important for the secrets it's going to remove the new line at the end of the string and this is the this thing itself and then we're going to pipe this to the base64 tool to encode it in base64 okay let's go ahead and grab this and paste it here also let's go ahead and do the same thing for the password for the password that's gonna be our password password123 let's let me grab the whole thing and paste it under the password okay that's done now let's define the deployment for the grafana let's take a look so it's a type of deployment some labels grafana and here we're gonna have the init container which will run once and it will update the ownership of that directory then we have the the main container with the grafana we gonna mount some config and also the storage to resist our dashboards here we're gonna use those username and password as the environment variables and provide them to the grafana also health checks and resources here okay and this is the you define the volume block for the config and this is the persistent volume all right so that's that now let's go ahead and create the service for the grafana and just a reminder the service is used to load balanced traffic between those spots so it's going to be the just a regular service with the port 3000 and selector app grafana this has to match with this particular label also optionally you can create the ingress if you do want to use it let me just leave the example here as well it's pretty much the same thing your domain you're in grass class and this is the service and port and grafana also provides the endpoint parameters endpoint that we can scrape so let's do that as well and that's going to be the label and then we're going to select the endpoint for the grafana and call it grafana all right so grafana is ready now let's go ahead and apply all those files graphana all right so graffana and now let's get the pots all right so now it's initializing so let's wait till it finishes all right so now it's running okay now we can use the the same port forward command keep still get svc in monitoring gamespace and let's use this service and port forward it as well so let's plan the script and let's run keep still monitoring power forward svc grafana just different port all right so let's go to the local host let me refresh this as well so now you have the grafana target let's go to the http localhost 3000 all right so we have the grafana and the password the username is admin and password plus word one two three from the secret right so let's sign in not now and let me remove this pane okay so the first thing you need to add the data source let's go to data sources and let's click on add data source and parameters okay so here just one single field that you need to fill up is the url and let's use http and then since the grafana and parameters deployed in the same kubernetes cluster we can use the kubernetes service to accept to access so here we need to use this particular service for the parameters and then well since they are both deployed in the same name space monotonic namespace you can leave it as is and it will work or you can also use the monitoring namespace at the end and also you need to provide the port for primitos which is 99 right so the this comes from here 99 now we can go ahead and test it save and test all right so data source is working let's go ahead and create the dashboard just to test and in this lesson we're going to be using the open source dashboard for parameters itself so we're not going to create anything from scratch we just import the existing dashboard let's and that's the dashboard that we're going to be using from the grafana.com website so that's done prometheus and in order to import this dashboard we can copy this id let me copy this close this window and just click on import you can you can actually import the json object you can paste it here or like i said you can use the id from grafana.com and just click on load and it will load that dashboard so let's remove this just let's keep it prometheus simple then we're gonna we can select the folder we don't have any rather than general so let's keep it here and we can change uig just random stuff and also don't forget that you need to select the data source which is the primitives and let's click import and now well we have dashboard and we even have some data in the dashboard so since we added this target so the data comes from the that target and that's basically just default and primitives dashboard for to monitor primitives instance itself so it's uptime some total series bunch of stuff and that's how would you just use the grafana with existing well with the parameters all right so that's all for grafana and now let's go ahead and deploy the alert manager let me cancel this for grafana and let's go ahead and deploy the alert manager well first of all we need to create the folder and then the first file will be the config for that alert manager all right so this is the native primitives i would say alert manager config and this is the the key and this is just a multi-line string that used here so and we're gonna store this alert manager config as the secret and it has to have this particular name too much and in order to apply for our alert manager and this is the config so for example i just want to point out some some things so those are the routes and this is the the route that we're gonna use to create the alert and later we will create alert and uh create the label on that already separated warning so this is the route and if it will match that particular label it will send notifications to this particular receiver and this is the definition of that receiver which is the slack config so we're gonna send alerts to the alerts channel then when it's gonna be resolved we're gonna send confirmation that it was resolved and here api url so let's go ahead and get that api url from this log let's go to the google and type slack api and the first link will be to slack api then let's go to your apps and create new app let's call it primitives and let's use my workspace and create app now we need to create the incoming web hook let's click on webhook and let's enable it and down here you can add this webhook to certain channels so let's go ahead and add this channel well let's add it to the alerts channel to this channel and let's click allow now we have the webhook and well we can even test it let's use this to test let's copy let me show you my slack so this is the alerts channel in my slack let's go to the and well i just copy this content curl command and in the in terminal we can post it and just run it and it will show this hello world to the slack channel send all right so this is the hello world okay so it works now we can go ahead and grab this particular url let's copy it and use it in our slack well in our alert manager config and this is just a template for the alerts to send to the slack okay now this is the config now let's create the alert manager custom resource itself all right so it's going to be very simple config so this is kind alert manager then we're gonna provide the version replica there is a way how you can create the highly available alert manager cluster as well but here we just use one replica and this is just a resource block for that alert manager also i included the ingress example in case you want to use it i'm gonna just leave it here same thing just close it and it also exposes primitives matrix so let's go ahead and add another target and let's create the alert manager all right so this is the series monitor for that alert manager which is the different well when you uh create the service monitor in the same name space where the the services you don't need to provide this namespace selector but if you have the service in different namespace you would need to use this much namespace and then provide the namespace where you deployed your service microservice or whatever it is all right so okay so it's ready and now let's deploy it alert manager all right cubes shell get pots monitoring namespace and yeah it's 7 seconds it's ready okay uh well let's use cube still get a cc in monitoring namespace and let's use this service to port forward it as well let's play that again and let's use monitor port forward and then that alert manager the difference will be well service name and the port which is 1993 let's go let's close this let's go to http localhost 1993 right yeah and this is our alert manager and if you go to the status you'll see your config and this is the cluster status cluster status disabled means it's not a higher highly available alert manager setup so it's just one single instance all right and well the finally i wanted to show you how can you create the alert so let's go let's go back here to alert manager right now and right now we're gonna have another target alert manager all right so we have the four targets now let me show you how we can configure the alert and in case we're gonna set up we're gonna break the parameter separator and i will show you that the parameters will detect that one of those targets went down and it will send the alert in the slack let's get started by creating the rule itself under the alert manager let's create the last file we're gonna call it the rule.yaml and that's the rule so um don't forget this particular label so this is not serious material this is the another custom resource definition parameters rule and this label has to match to primitives configuration so that's the rule selector and those two labels have to be the same and here under the rule we're gonna create the new group called general and this is the the alert name target down so it will just look for all targets available in prometheus and if one of those targets will go down for longer than for one minute it will send the alert also here the severity warning label important as well so here this particular label is used in my config so for example here this is they have to match because when when this alert manager uh see that particular alert with that label it will use this receiver and we'll send the alert to that slack since it's a slack receiver okay so let's go ahead and let me show you in the parameters under the alerts right now we don't have any alerts just yet let's go ahead and create it let's apply cube still apply alert manager and it's going to create the that parameters rule and also all of them are just a reg well not regular but kubernetes objects for example you can get you can type keep still get service monitors so you're going to get all the service monitors and the same thing for the primitives roles or instead of serious monitors you can use the this particular kind and get all the rules so this is the general rule and now let's go back to the parameters and let's refresh and well it will take some time a few seconds to update the alert manager config all right now we have the target and it's green it means well it's active but it's not triggered so and this is the query you can click on that parameters query and it says no results that's good and since there are no targets that are down and well when it's gonna detect one of the targets is down it gonna send the alert right so let's go to the status targets and let's break one of those so like i said we're gonna break the primitive separator and let's just go to the parameter sprouter to deployment and uh it knows that primitives that it needs to scrape this parameters operator on port 8080 and if you're gonna change the port it will no longer will be able to scrape it and the target will be done so let's update the parameter separator so it's like configured let's go back to targets refresh and all right so now it's down first of all it's going to be in unknown state for a little bit and then it's going to be in down state so let's refresh it again all right so the state is down and that's the error not able to scrape it okay and if you're going to go to the alerts now it's in the yellow and it's in painting state so it's gonna wait for one minute so if we're gonna click on that query again now we have the result that the parameter separator is down let's go back to alerts and and well i'm gonna show you that it's gonna send that alert to this lock let's go back here and let's wait like i said one minute and it will send that alert here all right so we got the alert notification through the slack and one hundred percent of targets of amethyst router down so all of them right now and if you're gonna refresh this page you're gonna see that it's in red it's in fighting state and like i said all of the primitive targets are down right now okay so and in the alert manager you can also click on alerts and you'll see that the alert is active right now and that's pretty much it for this video and please if you find this video helpful please like it and also subscribe to my channel thank you bye you

Channel: Anton Putra

Views: 11,624

Keywords: Install Prometheus on Kubernetes Cluster, prometheus monitoring kubernetes, grafana kubernetes, prometheus alertmanager kubernetes, prometheus monitoring kubernetes tutorial, prometheus operator, prometheus monitoring, monitoring kubernetes with prometheus & grafana (2/5), prometheus operator kubernetes, install grafana on kubernetes, kubernetes, anton putra, devops, sre, prometheus tutorial kubernetes, prometheus monitoring demo, alertmanager slack, prometheus operator explained

Id: mtE4migphGE

Length: 43min 9sec (2589 seconds)

Published: Thu Mar 04 2021