Prometheus Grafana dashboard for Kubernetes monitoring - Part 2

hello there this is rahul and in today's session we are gonna see how to set up prometheus server and grafana dashboard for monitoring your kubernetes cluster running on aws so here is the quick glimpse of what will be the end result of this session so you will be able to set up the grafana as well as the prometheus server running under kubernetes cluster on aws you will be setting a kubernetes cluster with the name test cluster one and after that you will also set up the grafana dashboard so this is the default grafana dashboard we will be setting and also we will be deploying an application inside our kubernetes cluster and we will be monitoring the status of that particular application that is going to be a j hook spring boot application inside our grafana dashboard so these are the main objective we will be achieving throughout this session so let's switch over to our desktop and start with our session let's go through the prerequisites which is needed for setting up the grafana and prometheus for our kubernetes cluster running on aws in this session we are gonna install each packages from the scratch so if you have any of the packages already pre-installed onto your system then it's fine you can skip that particular packages or otherwise if you are also starting from the scratch then you need to have these four packages uh the first one is aws cli second one is eks ctl for aws kubernetes cluster services third thing is kubernetes ctl that is kubernetes command line and the fourth one is going to be the helm chart along with the today's session i'm just gonna share this guide where i have already documented all the steps which is needed for setting up the prometheus and grafana dashboard for our kubernetes cluster so you can take a note of this link and here you will find all the instructions and the command which we will go through in this session the first package which we are going to install is the aws cli but the question comes like why do we need to install the aws cli so on the right hand side here you can see this is our aws environment where we want to set up our prometheus and grafana setup along with our kubernetes cluster but on the other hand this is our local development machine so this local development machines is going to be uh ubuntu because i'm using the ubuntu over here so to communicate uh with our aws environment we need aws cli then that is a command line uh utility provided by aws but this aws cli will not be able to communicate with our aws environment and for that we need reason access key and secret key so first of all uh we are just gonna install the aws cli and after that we are just gonna provide the region access key and secret key and once we provide these uh characteristics to aws cli then it should be able to communicate with our aws cluster or aws environment now switching to my terminal and on the left hand side you can see this is my terminal and on the right hand side the guide which i have prepared and which i have already shared with you so here you will find the instructions for installing the aws cli so this is going to be our first step and you need to be careful like what kind of operating system you are using so if you are using mac os then just follow these instructions for installing the aws cli and if you are using windows then use this instructions but since i'm using the linux or the ubuntu as my base operating system so i'm just going to follow these instructions for installing the aws cli so i'm just going to copy this command and this is simply a curl command which is going to download the zip file onto my local system and also one thing if you don't have the unzip utility installed on ubuntu this is specific to ubuntu then you need to install that utility and here you can see the file which i have downloaded so this is the zip file which i have downloaded that contains our aws cli so simply i'm just gonna unzip it so this is the command for unzipping it i'm just gonna clear the screen and then finally i'm just gonna run the install and you can verify the uh aws and cli installation by running the command aws version and here you can see we have installed the aws cli 2.7.4 after installing the aws cli the next package which we are going to install is the eksctl this is the utility which will help you to interact with your kubernetes cluster which is running on to aws so here are the instructions and the instructions is again going to be the same uh if you are using for the mac os then use this brew instructions or if you are going it with the windows then just use the chocolaty you need to install the chocolaty if you are using the windows operating system but here this is the command which which is there for installing on ubuntu or the linux operating system so i'm just going to copy this command paste it over here and simply hit enter and after that i'm just going to move this to user local bin so that it's available from the command line i'm just going to simply paste it and you can verify it by running the command ekcdl version i think version is the incorrect flag then it should be i think it should be version yeah so uh we have installed the eks ctl version 0.99.0 moving further into the guide the next thing which we need to do is we need to set up the aws credential so if you go back to our previous slide then you might have noticed that we need to set up the region access key and the secret key so till now we have only installed the aws cli but we haven't configured the region access key and secret key once we configure these three component with our aws cli then we should be able to communicate with our aws cloud services now the question comes like from where to get those details so switch to your aws account so this is my aws account and here first of all you need to select the region so i'm just gonna select the eu central one so you can make a note of these settings over here so first of all eu central one this is optional and this is not optional but this is like uh depends in which region of the world you are so if you are in asia reason then select some some of the uh data center which is nearest to you since i'm in europe that's why i'm choosing the eu central one second thing how to get the access key and the secret key so go to access key over here uh click on new access key and it will click on show access key copy these details uh these are super confidential but i'm just gonna delete this after the session so uh just remember not to share this detail with anyone otherwise it you might compromise with your aws account so copy this details over here after that what you need to do again uh go back to your guide and here you need to run the command aws configure copy this command over here go to your terminal and paste it over here sorry i'll clear the screen and you need to run the command configure remember we have already installed the aws cli that's why we are able to run the command aws configure now it is asking for the access id so i'm just gonna copy from my notepad and i'm just gonna paste it over here hit enter then secret access key i'm just gonna copy it again from my notepad i have a notepad open onto my other screen that's why you can't see it and the reason name so i'm just gonna choose eu central one and hit enter the default output format is optional so you can just simply hit enter and that should be okay i'll clear the screen all right so till now we have installed the aws cli and also we have configured the reason key reason access key and the secret key the next utility which we need to install is the cube ctl that is the kubernetes command line utility and that utility we are again going to install on our local development machine so everything we are doing we are still doing on our local development machine we haven't uh started any kind of a service on our aws environment so we are just preparing our local development machine so i'm again gonna switch back to my terminal and i'm just gonna open the guide also so these are the instructions for installing the cube cdl so you simply need to copy this instructions and execute it so i'm just gonna run the curl command first and hit enter okay after that i'm just gonna run the second command which is just changing the mode of coop city and after that i'm just gonna move this to user local pin and hit enter and you should be able to run the command cube city yeah okay yeah so we have successfully installed the cube ctl the next utility which we are going to install after the coupe ctl is the helm chart uh helm chart is needed because we are going to install the grafana as well as the prometheus packages using the helm chart i'm just gonna explain it later uh but uh just to make our local development machine ready we need to install this utility that is helm chart so again switch back to your terminal and also open the guide i'll clear the screen over here and just scroll it down and find the instructions for installing the harn chart so these are the instructions for installing the helm chart i'm just going to copy the curl command first sorry it just not copied so what i'm going to do i'm just going to copy from here and i'm just going to paste it over here so here i just executed all these three commands so that should also work so first of all we are just going to curl then we are just going to change mode and then we are just going to execute the installation uh bash script and this should install the helm so you can check it by running the command help and here you can see it is generating all the uh like available commands along with the helm so that means our help has been installed i'm just gonna clear the screen now after installing those packages we need to create a kubernetes cluster and for that we are going to use the utility eksctl and if you remember we have already installed the eks ctl onto our local development machine now this is the step from where we will be interacting with our aws services and what does it mean we will be creating our kubernetes cluster uh on aws using the eks ctl utility then question comes like uh how we can create a kubernetes cluster using the eks ctl utility so this is the command which i'll explain you but this is the command which you need to run from your terminal because we have already configured aws cli we already configured the credentials which is needed to communicate with aws and this is the eks ctl utility which will be creating our kubernetes cluster so the command is pretty simple first of all i'm just going to use the highlighter over here sorry i need to switch back to previous slide yeah so here this is the command eks ctl create cluster so this is these are the keywords which you need to put in after that you need to specify the name of the cluster so i'm just using the test cluster one that is going to be my cluster name uh for my kubernetes third thing which you need is the version uh so you need to specify the kubernetes version which you will be using so i'm using here 1.22 but depends like uh what version is available you can choose whatever suitable version available at that point of a time next thing which you need you need to specify the reason since we are working on aws cloud services so you need to be very specific like in which region you want to this cluster to be available so here i'm using the eu cluster eu central one so this cluster will be created into eu central one reason so here on the next thing which we need to specify is the node group name so here we will be using the worker nodes then after that we need to specify the node type so we will be using the t2 large so our ec2 instances will be of t2 large how many nodes we will be needing so we need two nodes and what's the minimum and the max so we will be needing at least two nodes and maximum of three so these are the parameter which you need to provide to eksctl so that it can create a kubernetes cluster onto aws now switching back to terminal and on the guide you will find the same command over here so you can simply copy this command and paste it over here and this command is exactly the same but here this command is not in so much readable format that's why i have already explained to you and here only one thing you need to change uh here i have used the version 1.8 so i think uh i i need to update this guide because 1.8 version will not work at least the minimum supported version is one nine and i will go with 22 which is i felt is quite lettuce so i'm just gonna use that one i'll update this guide uh after this session so simply gonna hit enter and it might take around 10 to 15 minutes or maybe more so be patient with it uh it will finish creating your cluster and you should see your we will see the aws cluster by logging into our aws console so i'll be back once my cluster is ready so here you can see my eks kubernetes cluster is up and running so uh this is my aws console and i'll just uh type eks over here just to check my cluster is there or not so i'm just gonna click on elastic kubernetes service and here you can see this is the test cluster which uh we just started using the eks ctl utility so i can see my cluster is there and it is active right now while installing the or while setting up my kubernetes cluster there was an issue which is reported on to my console and here you can see parsing cube ctrl version string and we need to fix this issue and for that we need to run the command by that will update the kubernetes config so the command is aws eks update cube config and the name of the cluster and this should fix this issue i noticed this issue while my setup it might be possible that you might not get this issue but in case if you if you see this kind of a issue onto your console then i would recommend running this command i'll update my blog post with this fix also so that it is handy and if any one of you are facing similar issue then you can simply run this command so i'm just gonna paste the command and hit enter okay that's been done so just a quick recap we have installed the aws cli along with the region access key and secret key after that we have installed the cube ctl utility to interact with our kubernetes cluster and also we have installed the helm chart and after that we have used the eksctl to create our kubernetes cluster and this is the command which we have used it for the next thing which we need is we need to install the matrix server so what does it mean by matrix server so here you can see this is our kubernetes cluster running onto aws but it cannot provide any data to prometheus and grafana by itself because there is no inbuilt functionality which can push this kind of a performance data to prometheus and grafana so we need to find that utility which can communicate with our kubernetes cluster and transfer those data to prometheus and grafana so that those performance data can be visualized onto our grafana dashboard so what is that utility so that utility is called as matrix server that is called as kubernetes matrix server and this is the command which we need to run to install that utility inside our kubernetes cluster so this is the command which we are going to use and the same command i have uh mentioned in the guide also so here you can see uh install kubernetes matrix server so this is the command which you need to run on to your local development machine because we have already installed cube ctl onto our local development machine and this cube circle command is going to communicate with our kubernetes cluster which we have recently created and that is our test cluster one uh if you remember this cluster so this is the cluster which is running there and now we need to run the cube ctl command that is kubernetes command line utility that will install this metric server onto our kubernetes cluster i'm just going to copy this command from here and i'm just going to paste it over here and simply hit enter and that's been done you can also verify the kubernetes matrix server installation by running the command this cube city will get deployment matrix server just copy this command and paste it over here and hit enter and here you can see the matrix server is ready one slash one that means it's up and running and there are a few errors you can see over here but you can ignore these error because here there was only one instance of matrix server and that is up and running so we just only need that matrix server uh which should be up and running all right the next thing which we need to do we need to use the helm utility and this helm utility is going to install the prometheus onto our kubernetes cluster so for that we are just going to run the command helm install prometheus but before that we need to add the prometheus repository into our helmet repository i'll show you what i mean so switch back to your terminal and i'll also open the guide over here so first of all this is the command for adding the uh prometheus repo prometheus helm repository into our helm chart repository so i'm just gonna copy this command uh oh i have already executed this command so here you can see helm repo ad prometheus community so this command is going to add the prometheus community repository into our local helm repository after that you can run the helm rapport update command so i'm just going to copy and paste it over here so this is going to update the repository definition so here it has updated successfully after that we are going to create a namespace and that namespace we are going to put as prometheus namespace so this is going to be our kubernetes namespace i'm just gonna hit enter i'm just gonna clear the screen and just paste the command so this is the command for creating the namespace with the name prometheus so the namespace has been created after that we are just going to uh install the prometheus remember we have added the prometheus repository into our helm chart after that we have updated the repository definition by running the helm repo update command and after that we have created a prometheus namespace and then we are going to install the prometheus so these are the steps which is which you need to execute to install the prometheus onto your local development machine oh sorry onto your kubernetes cluster which is running on to aws so i'm just going to copy this and paste it over here and hit enter and that's been done but we need to verify the status of our prometheus uh after installing it i'm just gonna clear the screen before we verify the status of our prometheus installation to verify the prometheus installation we are going to check all the resources which is running under the namespace prometheus and for that we are just going to run the command cubectl and for namespace and the namespace is prometheus because we have done the installation of a prometheus under the namespace prometheus so the resources which we are going to verify is the part services demonstrate deployment and the replica set so i'm just going to hit enter and this should put all the stats of our kubernetes cluster and the name space prometheus onto our terminal so i'm just gonna increase the size of my name terminal so here this is the command which we have executed and here you can see the status so the all the status are running and all the instances two slash two one slash one so everything is up and running here in the part similarly we are gonna verify the services so everything seems to be fine the demon set is also ready and the deployments are also available and up to date and similarly the replica set also those are up and running so that means we have successfully installed our prometheus using the helm chart onto our kubernetes cluster there is one more utility uh to verify our prometheus server by accessing the ui of our prometheus server but since uh this is the whole setup which i am doing over here i am doing on my vagrant machine so i am running a virtual machine inside my ubuntu machine so in this terminal or this machine which is over here here i cannot here i do not have a browser installed inside that machine it's a very virtual machine very basic virtual machine uh where i can only access the terminal so that's why i cannot access the ui but we can run this utility and we can use the curl command to verify this endpoint so i'm just gonna run the port forwarding first of all so it is just simply forwarding the port uh of my prometheus server which is running on to aws environment onto my localhost on my local machine so this is just doing the port forwarding so here i have done the port forwarding and i can access the same port onto my local host so i'm just gonna switch to my another terminal and i'm just gonna run the command curl localhost 1990 slash graph and you can see the same thing in this this screenshot in my guide and simply hit enter and here you can see this is the output which i am getting so it is just generating the html output but that means you have successfully installed the prometheus server onto your kubernetes cluster so if you turn off this port forwarding and go back to the terminal and if you rerun this then it should not work so it's not working so that means we have installed the prometheus server and we should now next move to the next step that is install grafana and again to install the grafana we are going to use the helm utility and this is the command which we are going to use helm install grafana and followed by these parameters and here we will be setting the initial password that is eks exclamation and then awesome so this is going to be the default password for our grafana dashboard which we will be seeing later once uh everything is ready uh so let's switch over to our terminal and also open the guide so here the installation is again going to be the same first of all we need to add the repository that is grafana repository into our local helm repository so this is the command so our grafana repository has been added we need to update the repository definition by running the command helm repo update that's been done and then we are just going to install the grafana but before that we need to create one data source because prometheus and grafana both are different and to communicate with prometheus we need to create a data source so this is the data source which we are going to create so simply create a yml file over here just copy this name of the file you can create any name of the of your choice but since i'm just using a relevant name that's why i have put that as a prometheus data source because grafana needs a data source uh from from prometheus so this is the yml file which i'm going to create uh simply hit enter and here you can put this definition over here so this is the data source definition so since grafana is just a visualization uh thing it just creates a dashboard but it needs a data and that data is provided by a prometheus so we need to create a data source definition between grafana and prometheus so that's why we are just going to create this data source yml and after that you can just simply save and quit and our prometheus data source has been configuration has been created in the form of yml the next thing which we need to do is we need to create a namespace if you remember we have previously created a namespace for prometheus that is this one and we have done all the installation and for prometheus under that namespace but here in grafana we are just going to create a one more namespace with the name grafana and we are going to do all the installation related to grafana under this namespace so i'm just going to copy this command i'm just going to clear the screen and paste it over here and hit enter so this is going to create our grafana namespace that is kubernetes namespace i'm talking about and here these are the command for installing the grafana and here you can see this is the data source yml which we have created and this will be responsible for communicating and fetching the data from our prometheus server and showing it on our grafana so just copy this command and i'll put some spaces over here and paste it over here and simply hit enter and i'll be back once this installation is done so here you can see our grafana installation has also been done i'll just clear the screen over here let's verify the grafana installation by running the command cube ctl get and then the name space and the name space is going to be the grafana so cube city will get all and followed by the namespace and remember we have run the same command for the prometheus inside our prometheus namespace so i'm just gonna run for grafana and here you can see there is one board which is running and the status is also running and the deployment is also ready and that is also available and the replica set is also one and that is also running so our grafana installation has successful and that our grafana pod service deployment and replica set all are running now the question comes like how to access the grafana dashboard so for that uh if you remember that we have run the command cube ctl get all so it should show all the statistics of our grafana and here you can see this is the services status text each which you can see i'll just extend it a bit more over here so that you can see a bit more clear so here this is the service and this service is running on a load balancer and that is accessible on this external ip so this is the external ip on which you can access your grafana dashboard so this external ip will be accessible from anywhere and if you are able to access your grafana dashboard that means you have successfully installed prometheus kubernetes cluster as well as grafana onto aws i'm just going to copy this external ip and go back to my browser and paste it over here and you can hit the enter so here you can see our grafana dashboard is loading and this external ip you can share it with anyone and that should be accessible and here you can enter your default username and password if you have come this far that means you have successfully set up your kubernetes cluster which is running on aws and also you have successfully set up the prometheus along with the grafana and all these three components are working fine and your grafana dashboard is ready so now what we are going to do we are just going to enter the default username and password and then we are just going to see uh some dashboards so the default uh username should be admin and the password should be from our if you remember we have created prometheus data source yml and inside that we have set up a password that is eks awesome so i'm just gonna copy that i hope it is going to be the same password i'm just gonna switch to my dashboard and paste it over here and click on login and yeah that is the default password which we have set inside our yml and you need to use the same password and then if you are using it for production purposes then please change this password and keep something else so this is the uh login of my grafana dashboard and this is the default view of my dashboard now the next question comes like after setting up your username and password uh how to import or how to create a dashboard so one thing which i like is i can reuse one of the dashboard which is created by someone else and for that what you need to do is you need to click on this link on the grafana open source community where someone has created similar dashboards for kubernetes and if you found that dashboard useful then you can simply use that dashboard using that number so here i liked one of the dashboard which is which has a number six four one seven i i'm just gonna show you so i will increase the size of my window over here so here you can see there are lots of dashboard available over here so that's let's take an example nginx ingress controller but here you will find a unique number associated with that particular dashboard that is 9614 so if you like this dashboard you can simply copy this number and you can come to your grafana dashboard and click on this import option here and then you can simply enter that number and you can simply download that particular dashboard but since i'm interested uh into one dashboard which i liked for my uh this session and the dashboard name is kubernetes cluster prometheus i'm just gonna hit the enter so i'm just gonna click on this and here you can find the 6417 so this is the number uh assigned to my particular dashboard so i'm just going to copy this number go to here i just went to this plus sign and here there is an import sign so just go here paste that one click on load and then uh your you should be able to load your uh dashboard over here but here you need to select a data source and here i'm just gonna use the default data source that is going to be the prometheus and then you simply click import and here you can see this is our dashboard uh for our kubernetes cluster and here you can see all the statistics like prometheus server which we which we have installed which is up and running and here we have some other statistics like alert manager stat matrix and metric server graphana core dns so this is the default dashboard which i like which i have simply imported with the number so you don't need to recreate this kind of a dashboard by your own so you can simply search for some suitable dashboard from here copy the number and put it over here and you should be good to go with it so these are the instructions which i have already mentioned in the guide so how to search for the grafana dashboard number and how to import that particular dashboard so i hope you should be able to import a suitable dashboard which you find useful for your use case so this is the way you should do it the next and the last thing which i want to try out is i just want to deploy something inside my kubernetes cluster and i just wanted to monitor that particular service onto my grafana dashboard over here so here's something something has to show up over here once i deploy inside my kubernetes cluster so for that what i'm going to do i'm just going to deploy one springboard application inside my kubernetes cluster so this is the manifest you can copy it from here so this is a simple yml for creating a kubernetes deployment and also running as a service so i'll just copy this whole thing and go back to my terminal i'll clear it i'll just okay i'll just create uh the file over here i'm just gonna name the same maybe i paste it over here and after that i'm just gonna simply copy the whole yml definition over here and save and quit i can show you once again also so it should be with the name okay and as a springboard deployment so here i'm just creating a service kubernetes service which is running on a node port on a port 8080 and this is the deployment and this is the springboot docker container image which i am fetching and it is going to deploy on a port 8080 so uh i'll clear the screen and you can verify this is the yml which i have created and i'm just gonna execute or apply this yml onto my kubernetes cluster using the cubectl utility so the command for that is if you go back to my guide which should be here so this is the command which you need to run and the command is cube ctl apply dash f after that the name of our yml and simply hit enter i'll clear the screen over here now let's verify the status of our deployment uh of our spring boot application so i can run the command cube ctl get deployment and the name of my deployment that is j hook springboard i'm just going to simply hit enter and here you can see uh there are two replicas set and both are up and running so my deployment of my spring boot application inside my kubernetes cluster is successful and after refreshing this grafana dashboard over here and i should be able to see that springboot application so here you can see the springboot application uh which i have deployed inside my kubernetes cluster and it can see the value there are two replicas set of my deployment so that means uh we are able to set up our prometheus and grafana dashboard we are able to deploy our application and as well as we are able to see uh the states uh of our or the status of my deployment inside my grafana dashboard so that is going to be the last step and i just wanted to take the complete use case uh so that you can create prometheus and grafana dashboard for kubernetes and also you can deploy something inside your kubernetes cluster and monitor using the prometheus as well as grafana dashboard so that is all about how to set up your prometheus server and graphana dashboard for monitoring your kubernetes cluster so if you are interested into the similar content on devops then considering subscribing to this channel and if you have any question related to the today's session then those are most welcome and you can put down into the comment section so see you into the next session of our devops till then take care and bye

Channel: Rahul Wagh

Views: 12,192

Keywords: prometheus, grafana, kubernetes

Id: 3Xs49Urq16M

Length: 36min 41sec (2201 seconds)

Published: Mon Jun 20 2022