Monitoring Azure and AWS with Grafana | Azure Monitor & CloudWatch Integration with Grafana

hey everyone I hope all of you are doing great so in this lesson let's go ahead and see how we can monitor Cloud environments using grafana uh so I have Microsoft Azure instance and also I have AWS we are going to go ahead configure each one of these and then we arey we are going to create some dashboards or probably we are going to leverage some existing dashboards which is available on grafana Marketplace so first of all let's go ahead and uh click on connections and go to add new connection now here if you go to Azure uh you can see we have option of azure Monitor and Azure devop so we are going to go ahead and select Azure monitor uh if we want to monitor like uh virtual machines anything which is basically there in our Azure environment now here let's go ahead and click on add new data source now once you do that you will notice that these are the things which we need we need tenant ID we need application ID and client secret so it means we have to have these these things first in order to configure Azure to be monitored in Gana so let's go ahead and and create all of these so I'm going to go to Azure and all of these actually needs to be done using App registration so either you can go ahead and search for app registration here and then under Services you'll be able to see app registrations or if you just have uh available on your dashboard just go ahead and click this so I'm just going to go ahead and click on app registration now here you'll be able to see owned application and all application so here you can see I do have a application created here but I'm going to go ahead and click on new registration for registering a new application and for some reason uh this web interface is quite slow in my area I'm not sure even the internet speed is quite good but this is quite slow anyways um so we are going to register an application we need to provide a name I'm going to call it grafana and uh probably let's call it grafana we don't need to fill anything else these are the things required when you are creating this authentication for web application or some other purpose but in our case the these are optional we can just go ahead and leave it as it is let's go ahead and click on register now you can see we have got application ID we have got tenant ID so two things are sorted out now the other thing which we need is this client secret so here you can see in the client credential we have option of adding a client certificate so let's go ahead and click there and here we can go ahead and click on new client secret we are going to give it some description uh let's call it grafana your secret and here let's click on ADD now here you can see successfully updated application blah blah blah all these credentials so here you can see the good thing is we have got this credential so I'm just going to copy it and for the time being I'm just going to keep it here in the address bar and now uh we have basically got our application registered so first thing is what I can do is I can go to API permissions and here you can see by default it has user read permissions so that's fine the other thing which we need to do is let's go to subscriptions here you can see under services we need to go to subscriptions and we need to give access to this subscription basically so here you can see I have pay as you go subscription uh this is a paid account uh but if you have free tier or whatever just go ahead and select the subscription and then uh you can go to access control IM and here if you scroll down you should be able to see Grant access to this resource so let's go ahead and click on ADD roll assignment and here we are going to go ahead and give reader access because uh there's no point of giving any additional permission reader is enough so let's go ahe select it and click on next now here we can see we need to assign it to some user group or service principle so we are going to go ahead click on select members and going to search for that application which we registered so here you can see if you just you know click on select members you won't be able to see all the applications which you have created I don't know why Microsoft hides it it should have been shown here but we don't really see it so basically we need to search it by name so I'm going to type Gana here in the select and here you can see we have grafana popping up let's go ahead and click it and then click on select now we have option of next or we can just go to review plus assign and then click here again and adding Ro assignment is completed so everything looks okay now we can go ahead and click on app registration again to see the detail of our applications uh let's go to all applications click on Gana and let's go ahead copy application ID we are going to put application ID here in application ID then we need tenant ID so let's go ahead and copy this tenant ID paste it here in the tenant ID and then we need Secret so let's click here and let's copy this secret I thought I won't be shown this again uh because in this is what happens in AWS but we still can see it so anyways I'm just going to remove this tab I just copied it here so just go ahead and put that secret here uh looks like I paste it at the wrong place let's go ahead and paste it here in the client secret click on load subscription if everything is good you'll be able to see the default subscriptions or basically you'll be able to see all the subscriptions which are available and here you can make one of the subscription as default one so I'm going going to select uh this one because I have the only one subscription which is available so we are making it as default and then let's go ahead and click on Save and test now if everything is okay it has the permissions you should be able to see the successful message just in case you don't see the successful message you are probably going to see some error logs which is going to let you know uh what is the issue most probably it is the permission issue so make sure you have given subscription access or subscription read permission to the application which you just registered now here you can see everything is successful now if I just scroll up I can see dashboard section as well now here you can see Microsoft or basically I can say graan has provided some dashboards which are available by default and I should be able to just import any of these dashboards and start using it so let's go ahead and select this one which is azure insights Virtual Machine by Resource Group uh let's click on import and here you can see we can now click on this link and we should be able to move to the dashboard so here make sure you have Azure Monitor and WR subscription and here you you can see I mean uh by default I had the resource Group selected under which I don't really have any resource so here you can see if I go to resources or yeah if I go to Resource groups here you can see I have two resource groups available and those two resource groups are being shown here now under default Resource Group us I don't really have anything if I click on here on the other resource Group you can see I have like so many items I have a virtual machine public IP and so many other things if I click on default Resource Group I don't have anything so that's the reason we are not seeing any data which is related to Virtual machines so I'm going to go to grafana and uncheck this one and select this one my Resource Group let's just refresh it and for some reason it is still not showing any data I don't know the reason so what we are going to do is just go ahead and click on ADD and click on visualization we are going to go ahead and create our own visualization so let's select Azure Monitor and here we are going to go ahead select a resource and here you can see we have got the resource and I can see my VM which is the name of my virtual machine it is called my VM so I'm going to select it and click on apply now if you want to use variables you can go ahead and click on advance and you can re replace these with the variable names and then you'll be able to able to make a selection of these using grafana variable but for now everything is okay let's go ahead and click on apply and now in the matric name space you can see I have viral machines in the matric I'm going to select CPU utilization or percentage CPU is fine as well everything looks okay you can see we have got the data just scroll down everything looks okay here as well I don't think we need anything else is there anything in dimension nothing so we don't need to go ahead and select anything else now here you can play around you can uh change title you can go ahead and make other changes for now let's click on apply and you can see we have a panel ready now let's go ahead and probably create one more panel I'm going to go ahead and duplicate it and then going to click on edit now here rather than percentage let's see if we have something related to network we do have something related to network in so let's select Network in total and then let's go ahead and duplicate this query and in the query B I'm going to show Network out total so we here we can see networking how much traffic has come in how much traffic has moved out now here on the right hand side we can just scroll down you can see this connect null values because there are sometimes which there's nothing has moved in or moved out so we see null values so I'm going to go ahead and click on connect null values to always so we can have like Smooth trend line so let's go ahead and click on apply and here you can see we have Network in network out percentage CPU now for some reason this one the one which already exist uh that dashboard didn't work I don't know why uh but what we can go do is we can go to grafana dashboards which is this one graana.com grafana dashboards and here we should be able to see some of the existing dashboards you can see we have like all the dashboards which are pre-created and we can just go ahead select the data source which we are interested in like in this case we are interested in Azure monitor so I'm going to select that one and I'm interested in this Azure virtual machine dashboard so let's select it and go ahead and click on copy ID to clipboard and then we are going to go ahead here going to move to import dashboard now when we clicked on import dashboard basically it is asking do we want to save this existing dashboard so I'm just going to click on discard I don't want to save it and here we are going to give the ID of the dashboard which we copied which is this one and I'm going to click on load here you can see now everything looks okay a dashboard a folder with the same name already exist so let's go ahead and change the name to one and Here Also let's go ahead change the U ID let's call it one click on import and we should be able to see the data brilliant so in this dashboard we can see the data Yeah so basically I was looking for something like this network in bytes versus Network out bytes now here you can see this plug-in seems to be like older one and seems to be deprecated so you don't need to worry just go ahead select the time series which is the newer one and then click on apply and we can do the same for all the plugins which is expired now here you can see for CPU usage uh we don't see the data at times you do get some stupid error I don't know why we are getting this here error uh so we are just going to go ahead uh click on edit and here you can see in the resource name it is using variable name so that's fine we are going to go ahead click on apply and here I'm just going to select this matric name name space again and just try to I'm basically trying to refresh it and see if some data pops up but for some reason I don't know what's the reason but this data is not coming so I'm just going to click on resource name remove the variable going to go ahead select the resource name which is my virtual machine click on apply and here this is fine here you can see we now have metric coming in so I'm going to go ahead click on percentage CPU and here you can see we have CPU usage data now so you can see this is fine now and you can do the similar thing on available memory so you can see how quickly we have our dashboard up and running and now you can go ahead show your creativity play along with this U go ahead change the queries based on your need create alerting and you have dashboard ready like within 1 hour so let's go ahead now and I'm not going to save it let's go ahead and click on homepage click on discard and now I'm going to close Microsoft azure and we are going to move to AWS and we are going to see how to monitor AWS using grafana so let's go ahead and click on connections go to add new connection and here if I search for AWS uh nothing pops up other than AWS iot sitewise this is not something I'm interested in let's search with Cloud watch and here you can see in the data source we have Cloud watch so click on cloud watch click on add new data source now here if you see uh what what are the things we need we need authentic ation provider and here you can see we have accs and secret key option so I'm going to select that one and everything else seems to be optional so now here we need access key ID and secret access key now something similar to what we needed in Azure so I'm going to go to AWS and then here we need to search for IM am because we don't have any app registration things here in AWS we need to go to IM and create a user there and we'll be able to create accs ke and secret access ID for the user so let's go ahead uh you was signed out for some reason it has signed me out I'm just going to go ahead and click on reload so that's fine we have logged in again now I'm going to go ahead search for I and here you can see under Services we have Im so let's go ahead click on IM and it should take us to the interface where we can see user groups and users so uh I'm just going to click on user groups so you can see I do have a user group which is available and If I go and click on this user group you can see this is having some permissions if I go to permission tab you can see it is having this Cloud Watcher event full access so this is something which you will need to have so either either you can attach this permission at user level or rather I would suggest you go ahead create a group and give this cloudwatch access permission to the group and then you go ahead click on user and you create a new user which I'm going to do right now going to click on create user going to call it grafana AWS and do we want to provide this AWS console access to this user so this is not required we are just going to leave it as it is go ahead click on next and here you can see option of adding this user to the group now if you don't want to create a group and give the policies there you can just go ahead and click on attach policies directly and you can go ahead search for like uh cloudwatch and just give Cloud watch uh Cloud watch full access just to make things easy but for my case I have kept it organized I'm going to add this user to the group and this is the group which I want this user to be added in so let's go ahead click on next next and if you want to create some tags you can do that I'm not going to do that go ahead click on create user and we have user created which is grafana AWS now if you click on the user you can see we have option of creating access key so let's go ahead click on that and I'm just going to choose application running outside AWS here click on next give it some tag value let's call it grafana AWS user click on create access key and here you can see we have access key and secret access key available so I'm going to go ahead uh select this key and going to paste here and then I'm going to go ahead click on this value paste it here and everything looks okay let's go ahead click on Save and test cloudwatch Matrix query field missing default region okay so we do need to provide default region I'm going to give Us East one this is where most of my ec2 instances are so I'm going to select that one click on Save and test and should be able to show the green signal here you can see it is showing green so everything is okay now scroll up here you can see dashboard section available as well which is having some uh dashboards available so Amazon ec2 you can go ahead and click on import to import it I have already imported it so I'm just going to click here on the dashboard and we should be able to see the data popping up here so this one looks much better than the Azure one because in Azure we were getting some error and the default dashboard was not running but here you can see we do have default data coming in and we are able to see all the details ready made so that's all basically that's how you go ahead and U create or configure different cloud data sources in Gana so now what I'm going to do is I'm just going to to go ahead uh to the users going to delete the user which I just created because sometime people are able to notice the secret key and they try to attempt that uh using my secret ID and value so I don't want that to happen I'm going to do the same for Azure as well so I'm going to go to Azure and there we are going to go ahead and go to app registration we are going to remove the apps which we created so there is no unauthorized access to uh my cloud environment so make sure you do that as well and make sure you always enable multi- authentication because uh multiactor authentication is going to save you uh from hacking or things like that so here you can see we can go to all application graan aure let's go ahead click on delete and here you can see I have option of deleting it let's go ahead delete successfully deleted so everything is okay everything looks good now so that's all in this lesson see you guys again in the next lesson

Channel: Vikas Jha

Views: 2,632

Keywords: monitor azure with grafana step by step, monitor azure with grafana, monitoring azure vm with grafana, azure monitor integration with grafana, azure monitor grafana dashboard, azure monitor grafana datasource, azure monitor grafana variables, monitor aws with grafana, cloudwatch grafana, cloudwatch grafana dashboard, cloudwatch grafana plugin, cloudwatch grafana integration, cloudwatch grafana query, cloudwatch grafana variables, cloudwatch grafana logs

Id: Y61Obc6Bs6Y

Length: 18min 3sec (1083 seconds)

Published: Tue Nov 21 2023