How to collect metrics and create dashboards using Grafana, Prometheus and AppMetrics in .NET Core

hello everybody I'm Nick and I'm gonna show you how you can create a beautiful dashboards and monitor your application using app metrics Prometheus and Griffin these might be three unknown words to use I'm gonna go through a quick architecture design on how all these rings hang together in order to provide us with beautiful dashboards like this one right here if you are familiar with these three technologies you can skip to that time stamp and move to the implementation part so the first thing that you are probably familiar with is a dotnet API so this is probably gonna be a nice peanut core API and it this is nothing we haven't seen before it can be just a Web API it can be a blazer application even a console API it can actually be anything it doesn't have to be a spinet core what it needs to do is to expose its metrics through a forest last metrics endpoint so it needs to have an endpoint but even the console app can do that it can have a matrix endpoint and this endpoint will have everything formatted in a very specific way for Prometheus and I'm gonna show you in the video how Prometheus accepts data and how they should look like and now this is where two things come into place the first thing is from if it's and Prometheus is this file over here Prometheus all it released is a time series database that our time goes by it collects some data now how does it collect the data there's actually a couple of ways you can push data to it but this is not a recommended way the recommended way is for Prometheus to actually be configured to scrape that metrics endpoint every let's say one minute or if you want to make lower like 15 seconds which i think is a default when you install it or something like that so Prometheus will go in your metrics endpoint and we'll say okay give me all your metrics and I'm gonna take them here and it's gonna store it and then once it stores it it can also expose it to other services as a data store and one of those services is Griffin on Griffin I can be configured to use Prometheus as well or we're gonna go with we're gonna have a metrics endpoint here and we're gonna show you how you can configure that to output data in a way that promises likes it and can work with it and then we're gonna configure Prometheus to go do that pick up the data store it internally and then graph on ax will come and take the data and we're gonna make beautiful dashboards out of that so let's go into the video this video is part of my dog Nicole series so if you don't wanna miss any episodes please subscribe bring the certification well to get notified when I upload a new episode okay so with that out of the way I want to show you what I have here I have a single API which is just a single controller API with a post endpoint to create customers and then a get in point to get a customer by ID and I get endpoint to get all the customers and it's the very same one that I used in the X unit and mocking videos you can find all the code in the description down below if I go to postman I can show you what I have here if I just hit the customer's endpoint currently there is one customers eyes me and I can go ahead and create a new customer in this endpoint and as you can see the customer is now created and if I get all the customers now I have two customers very straightforward very basic stuff now what I want to do is I wanna integrate app matrix into this application so I'm able to extract some meaningful metrics for Prometheus to scrape so the way I'm gonna do that is I'm gonna go to the dependencies and I'm gonna add a few packages the first one is the app dot matrix dot asp net core and it's this one here I'm gonna use pre-release versions for all of them because they have some improvements but by the time you see this 4.0 might be the major version if you don't want to use a preview use three point two point zero it's gonna be the same thing basically and you can go ahead and install it and then the next thing we need is the dot endpoint package this one right here so I'm gonna go ahead and include that then the next one we need is the tracking and I'm gonna explain exactly what this tracking is in a second it might look like a lot of packages but app metrics did a great job to separating all the functionality to different packages so you don't have to include a huge one you just select the ones you want to obtain and then the last one is not a spirit core you go again up my up dot matrix and then for matters and Prometheus and this will allow us to format our matrix from up matrix to promise this compatible matrix and these are all the packages we need and now the way we can simply set this up is we can go to the start up dot CS and up here I'm gonna add something that if 4.0 is actually fully released you probably won't need to do but because there's an issue with the synchronous i/o that up matrix is doing behind the scenes I have to configure kestrel to not throw an exception this is not normally part of this video but I'm gonna have to exclude it just to make it work but again this is not up matrix specific you probably don't need to do this by the time you watch this video if you get an exception please do it and I'm gonna say that I need to allow synchronous i/o to true ok and now that I configured Carol to allow that all I need to do is say services dot add matrix and this will register the I matrix interface in my application and then I can go to the program dot CS and on the create host builder method if you don't have that it should be in the create default builder method here you can say dot use matrix web tracking and this is where the tracking package is coming in it will add a lot of API generic metric collection automatically for us so we're gonna see what these are and then you also have to say use matrix and we're gonna configure that now we're gonna say options and then options dot endpoint options equals endpoint options if you lambdas here and you have the same matrix text endpoint output formatter this is the way we're going to output the text version of our matrix because Prometheus can export in protobuf and in text but because we want to see what we're doing I'm going to use text for this video obviously protobuf is more efficient and I'm gonna say new matrix Prometheus text output format Wireless marshal and then the next thing I'd say is endpoint options dot metric endpoint output formatter and I need to say that new matrix Prometheus protobuf output formatter to enable protein off as the default one and the last thing I'm going to do is I'm going to disable the environment info endpoint so environment info and point enable equals false so here you can see all the configurations that I did and this is everything I actually need to do on the API front for the default matrix and let me show you what I mean by default matrix I'm gonna go ahead and run this application now and in here I have the matrix endpoint which I cannot use because this is protobuf and we cannot actually read prod about here but if I go for the text version of these matrix and I say send you can see that these are now formatted in what Prometheus can actually read and you can see them getting a gauge here a counter a summary promethease support for different types of matrix this is a gauge which is a value sensor we can go up and down imagine it like the speedometer in your car but you also have a counter which can only go up and then you have summary and histogram which I don't really want to go in because it's a bit more complicated and harder to explain and we won't really need it for this video but if you want me to make an in-depth video on Prometheus please leave a comment down below and I'm going to explain in detail what summer in histogram is and also counter engage and there are also a few conventions from how you should name things counters you usually have a total suffix as you can see these are just all the basic out-of-the-box ones but if I go to the API so here and I execute it a few times so I hit that and then I also create another customer here and I go back to the matrix you will see we have even more matrix here because more things got triggered in my application so we have more summaries so we basically have a few out-of-the-box basic matrix and what we want to do now is we want to configure Prometheus to scrape that end point and then graph on ax to talk to Prometheus and populate its data source to show that graphs so how are we gonna do that we're gonna go to Google Chrome because that's what I'm using and I'm gonna say Prometheus download I could do this in docker but because you're probably gonna have to install this as a service somewhere on a box same for Griffin I'm gonna show you the actual process as if I was just having it locally but normally I would use docker so this is the first thing you want to click you're gonna download they don't select the stable release not the Aussie version and I got a pic windows and I'm gonna download it and we'll also go in a new tab and I'm gonna say graph Anna download and I'm also gonna download Griffen Griffen is another thing you you will need to host they're both free by the way and we're gonna pick windows and I'm gonna select the zip file because I don't actually want to install it on PC and then I have a folder here where I'm gonna extract them and I'm opening this up and this is everything you need to extract in this folder so I'm just gonna go up and I'm gonna quickly extract it as you can see this didn't take that long and then I'm gonna do the same for Prometheus so I'm gonna have a folder with both Prometheus and Griffin the services Griffin is extracted I'm gonna do the same with Prometheus now extract that here and what I'm gonna do is I'm gonna first start Prometheus now Prometheus has this executable that you need to run but I also need to configure the scraping staff for Prometheus so what we're gonna do is I'm gonna drag and drop that in ryder and ryder will detect the Yamal format and as you can see it's a Yamal file which has some global variables such as the scrape interval and evaluation interval this means that every 15 seconds Prometheus will scrape the endpoint we specify to get more information and put it in its database to display and the default jobs is Prometheus itself because primitives actually exposes some metrics for itself but we can go ahead and add another job and I'm gonna say job name and this will be for us the I don't know customer's API and then I'm gonna say static configs and I'm gonna add the targets and the target we want to hit is localhost 5,000 and that's it and I also need to add of course if that in point is under authentication you can have basic auth and Barry talked enough to allow Prometheus to authenticate with your API I don't have any so I'm not going to use any what I do need to do though is I need to specify a different path because I'm looking at use the metrics I'm gonna use the metrics text and now this will be able to detect it and with that out of the way I'm gonna go back to the folder and I'm gonna start Prometheus and as you can see prometheus is ready and if I go on the browser I think the default path to access it is 1990 and as you can see this is prometheus so if we are successful in collecting the data I should be able to say something like I don't know let's just give it a second yeah here we go application HTTP requests active and it will give us the value so it's already detected our app matrix API and it started collecting information which is awesome now the next step is Griffin and Griffin are actually doesn't need anything we just go to the bin folder and we execute the graph on a server and you're going to do some magic and once that is finished Griffin is accessible in localhost 3000 that's a default port now we're gonna need to login is just admin and admin the default one I'm not gonna change that you go away and now we're gonna be asked to add the data source and our data source in this scenario is Prometheus the time series database that is collecting the data from our API so I'm clicking that and I'm gonna point graph on to my Prometheus instance which as you can see here is 1990 so I'm gonna copy that and paste it here and we're gonna change anything else I mean I could change the name but that's totally fine again if you have more indication you can configure it here scrape intervals no log and I'm just gonna say save and test and my data source is working so it's detected and working fine and now I can actually create a new dashboard these specific metrics package that we added for up metrics actually comes with a graph on a dashboard out of the box and we can import a dashboard by going on this plus button and then import and we need to specify an IB or a URL from Griffin accom now if I go here and I say graph Anna up metrics Prometheus I will be able to find this web monitoring dashboard downloaded 4,000 times and all I need to do I mean it is a bit old last of a three years ago but I just wanted to show you how we can quickly visualize an import and nap metrics dashboard in your case you might want to tailor your dashboard around your needs and that's what what I recommend I don't want to spend time creating every single thing in the dashboard manually I just want to show you how quickly it is you just copy this number here and you go to this and you paste it and it automatically detects everything about this and all you do is you select the folder which is the general and then you select the data source from if you think you imported and if i zoom out a little bit you can see that it automatically detect the environment the application the data source everything and it shows me stuff and if I go ahead and I go to the API and I start spamming it with requests as you can see this is quite a few I should say but in the dashboard that my rpm will go up and sure enough it's already detected that my requests per minute now are 31 and this is what comes out of the box I think it's potentially too much because you cannot say it without scrolling also you would need to tailor this around your needs but this is a good way to start and I just want you to be aware of it now that's great but I want to create my own dashboard and I want to show my own things so let's quickly do that and let's discard those changes so everything starts with a new panel in the new dashboard over here and you have to say let's say add query and you have to specify what you want a query for now gravano by default will detect through prometheus what metrics we have and we can go application and get all these application metrics for example something that we have here is the transaction count and this essentially is a request count so if I click that you will see that we have something like 56 now what I want to show is how many requests I have per minute on my API how can I do that let me just cut this I can say I want to use the increase function and I'm gonna say use this metric and aggregate for one minute and I'm gonna say enter and as you can see now I have four here and these four are because we're scraping every 15 seconds the 60 seconds in a minute obviously so this is for requests per minute just for Prometheus to do the scraping and everything you see outside of that is the request that's happening on our system so this is why we have four four four and then 30 here assume this a little bit and then I can choose visualization so a graph is fine but you can have anything else customers you want you can have a start you can have a gauge you can have a bar gauge you can have so many different ones and if you want me to do an extensive videographer now please leave a comment down below because this is a very fascinating tool and I really like it so general is where we give our pun and name and I'm gonna say rpm for past actually I don't need to say for past 24 hours I'm just gonna say rpm and I'm gonna show you why I don't need to say past 24 hours because I can go to queries and I can say relative time 24 hours and this will automatically show them stuff for the past 24 hours now I don't actually want to go 24 hours because this will make this very very small let's just do 6 hours yeah this is potentially be good for visibility and then you can also set up an alert for that I won't touch alert up all on this video there is another video coming for alerts and grow fauna so I'm gonna leave that outside but just know you can do it and then I'm gonna say save dashboard and I'm gonna say API or actually customer API and I'm just gonna save it and here I have my dashboard with my one graph I can make this smaller bigger I can do whatever I want with it I'm just gonna leave it here for now now this is all with out-of-the-box metrics what happens if you want a monitor or something different like how many customers were created in a day or in the past five minutes or how many database connections were created well this is where the I matrix interface comes into the picture and let me show you what I mean I'm gonna stop the application and make this smaller and now I'm gonna add two metrics that I want to collect first I wanna go like how many customers were created so here I need to say that I create the customer and collect a metric how will I do that first I do private read-only I matrix this is the interface that I registered with the add metrics call here in my services and then I'm gonna inject that from the constructor and all I need to say enter the created customer is matrix dot measure dot counter and then I'm gonna say increment and I need to give the customer options for the counter or gonna increment now I could just say new customer new counter options and boot some stuff here but not only this is ugly but it's highly recommended that you centralize where you stole this and for that I'm gonna create a new class and I'm just going to do top level for now I could actually just create a folder called matrix and put it there I'm gonna do top level and I'm gonna call it matrix registry and I'm gonna add yeah okay and in this matrix registry I'm gonna make a public counter options and I'm gonna say what we want to collect oh yeah created customers counter and that will be a new counter options and the options will be the following first I need to specify a name so created customers will be the name and then I need to specify the context so the context for this we'll probably just be customers API you can also specify the unit if you want so you can say that I want to collect this value in bytes and calls and commands in errors in items or whatever we can just go with request to be honest buddy for this specifics now it does really matter we'll just gonna go with calls and that's it so we have that here and let me just make this static and now I can go back to the controller and say matrix registry dot created customers counter and this will increment the specific metric and the other thing I want to do is if you remember we created in the repositories I think it was a IBB connection factory where we create our connections so what I will do here is I'm gonna say every time I open a connection or actually create a new collection let's collect a metric so in order to do that I need to say private read-only I metrics again and I'm gonna inject that through the constructor I might need to actually change that in the startup that's fine and again metrics dot measure dot counter dot increase and I'm gonna go ahead and create another counter I'm not gonna call it counter options called created DB connections and database connections and this could actually the context could be database but it could also be a customer API so it's up to you on how you wanna go with this and I'm gonna say matrix at registry dot create deeper connections counter and that's it and now if I run my application so of course I need to actually inject that here so the way I can do this is by saying I think I can do comma not default matrix but I can do X dot get required service I matrix here because I'm able to access the service provider here and provide this method so cool let me quickly run this now and with lists running I can go to postman and I can say okay here you go and this is now running stuff and if I create one two three customers I can go to metrics and see my own metrics again and actually this is where what I had before the counter is automatically converting the friendly names I gave to a Prometheus friendly name and here you can see that my customers API which is my context automatically populated the created customers to be three because I created three and if I do for now and run this again you can see four you can also still database connections increasing because obviously creating a customer is calling the database and looking at this actually I shouldn't have the database here so let's go ahead and change that because the context is database already so I shouldn't have this year so we just delete it and then run it again you might ask okay but why this reset is it not maintained no what is maintained this promethease is data the data actually in here is reset every time the application is reset that's the whole concept that you're not storing anything you're just exposing for Prometheus to scrape so if we just go and create a few customers again one two three and we get them a few times and we get the matrix again you will see that we get database called so connections and then created customers three and six and now how do we actually expose them in gravano well let's go ahead and go to the dashboard and I'm gonna say add panel I'm gonna say choose virtualization and I want to display a stat we're gonna go back to the queries and I'm gonna say what stuff do I want to see I want to see the database connections for example and I want to see the customers created and this is 3.2 which doesn't make sense and the reason why this doesn't make sense because it's trying to do an average I think based on time but what you actually want to do gets the mean what you actually want to do in this scenario is choose the last and this will give you the last value and really this is all you need to do you can change all them you can change a few of the here like thresholds and stuff like that we don't need a special table for this if you have a threshold it will go a different color if it goes over a specific number but we don't actually need to touch that and then all I need to say is created customers here and in fact if I go to the query I can also say six hours so I'm only gonna see the last six hours created customers I'm going to save that and go back here and now I have this nice metric which I can resize to whatever I want I'm just gonna make it big for now so we can see it maybe make this a little bigger I don't know yeah that is fine and let me just show you live what happens if this actually increases so I'm going to make it smaller I'm gonna go to create customers I'm gonna create one two three so if you just wait because these auto updates every five seconds now this from three should go six once Prometheus scrapes my application and Griffon app picks it up and exposes it so as you can see six created customers and then you can do the same thing with the database connections if you want you can go again and choose a query and I'm gonna say again increase and that increase function I'm gonna add the database it should autocomplete it's a bit buggy if you're typing cause it's refreshing but I can say database credit connections this is what I want and I'm gonna do a one-minute aggregate on that as well so show me all the database is created the database call created within a minute and you get this nice graph here and then I'm gonna say for the past six hours again and in the general can say that DB connections and that's it actually and I'm just gonna say safe dashboard and go back to the dashboard make this a bit bigger potentially move that here make this bigger as well and this is how you start customizing things now there's plenty documentation for Griffin and Prometheus but again if you want me to making more in-depth video please leave a comment down below and I'd be more than happy to do that this is a very interesting topic at least I think so so if you want to see more just ask for it that's all I had for you for today thank you very much for watching special thanks to my github sponsors for making these videos possible if you wanna support me as well you're gonna find the link in the description down below leave a like if you liked this video subscribe for more content like this and ring the bell as well to get notified when I upload a new episode and I'll see you in the next video keep coding

Channel: Nick Chapsas

Views: 74,215

Keywords: Elfocrash, elfo, coding, asp.net, .netcore, dot net, core, C#, how to code, tutorial, asp.net core, javascript, 2.2, csharp, rest, rest api, development, lesson, software engineering, dev, microsoft, microsoft mvp, .net core, nick chapsas, chapsas, asp.net core 3, dashboards, grafana, prometheus, appmetrics, .net core metrics, .net core dashboards, dotnet, .net

Id: sM7D8biBf4k

Length: 27min 14sec (1634 seconds)

Published: Fri Feb 14 2020

Reddit Comments

[u/SideburnsOfDoom]

Great demo! It gets to the point and answers many of my questions.

I would like to know more about who owns these packages https://www.nuget.org/packages/App.Metrics.AspNetCore/

quality levels and how actively they will be maintained.