Distributed .NET Core (DShop) - Episode 10 [Metrics with AppMetrics, InfluxDB, Prometheus, Grafana]

hey welcome to the 10th episode of distributed inner core course it's Peter and Derek and today we'll continue our previous topic which was you know all these things about login distributed tracing and so on and we'll talk about monitoring tonight everything related to monitoring and alerting how we can track your data and monitor your applications resources and some even some data that can be corrected with your business logic pretty much everything that you can imagine pretty much everything that that will help you you know to manage your micro services after you actually deploy them somewhere of course we'll get to the deployment part later in this course but for now we're just focusing on the stuff that will actually help you get some useful information from your app micro services so I think the good starting point is actually see what we can do about the monitoring and we'll start with this AB metrics library which is a free library built for dotnet car and it provides you a lot of integrations which popular open-source tools that you can later on use for monitoring purposes I think that we should I think that we should actually start with maybe not the definition but distinguish what actually we can call the metrics so because you know people you know people usually talk about logging on your you know inside your application but metrics is something slightly different than the interlocks right oh yeah you're correct yeah because locks can be something like there was an error there was an exception or user critical comments or something like this and metrics is more like actually maybe counting things like yeah how many users is online or how many user clicked on this you know post endpoint or that's a so it's something that you can actually measure because you cannot measure your error messages because that's just information for you that potentially can help you fixing some back but yeah metrics is more about numbers and that helps you analyzers you're in a whole system I mean basically you you could you know do some do some metrics with locks you know you could define maybe lock types or add some special attributes to your login and count by you know looking for these references of these lock attribute lock type attribute within your text file or database but it can be done this way but doesn't mean yeah but should be done this way yeah but in this case you would part you actually count the locks but the law itself would not be yeah it would be part of the metrics but not matrix itself yeah so so that was worth worth mentioning and yeah so we have this app metrics and as just as you said this is a open source project as we see for don't miss current opens framework as well yeah 4.52 okay so I don't know whether we are going to go through this up metrics right now and see different types of metrics or we just go straight to the code maybe let's go to the code and see how we can start using it what it does for us yeah okay okay so we have the solution and I know I think that we should start with our discounts I guess yeah sure and we have already added this to things I don't know maybe whether we want to add them now or later maybe we can just leave it yeah we are logging in the previous episode because we used Seki oh we've set a lock so this is the login part and let's go into this AB metrics extension method okay right so app use up metrics this is another extension method that can be found in the carbon package under the metrics directory and as you see there's a couple of things that actually happened here so basically on the you web host builder so the builder provided by the asp.net score we call this we call this method called configure metrics with defaults and what we does first we actually get the settings from the from the app settings dot JSON so I think that we should show this settings yeah we need to copy this settings oh yeah okay this is not here in the discount yet so a gradual of a copy and paste yeah this is this is awesome because we can copy this easily yeah so we have this matrix section which is going to be placed right here under the center doc maybe for now this is oh sorry now this does not matter let's make it let's sort it alphabetically let's put it under the maybe okay as you raise the yeah right all right so we have metrics right no oh yeah you're right this should be before okay okay I will put this later okay so we have we have few few options here so maybe you should go through them and briefly yeah so as usual the switch whether the metrics are enabled or not so by default the metrics will give us a few helpful end points that later on we can consume from the special databases that are that know how to deal with the metrics we have the integration with influence the beaded we'll talk about integration is pro Mateos Proteus there is also a possibility to add the whole elastics elasticsearch stack integration but we're not gonna use it here but you can also do it with AB matrix library and you know some basic settings such as the influence database URL if you like to you know send your metrics to this database the name of the database and the interval which is all about how often your metrics should be sent because by default they will dimensions will be also send usually in some sort of batch requests ok so so I see from my point of view this looks kinda similar to what we presented in the last last episode so when we presented the Jagger because Jagger was also divided I mean the I mean the the mechanism so first is that of course then we have some sort of metrics provided by the up metrics library so same thing kind of like we did with the open tracing so we have some built in metrics right now here but also we need to store this data somewhere right so this is like a reporter from the jogger when we need to store this some data sources so yeah it could be either the influx or the Prometheus or something right yeah I don't need to store data meaning that you need to pull data summer or either need to expose the data so for example if you're in primitives there will be a service that we were able to scrub your data from some endpoint yeah but but but basically what I mean is that we need to extract these metrics from the app somewhere else right because that would make no sense if you just keep it in the in the app yeah the last thing is that we need to sample the data because we don't want to store or maybe push or pull you know too much so we have this also this interval so this this looks are kind of similar so you can actually start the app now by adding these up metrics metrics section and enabling you can start the app and open the the endpoint so without any okay I just need to maybe run into the console be faster yeah okay well let's start the app using dotnet CLI will see should be hosted yeah okay and they're 5010 yeah and we'll have now for four new endpoints so you can open these five thousand ten and let's start with the slash pink okay let's see first whether this works okay so it should be up and running yeah so for example the add the pink who we can use for pinning which which is used by default for example by by the console for these endpoints we also have the healthy okay so the healthy I think under the hood it uses the ice-pan SP net score how free or health helpful maybe of help I have yeah yeah I think that by default uses the asp net score health checks I'm not sure but I think it does and we also have the two interesting ones which is matrix and matrix - text ok so matrix so this is the first yeah and it exposes some of the default matrix and also matrix - text which exposes the mattress just in different format okay - sorry okay that's looks okay so we have the counts up measurement yeah and some different metrics that are actually calculated by the matrix and we'll also a turn out our own matrix here right so we have we have this bunch of data here so how we're actually going to make use of them this does not look like yeah now now we need we need to somehow browse for this data of course you could you know write your own own queries and parse this data but there are tools that can help you do that so maybe let's start with this influence database we need to you know store the data somewhere so we'll start with the influx which is and which is a time series data base which means that it stores you know the points in time so it stores the time and for this time it also keeps track of your metrics which are by default I think it's just a key value storage for the time so basically this time series data base and the it works this way that your apps your app will be pushing the metrics to the influx and it's for in it's free you can use it for free and also if you are if you were looking for some like a complete solution you can look for the infinite attack yeah and similarly to like elasticsearch stack there's the whole stack that allows you know to do this monitoring login tracing and so on so this is similar to elasticsearch stack beats and these sort of things so simply yeah yeah they calm yeah you can choose this tag or this tag or you can mix these tags so it's not like you are tied to one particular stack but you can maybe choose this part of the stack and use the other part of the stack of different you know different technology so it's up to you but this is but to make it clear the influx stack itself is free or yeah yeah it's open source and it's free I think the I think you pay here mostly for you know enterprise resource being something like in elasticsearch okay so alright but we'll just use the influx DB and we can start the beef through the docker yeah okay so first I think that we should yeah we should start so if you go to to the d-sharp project on our github there is a directory called they composed so this is the same directory that you used for starting the our previous infrastructure so whenever that was the manga rabbit readies or as we introduced in the last episode we have the council fabula and evolved so in this same directory you can see another compose called grow fauna influenza B Prometheus so that that's the one that you are going to that we are going to use in this episode so of course for this purpose you simply type topper compose then you need to use the is the f4 so you want to choose the particular file and then want to app and if you want you can add the detach for to run the containers in the detach mode and in my case I already have this up and running so I can prove this by doing docker PS so you will see that I have free I have prometheus influenced DB and the grow fauna and the other of course along with the and you can open this docker host 8080 86 and it should return not found which means that the influx API is up a running yeah wait yeah yeah yeah okay yeah so this is this fine and so let's go back into our app settings and I just set the switch to be true wait so we get back to upsetting so okay let's see we have this influx enabled from forcement to enable this yeah and you can also go to the extension part the extension method and here you will see that you have this switch yeah yeah influencing me yeah so I I would say that this is a as I said this is pretty similar to what we've done in a in a jogger so basically we set the reporter that we want to in a jogger you could report either to know so to elasticsearch or yes we've said look we'll have the scene and knocked our this or whatever so yeah so okay so basically what we do here is we specify okay I see yeah connection strength yeah and it will create a database for us if it doesn't exist because for the influence you also have the API and you can also use this Eli if you want to perform some queries or comments on your database so let's just restart the app now okay and it should start reporting the influx hopefully it will okay so let's do this one again should take while okay it's after running yeah it will start reporting to angels now now the question is how we can actually get the data from the influx I mean we could use the query we could you know open this Eli and query the influence data like we could query the sequel or no sequel data but we don't want to query the data to see the metrics because we can see that see the metrics from our endpoints as well so we - you know at some point of dashboard where they can actually visualize our metrics so we'll graph Anopheles yeah let's face it that the dashboard makes you know look even more Pro yeah say the same thing that with the dark themes if you're using this you're just before and okay so we have this graph on I think so basically so what graph ara does it basically presents you this ugly data as you saw here and make some magic to look it like more like here so yes this is pretty much pretty much about the visualizing data and presenting the metrics in some more useful way for for monitoring your applications because let's face it you cannot just consume the metrics in you know as a Jason's yeah and it's free and it's you know free and it's cloud agnostic so you can use it anywhere yeah so graph Anna as we presented to you was part of the docker compose so once you actually run this compose you should go to the local host for the audience for 3,000 yeah yeah and one more thing because maybe this is as you see I'm already logged in - yeah let's just outside let's remove everything just kill the docker compose clear the volumes and restart everything for now why I mean that I the only thing I I won't say this that the login is admin and there is no password and that's that's is admin admin I think no this just admit okay no password yeah so you don't actually need to create the create the account on the Integra phone itself if you want to use it locally but later so we need to add the data source yeah okay so we can go here under the configuration simply go to the add data source and we can choose from friendly different data sources so in our case we would like to choose the influx CP because that's exactly what we hope we will have and let's call this D shop maybe the name doesn't matter it's just the name that you can see the device you can put here anything yeah this is the the the part that actually matters because here you have to specify the URL and here need to put if you are running this through campos you need to put here in flux DB i think you can select the docker compose file what's the was the alias for this okay we'll see I believe so the container name is in focus dB okay so this will be our DNS and this would match it you can also you know sell about education here credentials you know certificates and so on and yeah for the details D shop database because that's what we specified in our in our app settings and it this shop database will be created by the app so here right yes yes and we don't have a username and password so that was sweep did - no just the shop already shop ok and just saving tests yeah and as you see this is working because this already tested the connection so yeah so just keep in mind that this is in the container so it uses the DNS is provided by the by the docker ok so let's say yeah I mean it also go with hold of her internal here and it finishes work yeah this is the same thing okay so now we have we specified the dot yeah the whole source of the day of the matrix but the question now is how the heck we actually present this data because if we go to the home right here we can create new dashboard yeah so we can you know adhere some pretty nice-looking we could create like a mime graph and some create this from different parts depending on our needs but I hope that there is a better way of doing this at least for our purposes yeah for the demo purposes let's actually use already instant template because you can browse through the free templates on a graph on our website so you can look for the template which is Windows created by the by the guy by the team who created our magicks library and there is a special up metrics influence DB template so you can just copy this ID or download JSON because the template is just a JSON JSON object and file so but basically this is and underneath there is simply mapping between particular name of the matrix and some tags that are needed yeah you will see this once we think that once we go to Prometheus more so basically what you need to do is simply grab this ID and go back to the graph an ax and you'll see import and I will discount this because I'm not interested in this simply paste the ID and you see that we have imported this yep yeah and now we have something let's try to open the overview section above yeah we have yeah but yeah it's not what yeah I had the same issue during the last workshops it used to work but I think it's something later to the father these templates stopped working somehow so I would say no yeah yeah but you can open the you can open this this screen shot and see how it should look like I mean it worked in the past but I think it migrated to the newer versions by far now because this yeah because it's it worked for me a few months ago but during the last World Cup it also fails that that's kind of weird thing yeah maybe maybe something changed in the final part hard to tell yeah but this right now yeah yeah unfortunately I mean apply only to the influx EP or yeah also with permit oh so there's something wrong with the latest versions of these templates maybe they weren't updated or there were some break-ins Befana but if it if you actually try to import this on your own maybe it will work on your side maybe yeah but maybe also you could try to it we don't want won't do anything else it won't help but maybe there's something with the you know the mapping or something could be could be but yeah but it's kind of it's quite a lot of code here so yeah yeah so maybe there's something you know in there with the mapping that could be that breaks it I mean it's to work at least a few months ago but last few you stopped working and also other people tested it so there is something wrong but basically you can you can actually import the in decision templates you can export your template and you can of course adhere your own elements so maybe let's just leave the in flux for now and because the like the the thing about templates and the dodgeballs will be the same with other and with the other tools so maybe let's keep the in flux and you know the app machine will pull the data to the influx in the background mode and maybe let's go to the pro materials because in the pro materials will also have the dashboard for the for this app itself and also later on integrated with the rivana yeah but you can't even trust us it's template used to work that's a it's the best expression you can travel yeah it actually worked it's used to but now it's it's gone yeah right so now we have another weird thing called prometheus and what what what the heck is this yeah so so previews is a quite blur tool because you might have heard about graphite and a graphite was one of the first Oh naturally one of the first but also it's also still quite a popular tool rated to you know monitoring and you know consuming the metrics exposed by your app so profuse is like the newer like maybe not a newer but sort of like graphite but I think it's even more popular now and it was released I think in 2012 so it's even like more recent version and I think yeah I think this is definitely more popular than a graph right now and what it does so it can help you with you know consuming the metrics it also has you know some nice alerting options and it works in a opposite way to them for example in flux so pro my fuse will scrub the data you will need to expose the endpoints so for example these metrics or metrics - tags from your app and it will just send the request to this endpoint for example every five seconds will scrub the data and then you'll be able to for example use either built-in profuse dashboard or use the fan or some others who do you know visualize these metrics so to make it clear we could change the we could get rid of the influx they'd be and here's the Prometheus as this data source which would say there once a while would pull the metrics from our app right yeah and so influx is a push model our app pushes the metrics and prom fuels is a pull model which means it will pull the data however you can also if you type in the Prometheus push gateway in the Google you are also able to you know switch to that push model because sometimes you might not want to expose the endpoints from your app and it works this way that you just start a push gateway push great way container purgatory service summer and now we are your application will be pushing the matches to the pull Gateway and the premier fuse will scrub the metrics from the Gateway okay okay so kind of site yes obviously twinned it between the app and okay exactly okay yeah and one more thing that it's kind of worth mentioning is that the Prometheus itself exposes this prompt QL which is a query language spaces built for their Prometheus which allows you to actually add query from easiest for some some for some metrics based on based on these labels that actually Prometheus exposes so this is you mentioned the graphite and graphite was as far as I know this wouldn't was known from this white cards or the patterns that you actually could provide for the querying for the particular matrix while the Prometheus itself actually uses the labels for query and this makes it more I would say that more flexible yeah for query that's yeah what is also great about premier fuels is that it has a lot of integrations for example it has built-in integrations with docker engine I think also if kubernetes and some other tools so basically you could start docker kubernetes some databases just enable the Prometheus plug-in and you have built-in metrics for a lot of services infrastructure apart application within your whole you know horse solution all your micro services yeah so this is very very nice okay so right so shall we go to app settings yeah and let's let's let's disable the influx I mean you could run both but it doesn't make much sense I think let's disable influx and go here and I will disable the influx integration and will enable the Prometheus yeah and let's look at the code maybe the extension yeah will seem to go here yeah so we have specified the Prometheus yeah so yeah we have the endpoint for our videos because it uses different is that I've set a different type of metrics in a different format so you have to enable these premiere fuels as you can see we have this endpoint out for matter for this premier fuel stacks peripherals also supports the protobuf but we'll just go with text and also if you scroll scroll up above we have these these stacks which are pretty generic but here it's done a little bit weird way but it had to be done like this so the app metrics by default you can enable these free and built-in tags up and a server tag which either uses your app name environment and server server name or you can also override it in this weird way but this is how I had to how had to do this in order to make it work so this is like an like adding you know this enrichment to your login with secure or an look so it's pretty much the same okay so let's restart the app and maybe also open so we overwrite the Prometheus yeah also yeah show the conflict for the Provisionals itself okay so we have it here so this is the same folder so this is the compose directory in a dish of project so we have this Prometheus I have a file yeah and here we have our targets or our jobs so you can name it anyway this is just an alias that you can later on browse through previews and the targets is the the host names so here we we can either specify for example API and call on our port where our API or the service are running but this will only work if you have for example here our dr. Network right because we are using the aliases so in order so yeah if you're not that's where we will use our special special you know endpoints so okay at first I think I need to add the the dictionary sorry about the discount so maybe let's remove all of these things just maybe just keep the discount service just rename one service to the discounts okay so that's okay so let's call this discount service and we need to specify so SPO said because the Prometheus itself will be running inside the docker container and will run the application from the dotnet CLI using this defensive role - yeah so this basically will be exposed so we need to add this magic URL so this is a docker for Mac right yeah doc confirm about the host but I think also the holes dr. internal should work but okay so let's let's go with host or internal I think it should should also a coder know okay and the port is five thousand and ten yeah and right now we have to actually restart the Prometheus so it will load these settings let me just just kill this man vana or just maybe yeah let's just do docker compose down but you have to you have this tab open what you have the top open layout here okay where is this okay now I screw something oh sorry downloads after ya Jesus kills me that sometimes is ya down die okay and let's restart okay the detach mode yes you should be able to open the promo fields now it starts okay 1990 yeah so I'll just close this localhost and we should see create bootstrap you why I believe yeah yeah and let's go to the status and our targets section and here you can see whether it whether it can you know grab the targets grab the database is down but I think that we should start the app actually oh yes makes sense yeah this makes sense okay so let's do doesn't run once again this will yeah yeah and I think we have by default 10 seconds every 10 seconds to scrub the data and I think the lowest value is probably 5 seconds the same in graph Anna so you I think I'm not sure if you can go below 5 say yeah this is our yeah it goes up so yeah if you open the others you will see nothing here but if you also define the others within the configuration file so you can do you know a lot of alerting based on some specific roles but let's go to the graph yeah and let's look for orbea and we can here query our metrics so maybe let's open the metrics endpoint and then see the difference okay so let's just refresh this one and also let's open the metrics - types but missions but that steps I think will be the same yeah so only the metrics which is by the phone which is the default endpoint for the prompt use change yeah yeah so this is actually how you specify the metrics for the for the Prometheus and that's the difference between the premises and graphite that is basically what you can see is that the name of the metric and then we have the set of the of the label yeah rock stars yeah tax and the value yeah so later you could say let's just maybe do something like this yeah but you have to copy copy depends on what you want to render but yeah wait I lost my Prometheus oh it's here okay I what I meant is just I'm not sure whether you can actually do something like this that query to data yeah so as you see this is the one of the usage of the drama q well yeah it's like you yeah and here we have some we have some metrics or actually being rendered so so so basically what what is what it does is simply give me the metrics that actually contain the app tag called with these values you know all metrics that belongs actually to discount services you see there's a plenty of them and if you just specify full name it also has the autocomplete built in just just our ear to start typing all right we're here in just remove this one and start typing okay okay and you can also you know browse metrics like this oh yeah okay yeah or either you can use this yeah yeah so okay let's maybe go to graph on ax because this UI isn't that pretty I mean it's okay but let's actually try to render this metric using the graph Anna and yeah we have to add the new data source for the Prometheus so we go yeah once again two data sources of discount this and let's start with the new entry as you see we have this Prometheus here so I think that we can add this as the browser like this and this should work okay we can add yeah let's just you just use the menus yeah the doctor Network you can change the straight interval Murray to 5 seconds ok yes 5 5 s ok this and let's save it yep ok it's working all right get back yeah we could actually you know import the dashboard that will not work again because something changed so maybe let's try to create a graph on our own okay well just will be yeah just create a graph and yeah edit so click on this edit and here we have the query part so in this a B you can add more queries you can put the metrics name stats and everything so let's just type our metric name it also has the autocomplete so you can try to oh yeah so just yeah just choose whatever and click somewhere on the trade with try without the tax ok and just put somewhere click somewhere on this screen ok and we already have the metrics being rendered and if you click on these last six hours at both last six hours yeah you can change a refreshing every and let's put it into five seconds maybe yeah it will it will ask for the data every 5 seconds and it will see the graph being render in real time and there is a lot of things you can configure you can for example add a new query and you can have like we like you've seen on the previous tab like multiple you know fins being rendered on a single graph and you can you can you know add the counter and other types of other types of elements so maybe all right maybe whether that was something like HTTP request something HTTP request no I think it depends on the library because there is also dedicated that library for does it call for primitives only and it has more built in metrics but this doesn't have too many metrics built in yeah maybe let's let's say this one and maybe that also let's add a new maybe counter so let's save it okay wait just yeah just save it okay yeah and there just are the new elements maybe whatever let's just add this single start okay and let's here's some some file so you can so for example for the singles that it has to be unique right so if you would try to put here some metrics that which name is used in other labels as well it will say it will you know render an error so you have to copy the whole whole metric label metric name which is unique which usually means that you also have to specify the tags so maybe we'll just copy the metrics from the from from the whole metric okay so let's say it's might be let's get back to okay yeah yep and you can go to the options and you know change the background whatever so this is how we can actually you know start dealing with start playing with the metrics and using premiere feels graph on a-- and you have a lot of options here you can there is a lot of building functions for some aggregation some statistical statistical functions you can format it using different unix different colors and a lot of things you can do so right now you can see as you can see we are able to consume the built in metrics but maybe let's try to create our own metric right because that's where the business value might come from from actually you know showing to the end users or to the business guys your own metrics that they are interested in yeah so i think that we should go to yeah that'll be cool to actually create something yeah so or our way to go to our app and probably the fun part yeah so let's say what would you want to measure we can actually measure anything and we have these few matrix types to start yeah I think that we should we should go to to the app matrix to show the difference because this is kind of crucial to understand if you go to the app metrics diet dot IO that you'll see that we under the metrics types you have ten different kinds of the metrics so for instance you can you can see we have the Galax we can the counter and the meters histograms timers and objects and each one is for different purposes so basically so this one could be could be a useful for describing let's say the CPU yeah memory disk usage right network temperature or anything else the counter itself simply yeah you can you can either increment or decrement this so it could be useful for I know truck lease request maybe like a home user you know ask for the data using this endpoint or query data or whatever but yeah yeah let's let's try taking some vegetable yeah maybe we should go with the counter but yeah this is fairly simple because you can yeah would be kind of hard to describe this alright so I don't know whether we should create some maybe some directory called metrics or maybe let's put it into services sub directory service yeah so okay so the way it works is the app metrics gives you the eye metrics root interface that you can use to manipulate the metrics however you need to define at first you need to define your own metrics so in order to do it maybe let's just do with the simplest way possible you just create a single interface and implement it in a in a class and our class will hold all of the metrics that we want to measure so I just call it I metrics registry maybe can we just create metrics that kills me that I hope this in the services yeah that's that just create yeah just quit because this simply close me yet okay so let's call this metrics and all right so what was the name of the interface all right let's call it AI metrics registry maybe all right so I metrics registry registry and then right and so you want yeah so let's before you create a class all right yeah let's just put here a method so what do you want to measure here we can actually think of anything let's say maybe we want to measure our handler so let's say in our maybe let's our in our handlers let's open the handlers yeah and the discounts and maybe create discount handler but this one or it might actually fail but we can actually measure it anyways I think or maybe how many times we actually I know yeah but hey there is an exception which will fail so maybe let's do something simpler because we will get an exception here how many times we query for the polio make sense so let's measure this one all right so let's say so have the find discounts or get discount handler yeah fine discounts okay so let's say we'll have a simple operation he'll let's make it avoid all right and maybe want to increment or incremental bands find discounts query you have a great increment fireman find this kind where it counts great worry okay yeah so this is alright so we won't decrement anything you can do all dented on the interface name I know this option is actually a options right here and now I need to do this once again right because I need to specify that one to wear that and extract the app move it's kind of weird to me okay so let's make this public and implement missing member okay so we have this now we need to as I understood we need to create a private which yeah we do our metrics so the way we do this we create the metric options and let's we gonna use counter so there will be counter options so we won't no no just counter options as a metric counter options counter options oh sorry let's yeah let's call it our find discounts this yeah let's just call it let's make it meaningful because we can have a lot metrics here five days behind discounts query great ever or yeah let's make ladies right and query or maybe query query usage counts queries okay yeah yeah let's make this and let's initialize this already okay no nose no nobody's oh sorry yeah yeah Jesus Christ new caltrops okay and after you have to put a label here so using the using the object slider you can set a name property and also different options but the name so this would be the label tag so we call this like whatever find discounts queries for example yeah fine disco and just keep in mind that Prometheus requires you to have the lower name the app metrics will change it to low so to the lower case the emissions will take care of this but if you if you wouldn't have the lower case labels diploma fuels wooden scrub data it'll fail so that's important and actually the previous version of our metrics had this back so I couldn't use it because it wasn't lowercase really and I and with the numeration of primitives it just didn't work but they fix it so you think you know yeah you can use this one but ok baby just for the safety out of this yeah okay basically to make it clear this will be translated as a as a tag for the label name my attacks are in the brackets this is the label okay so this is like okay yeah and this is the dog who's coming from the up matrix yeah so if you see this is AB matrix counter yeah Dobbs yeah so let's use the AB matrix let's use these metrics root in within our increment query yeah so we have this and now we can do something about this right yeah we can just say measure dot measure yeah and then we look for our metric type which is counter and then we have for example instructions for different types of metrics we have different methods and we can simply pass our yeah and that's it and is the only to care about so that's also the weather we are using influx of other reasons you know the probe use the app matches will take care of actually sending our metrics to the server or just exposing the endpoints and but we also have but you know you have to actually register it in a single tone sorry in the IOC container in a start doesn't it yeah just very registry yeah let's make it a transient maybe so okay just you can use default container just call services thoughts right at transients and so this is the opposite I believe as in out of fact so firstly specify the interface yes I find no it's I Metro service tree oh sorry metrics registry no no this is this one yeah yeah yeah and now let's inject our metrics registry to the handler yeah so let's put it here okay let's call this registry like this I got it okay and just introduce this is from discounts and we also want to initialize from the constructor and I know whether yeah let's put it here so maybe just registry okay and increment let's just start the app now yeah so let's go here it's kind a lot of stuff coming down and before you do any any query query let's just open the metrics so it won't be visible immediately you have to you know at least call the metrics ability once so if you refresh there is no metric yet but let's just hold it hold it slash this discount so it was right to the four I believe so this counts think pink yeah alright and let's refresh the metrics let's go here yeah and it's here the first one yeah fine discounts yeah yeah cool so we can actually add this to our graph Anna that for now because it's already in the Prometheus so yeah let's let's just add it to our graph Anna so maybe let's make it a single start as well yeah and we go here and we call this fine yeah it's here discounts yeah and let's maybe change the color let's make it more visit so in the options you can you know add the colors and whatever I don't know let's make it red okay but okay and I think you have to click on a value if you want to make a value to what to valid value checkbooks okay yeah and so let's go back into our our dashboard you can just back on the other you have to close the whole dashboard just go back yeah let's start refreshing let's refresh our metrics a few times just go yeah and as you can see if you go back to the dashboard it will be incremented in a few seconds after every five seconds once the ones he promised you scrubs the data and once the graph on a scrubs the data from our app yeah yeah it's eight so yeah and that's pretty much it you can do a lot of different metrics so for example what you can because what you've just seen is a very simple you know metric structure but you can also go with more more sophisticated needs yeah you can for example make use of the async sorry of the action filter attribute and you know just create an attribute that would that would use within the own action executing our own action execute these metrics registry and you could add this attribute to your control to your actions within the is political controllers or you could maybe you know use the decorator pattern and by using your you know ioc container you could also mark your handlers or some services with these attributes and you know based on this attribute even while resolving the particular service particular handler for your container you could look for these attributes and maybe you know update the metrics as well so you can do a lot of cool things with with these metrics I would say that especially with their speed up that scoring this how this gives you a lot of places potentially that you could actually plug this yeah middle words under and the whole whole bunch of their the filters and could you also open this this Bell and the the left panel will have this alert in tanks below I mean the yeah yeah alert Jules so you can also set the alert drill skills and if you open the notification channels you can for example enable slug alerting and we have different channels for alerting so from within a type you can you know send your others to different different you know apps and and or the mail to your boss yeah and the one last thing is the plugins because the fauna has a lot of planes and some of them are of course commercial so if we go to the plugins can you browse the it's called world pink world pink okay world yeah like this and yeah let's let's go to the brah fauna and this is a pretty cool plugin and the first one which you can finger up if you open these screenshots you can use for example for tracking your internal services alright like whether they are alive or not you know the request times and so on so we can also use some built in or paralyzed on plugins so this is pretty cool more developed yeah but but basically yeah this this gives you more I would say that the more real-time control so you can actually yeah assuming that you actually you're up and running on maybe production or maybe you're still on the QA or something but you can you know see what's actually happening with your with your apps and that's I would say this is pretty cool thing because of course locks can be stored somewhere but then you need to analyze them tracing is cool but of course you also need to go through this whole traces and analyze the particulars path but this gives you an a real-time overview of what's actually happening to to your to your app yeah this could be either some more business of yeah say the more business related stuff maybe more valuable for the business or maybe some more you know hardware infrastructural data that would be yeah valuable yeah for your for guys MEMS yeah what is well it's also great about to be father and one of its features is that if you go back to our dashboard we cannot see it here because we don't love it we didn't define it here let's open the dashboard but in the graph are you could also have the under these new dashboard copy where's our dashboard list you could also if you open these settings for this dashboard and you go to the annotations yeah and sorry the variables I think it was oh yeah yeah you can't for example specify the variables and you could you could for example create a drop downs where for example on the first job you'd have the maybe cluster or servers or knows list and then you could select a note and within your note you could have another drop down that would for example filter the instances of your applications run infinite note so you could also do this sort of filtering yeah and you can set the variables for the de fauna and you can do think if we go to the this up metric yeah you should see these variables yeah yeah so we can define any different and you can then create another variables that actually refer to yeah and if you go back to dashboard there are actually these drop downs visible here yeah yeah so I directly drop down so yeah you can you can configure it but you know the - good doesn't work because there was some backward you can define it through your graph on ax you can also expose this publicly you can you know are different folders within dashboards you can add teams with different permissions you can you know expose some of the dashboards publicly might be for read-only view so you can pretty anything with it there's a lot of stuff that you can actually configure and as I said this this gives you great tool for analyzing this in real time what actually happens yeah and I think that will be it right because this is how we can start working of monitoring but there are a lot of different tools of different stacks so feel free to use whatever whatever suits you so so so I think that we should sum is sum this up these two episodes of the of the let's say a two links so if you have any sensitive data or something just simply don't first don't hard-code this because that's stupid and you should not put this into app settings or something so as presented the last episode you can use some for party tools like the vault for storing your sensitive data then if you want you know what you have any kind of the information specified and you can want to browse this pretty easily then you can add some login using secure or something but just keep in mind that the logging itself is it's related to one specified service in this case because we have distributed application so if you want to see the overall picture more big picture of what's actually happening to your application then you can go and try a distributed tracing and as you saw today you can actually add one more value for your application which is a real-time and more I would say you know more data related numbers so lights like CPU usage or a number of calls that are actually made to your application so something that can be actually measured and for this purpose you can use some Prometheus or the influx DB and you can expose this or present to you using to grow fauna mm-hm so this is the whole stack I think that this closes the this this part of this series so that's it for this episode and see you next time yeah see ya

Channel: DevMentors

Views: 7,511

Keywords: .net, .net core, devmentors, dshop, aspnetcore, microservices, metrics, appmetrics, influxdb, prometheus, grafana, distributeddotnetcore

Id: 2OiE-h9sNM4

Length: 59min 47sec (3587 seconds)

Published: Sun Feb 10 2019

Reddit Comments

[u/dewijones92]

you are doing god's work