Nest.js Metrics - Prometheus & Grafana Tutorial

everybody today we're going to take a look at adding metrics inside of an sjs application we're going to use Prometheus to scrape these metrics from our node.js app and then we're going to export them to a grafana dashboard so that we can see how our application is performing at all times and see how this data is persisted so that we can continually monitor our application which is super important if we're ever going to deploy it into a production setting so let's go ahead and jump right in and see how we can develop nest.js metrics with Prometheus and grafana I'll see you there before we jump into the video I just wanted to let you guys know I've released my new productivity app called thrive on iOS Android and online at Thrive app.1 this app allows you to track your daily priorities habits and complete a Daily Journal and will allow you to create notes as well so you can maintain productivity as you go through your day and only accomplish what matters most to you if you'd like to check out Thrive it's totally free I'll leave a link in the description okay so instead inside of a new directory where we want to create this new project and I'm just going to run the nest CLI here and run Nest new nest and this is going to go ahead and scaffold our new application I'll use pnpm as my package manager so after our package installation is complete we can see the into our new project and we're in pmpm Run start Dev to start our development server that will recompile on change okay so I've gone ahead and opened up our project in a code editor and this is just a basic Nest JS application we can see here while you're starting up and listening on Port 3000 and we have our single app controller here which will be a get route that will just return a hello world string so if we were to go ahead and open up Postman and execute a get request at localhost 3000 we can see we're getting back our response so now let's go ahead and actually add a metrics route to this application so that we can scrape the underlying metrics and report this to Prometheus and grafana so we're going to use this open source package by Will Soto called nest.js Prometheus which is a nest.js module for Prometheus that will export our metrics for us and give us all these great metrics around node.js so you can check out this repository here I'll leave a link in the description and we'll drop a star here because this is a great repository and you can read more about this nest.js Prometheus module see what it does essentially all we're going to have to do is install this package register inside of our app.module and you can see by default it's going to register a slash metrics endpoint that will return us default metrics now if you scroll down you can also see that we have the ability to configure default metrics here as well as injecting individual metrics so we can easily add our own custom metrics if we so chose two we can also change default labels and so on and so forth but the default out of the box will be good enough for us so let's go ahead and copy this installation command so when you're going to install an sjs Prometheus as well as prom client which is a node.js Prometheus client and we're going to go back into our terminal here and I'm going to run pmpm install and I will add these two packages here so after we've installed these packages I'll just go ahead and run start Dev again here and back in our editor we'll go ahead and open up the app.module and all we're going to have to do to set up the Prometheus metrics is import the Prometheus module and call register on it and by default we can just use the default options and this is all we have to do now if we go back to postman and we make a get request at localhost 3000 slash metrics which is the default endpoint for this metrics module if we send this request off we can see we are now getting Prometheus metrics back from our application so we get this valuable information including CPU user seconds so we get node.js event Loop utilization statistics which is an extremely important metric inside of node.js we also get active resources active handles so the number of current requests that our system is processing we can actually get the actual number of active requests here we get information about memory being used so in general we get a ton of gray statistics around our app including node.js garbage collection as well so this is all great but out of the box this is a bit hard to digest and it doesn't give us a great reporting ability so we're going to actually export these metrics to a grafana dashboard so that we can view them in a really nice way over time so before we deploy this code inside of a kubernetes cluster which is how we're going to actually set up Prometheus and grafana I'm going to add one more file here it's going to be a logging interceptor this is going to log out all the requests to our application and so I'm going to paste in this code here which I've gone over in a previous video on building out a nest.js starter project essentially all it's going to do is it's just an Interceptor that will log out any HTTP requests and log out the current route the correlation key and so forth so I'm going to Clue the link to this repository where we created this logging Interceptor in the description if you're interested but for now this is all we need and we also need to add uuid here to get the correlation key so I will quickly install uuid here and now back in the app dot module I'm also going to add a new provider here which will actually be a app Interceptor here which will have a use class of our logging Interceptor so now we have a logging Interceptor if we go back to postman and send off a couple requests we should see in our terminal we actually see the requests logged out for our slash metrics endpoint which is exactly what I want to see okay so we're ready to dockerize our application here so that we can actually deploy it inside of a local kubernetes cluster so I'm going to create a Docker file at the root of our project and I have a previous video where we go about actually building out a Docker file for an sjs application and I go into each step in detail so if you'd like to see that video I will leave a link in the description where we build some SGS microservices and I went over a Docker file in depth for now I'm just going to copy over that Docker file and paste it in here where we can briefly see the steps to build this application all we're going to do is copy over the package.json Run npm install and then do the same thing in production mode so we only get production dependencies and then we start our node server with this command and lastly we're also going to add a DOT Docker ignore file here which will tell Docker to ignore any files which we specify in our case we're going to ignore the entire node modules directory so that we don't copy it over during this Docker image build so after we've created a Docker file I've opened up a new web page here and gone to Docker hub and I actually created a brand new Docker Hub repository for this Custom Image that we're building so that we can push it up so you can feel free to go to Docker Hub sign up create a new repository call it what you want or you can use my existing Docker Hub repository here with the nestvs metrics so once you have a Docker repository that you can actually push to we can run Docker builds and we'll give it a tag here in my case we're going to use my repository so I'll call it mgway nest.js metrics and then we'll provide the path to the docker file which in this case is the root folder and lastly make sure you have Docker running on your system and we can actually build this image so after our image has finished building we're going to go ahead and push it up to Docker Hub by running Docker push and provide the name of the tag that we just built okay so now that we've actually pushed up our application to a Docker repository we're ready to create a kubernetes cluster where we have this application running as well as Prometheus and grafana to view these metrics so I'm going to create a new folder in the root of the project called k8s to stand for kubernetes and then we will go inside of our terminal and CD into this new directory and then we're going to use Helm as our dependency manager to be able to manage these dependencies inside of our kubernetes cluster I'm going to run Helm create an sjs metrics which is going to be our new Helm chart which will describe how our application will run and all the dependencies inside of it so we can CD into here and then we'll open up this nest.js metrics folder inside of vs code we can see by default it's created some templates here as well as a default value.yaml which are the overrides for the third-party charts and templates that we're bringing into this application right now we can see we have a default chart.yaml with no external dependencies inside of our templates we have some resources to find some kubernetes resources we're not going to use any of these so we can go ahead and delete all of these default templates as well as the tests folder inside of here in our values.yaml we can just go and remove everything inside of this file as well so now we're ready to Define our kubernetes cluster and all the resources inside of it I want to actually write a custom template for our nest.js application here so the easiest way to do that is on our command line we can actually use Cube CTL which is a command line tool to interact with our kubernetes cluster we can use this to actually Define yaml for our template so let's CD into the templates folder and we're going to run Cube CTL create deployment and I want to call this nest.js metrics because this is our SGS metrics application we're going to specify the image here is where we pushed up to so in my case it's mway slash and sjs metrics I'm going to say that it has a port open on Port 3000 because that is where our container is listening by default and then we're going to run dry run equal climb this ensures that we don't actually execute this command but we are just wanting the resulting yaml from it and then we'll specify here Dash o yaml that the output should be in yaml format we're going to pipe this to a new file called deployment Dot yaml so after this command is run we should be able to open up a deployment.yaml file here with all of the necessary information to actually create this when we deploy our Helm chart we can get rid of the status key here as well as the creation timestamp we can also get rid of the resources and strategy section here okay so now that we have all the resources we need to deploy our SGS metrics application we need to grab some third-party dependencies to actually run Prometheus and grafana and so I've opened up a new page here to artifact hub.io and then we will search for Prometheus and we can see there's a Prometheus package here and you can read all about the description the dependencies as well as how to configure it and change any number of options that comes with this Helm chart which is quite a bit the most important piece here is scraping pod metrics via annotations so we can see to actually get Prometheus two scrape pods we have to add the following annotations to our pods where we tell Prometheus to scrape it we give it the path that we want to scrape on which is slash metrics as well as the port for that container which is hosts the metrics endpoint so let's go ahead and copy these annotations here and back in our own Helm chart before we install it we can add these annotations by going down to the metadata and underneath the label for the underlying pod will add a new annotation section where we can paste in those Prometheus annotations which will tell Prometheus to actually scrape this POD at slash metrics in the port of course here will be Port 3000 so this is all we need to do to get Prometheus to scrape our app application let's actually go back and see the version that we want to install we can use the latest version here 15.18.0 and all we have to do here now is get the repo URL so in this case it's Prometheus community.github.io we will get this repository URL and now we're going to open up our chart.yaml where we can specify a new dependency section so our first dependency here will have a name of Prometheus and the version as I specified will be 15.18.0 and we will use the repository that we just copied the Prometheus Community GitHub repository and so now that we have Prometheus specified let's go back and look for grafana as well so I see grafana here and this is going to be a Helm chart that we can get the URL for here and now we will specify this in our chart.yaml I'll give it a name of grafana the version field here will specify and if we look at the version we will use 6.43.5 so I'll give it a version of 6.43.5 and the repository here that we just copied over so now every time we install or deploy this chart this Helm chart we create it will bring in these third-party dependencies that we need to expose our metrics lastly we're going to open up our values.yaml here and actually make some customizations to these third-party dependencies and that is easy to do with Helm we just need to use the name we provide here and specify the values that we want to override there are going to be a few additional services that this chart brings in that we want to disable because we don't want to use in this video the first one will be alert manager so we can say that this should not be enabled as well as the push Gateway we don't need this as part of this chart and lastly the node exporter I'm not going to enable on this MacBook but you can feel free to enable it as well it just will give you metrics for the underlying machine that you're using in our case locally I'm not going to enable it so now for grafana here we can specify if we were running in a public cloud like AWS or Google or Azure and we wanted to run grafana over an Ingress and have a load balancer URL for that then we could specify here that the Ingress for grafana the Ingress setting here would be enabled and set to true and then as long as you have a valid Ingress controller running in this environment your grafana deployment will be available over HTTP on that common load balancer URL however since we're running locally we can just access grafana through a node Port service and then access it on localhost so to make this possible we will edit this service property here and we'll provide the type of type node Port so back in our directory here I'm going to run Helm install nest.js metrics and I will provide the path to our Helm chart and you can see here we have an error saying we're missing in charge Prometheus and grafana so we can run Helm dependency update before to allow us to actually pull in these third-party dependencies now we should be able to run Helm install and give it a second we can see that our deployment was successful and so now if we run qctl get pods in our environment we can see nest.js metrics is starting up as well as the Pod fork or fauna a kubernetes state metric server and lastly the Prometheus server as well is starting up so if we get the Pod name for an SGS metrics we can run logs and follow the logs for an SGS metrics app and after about a couple of minutes once Prometheus starts up you can see our logging Interceptor is actually logging out requests from that Prometheus slash metrics route here so we can see that it is actually completing successfully and that Prometheus is starting to scrape this pod to get metrics from it and so now if we were to run Cube CTL get service we can see by default the helm chart creates an SGS metrics grafana service of the type node Port because we overrode this and then we can see the port here the node Port that maps to Port 80 on the Pod it's 31791 so if we open up our browser and navigate to Local Host here 31791 or whatever port is your node Port we can see the grafana login page Now default username here will be admin now to get our grafana password this is stored as a secret inside of our kubernetes cluster to keep it secure so to get this secret out you can run this command here to get the secret and Echo it out to our terminal you can see I now have this secret value you can copy this and paste it into your login page and now we have the grafana dashboard here and the ability to browse and create new dashboards so I've navigated to grafana.com grafana dashboards which is a list of grafana dashboards that we can freely install and use and import into this grafana dashboard section to look at all kinds of metrics based off of Prometheus so the one that we're going to look at here if we search for node.js we want this node.js s application dashboard here and we can see by default it works with prom client which is what our nest.js Prometheus module uses so we're going to copy this ID to the clipboard here so we'll go back to Griffon here and before we import this dashboard we're going to go to settings down here and under data sources we're going to add a data source we're going to add the Prometheus data source here and then we need to specify the URL that Prometheus is listening at so we can type in HTTP colon slash slash and then paste in this nest.js metrics Prometheus server name here which is a cluster IP service so we can contact it by name here and on Port 80 of course all of these default settings will be good enough for us we can click save and test here and then we can see that the data source is correctly configured now we can go back to our dashboard section we'll copy the this dashboard ID and we will import here so we can say import via grafana.com with the dashboard ID we will load this in and we can call it whatever we'd like node.js application dashboard and then we select our Prometheus data source here so if we go ahead and import this now you can see all of this metrics from our node.js application so for example we can see in the last five minutes the process CPU usage for our node.js app we can see the event Loop lag number of active handlers and requests and total we can see memory usage and all of this great information about our server and lastly what's really cool is that for any one of these metrics that we see here if we want to further dig into it we can open up this tab here and click the explore button I'll save this dashboard the changes here and now we actually are exploring on this explore tab here the Prometheus server and the actual queries themself that this dashboard uses so if we want to remove these for example and actually look at all of the different queries that we have we can see that we just start typing node.js for example we can get all of these raw Prometheus metrics that are slash metrics and point is returning us and what's great about this is that we can pull this range back right from days and days and have all of this information and data available to us so I hope this was a useful video on seeing how we can incorporate Nest GS metrics into a grafana dashboard I plan on using this same video to implement horizontal pod Auto scaling in a kubernetes cluster in my next video this is a super cool concept that will allow us to scale our application and add more pods as more demand hits the system and things like CPU and memory go up we can add more copies of our application to cope with the load so if you'd like to see this video make sure you stay subscribed and I will see you in the next one

Channel: Michael Guay

Views: 21,229

Keywords:

Id: 2ESOGJTXv1s

Length: 22min 41sec (1361 seconds)

Published: Mon Nov 14 2022