Measure your ExpressJS API Performance with Prometheus

monitoring your application's performance is a critical part to building and maintaining production-grade applications in this video i'm going to show you how to add prometheus metrics to an express api so who is this video for this video is for those who have heard of metrics are good but i'm not sure why for those that have heard of prometheus but not a 100 sure what it is those building and maintaining rest apis but you can also add prometheus metrics to any api including graphql apis if you would like to see a video on adding prometheus metrics to a graphql api please let me know in the comment section below and it's for those who want to impress their boss when i used to manage a team if someone come to me with some prometheus metrics to add to our application i'd be very impressed so what will you learn you learn how to add a prometheus server to your application and you'll learn how to gather default and custom metrics with prometheus so because prometheus is a metrics tool it's used to answer questions that you may have about your application so the questions that we're going to be answering in this video are what server resources are being used how long does the rest endpoint take to respond and how long does a database operation take to respond and do i have any bottlenecks in my application i'm going to figure this out by these two metrics here so what does this tutorial not cover we're not going to cover hosting prometheus or configuring or hosting grafana and the reason i'm not going to cover that in this tutorial is because they are different topics and they're generally covered by your devops or sre team if you would like to see a video on these please let me know in the comment section below before we get started let's take a look at a little diagram so we have our operating system here and on our operating system we have an application running we're going to have our rest api and we're going to build out a metric server and then prometheus is going to scrape the endpoint that you provide with your metric server and then you'll want to visualize what prometheus is stored with grafana so we're just going to cover this bit here the metric server in this tutorial so there's a repository here on tom das tech rest api tutorial updated if you would like to follow along you can clone this tutorial if you would like to see the code written in this video please go to this repository and i'll commit everything that i write so we need to install two dependencies so i'm going to type yarn add and i'm going to add prom client and this is what we're going to use to gather our metrics and we're also going to install response time and response time is going to be used as a express middleware to gather the response time for each request because we're using typescript i'm going to install the definitely typed package for response time i'm going to type yarn add app types slash response time with minus d to store it as a development dependency so i have mongodb running so i'm going to be able to start this server with yarndev and we have ts node watching this over here and the app has started and we've connected to the database let's get started building the metric server inside the util folder i'm going to make a new file and i'm just going to call this metrics.ts i'm going to import express from express i'm going to import the prometheus client and i'm just going to import it as client from from client i'm going to say constant app keyboards express and this is going to be a different instance of express because we need to run our metric server on a different port you don't have to run your metric server on a different port but it's generally a good idea because you probably don't want to expose your metrics to the internet at the company that i work at we have our metrics only exposed inside the vpc and then prometheus is running inside the vpc and it scrapes all of the data from all the endpoints that are all internal so the data exposed by prometheus is all protected so i'm going to say export function start prom server actually let's call this metric server and i'm going to say app.listen and i'm going to listen on port 9100 and then i'm going to have a callback here and i'm just going to say log i'm going to import my log from logger dot info and say metrics server started at http local host port 9100 so we need to add a single endpoint to this server and that is so we can get our metrics and we're going to provide this end point to prometheus and prometheus is going to scrape this endpoint on a configurable interval so usually about 15 or 5 seconds let's say app.get and say metrics i'm going to give this an async callback with a request object and a response object so we're just going to return res.send and we need a way to send out our metrics here so i'm going to say await client dot register dot metrics so when you execute this function you're going to expose all the metrics that prometheus has gathered and we want to set a header for this so i'm going to set the header with res dot set i'm going to set a content type header say client dot register dot content type so we're going to get the content type from this content type string you could hard code it if you want but i find this is a little bit nicer so before we go any further let's collect some default metrics so when i say default metrics this is going to be things like how much memory and cpu your server is using as well as your event loop lag and all good stuff like that and so if for example your memory started spiking and then continued to grow you would know that you have a memory leak and if you didn't have prometheus monitoring your application the application would keep restarting and you wouldn't know why so let's say const collect default metrics equals client dot collect default metrics and then we're just going to execute this function select default matrix and we're going to execute it so the last thing to do before we look at what this endpoint exposes is to start our metric server let's come over to app i'm going to import start metric server from utils metrics and when we start our normal server i'm just also going to start a metric server so you can see here metric server started at localhost port 9100 so i have a postman collection here and you'll be able to find this postman collection inside of this repository in this postman collection json and we have a metrics endpoint here i'm going to say http slash localhost 9100 metrics i'm just going to make a get request to this endpoint let's send this off let's have a look at some of them here so we can see process cpu seconds in total we can see process start time in seconds process resident memory bytes some really good stuff that you'll be able to graph out in grafina so let's add a custom metric so i've come over to the github repository for this prom client you can see here that we can use it to create histograms summaries gauges and counters in this tutorial i'm just going to show you how to make histograms and the reason for that is because most of your metrics you're going to capture is going to be in a histogram and if you're unfamiliar with what a histogram is it's just going to be a distribution of numbers so if we come down to the histogram documentation you can see here that we have buckets so we have 5 15 50 and all your data is going to fit into these defined buckets i tend to not define these buckets i let prom client create the buckets for me and it generally does a pretty good job of that so when we create a histogram it's going to look a lot like this one here without the buckets defined let's come back to our application and i'm going to define two histograms i'm going to say const rest response time histogram equal to new client dot instagram i'm going to give it a name and the name is going to be snake case i'm going to say rest response time duration seconds and it's pretty standard to capture your response time in seconds so i'm just going to add some help text here and the help text is going to appear here say rest api response time in seconds so the labels that i'm going to give it are going to be an array of strings and i'm going to say method so this is going to be like get post delete the route and this is going to be the route that is in our routes file and then i'm going to say status code and this is going to be the status code such as 200 404 500 etc i'm going to spell histogram correctly i'm going to create one more histogram i said const database response time histogram equals new client dot histogram just going to copy the contents of this actually then i'm going to change this to db the database and i'm going to change this to database response time in seconds and we're only going to have two properties here we're going to have operation and we're going to have success so success is just going to be a string that is either true or false so let's go build out the function that's going to handle the rest response time so if we come back into app.ts we're going to build some middleware here that measures the response time and pipes it into that histogram so i'm going to import response time from response time then i'm going to say app.use response time and response time is going to take some arguments and the first argument is a callback and the callback has three arguments the request object response object and the time that it took to resolve this request let's type this request type the response and the time is just going to be a number make sure we've imported x make sure we've imported request and response from express so let's add a condition here to be type save so i'm going to say if rec dot wrap dot off then we're safe to continue let's say rest response and i think i need to export these two histograms quickly do that now we can say rest response time histogram dot observe there's a couple of different ways to use histograms and i'm going to show you two ways to use it so we're going to use observe here and we're going to use a timer when we measure the database response time so i'm going to give this a few properties here and these properties are going to match these label names here so i'm going to do method route and status code there's a method of request dot method a route of request dot route dot half and this is why we've checked that the path does actually exist on the request and we're going to have a status code of res.status code the final argument is going to be the time so the time is in milliseconds but our histogram here is in seconds so to convert milliseconds to seconds we're just going to times time by 1000 let's come back over to our postman collection and i'm just going to send some requests to our application send this request to create a user get an error here create a session get some sessions into create a product get a product and then we can come back over to metrics and we can send this request and you can see here that we have rest response time duration and we have all of the routes that we hit so we have we have slash api users and we got a status code of 409 we hit api sessions with a post request and we got some status code 200s and we got a few 403 errors here as well a few people have asked me as well when they create a session why the access token isn't being stored so you can see when i created this session before the access token wasn't restored and that's why we got the 403s when we created a product the reason for that is because you have no environment selected so come up to here and select your environment and then when i click send on create session it's going to store an access token so i can click the little i here and i can see that it stored my access and refresh token so let's move on to recording the response time for database operations so i'm going to come over to our services and i'm going to come into product service so when we create a product i want to be able to start a timer before we create the product and then i want to end the timer when the product is created so let's say const metrics labels is going to be an object and i'm going to give this one property for now it's going to be operation and that operation i'm just going to get from the function name and then i'm going to say const timer is equal to database response time histogram i'll let vs code import that for me dot start timer and i'm going to execute that function so we need to change this to an async await function so i'm going to say const result equals a weight then i'm going to return result but we have a success property on our metric here so we need to know if this operation here fails let's wrap this in a try catch block and move this up here to return result otherwise i'm going to catch the error and you should handle this error in an appropriate way i'm just going to throw it and so if the promise here resolves i'm going to say timer i'm going to execute our timer i'm going to spread the metrics labels and i'm going to say success is true otherwise success is going to be false let's come back over to postman and let's send this request again and you'll notice that all of the metrics that we captured before have disappeared and the reason for that is because prom client is just going to store these in memory and so once your application shuts down they're going to be lost that's why you scrape it every 5 or 15 seconds so let's log out and then we're going to try create a product and you can see we got a forbidden error let's try and gather our metrics again you can see here we have our rest response time and we have a status code of 403 but we have nothing in the database collection here the reason for that is because we have some middleware that says you're required to be logged in to create a product so let's try log in again logged in here we're going to create a product we're going to get a 200 response this time you can see here that we have our 403 still but now we have some 200s for creating a product and we have some database response time operations and you can see that one request fell into all of these buckets so let's add this timer to one more request and we can remove this return down here because it's never going to get called i'm going to add this to find product so i'm going to await find product i'm going to try it and then i'm going to catch and to move this into that's right catch box and say const result equals then i'm going to return the result i'm going to create a timer i'm going to say that this is successful but down here this is not going to be successful then i'm just going to pro this error as well but we need to change the operation here from create product to find product let's go back to postman i'm going to send a bunch of requests to find the product let's go to get our metrics and we can see database response time so we have a few requests that fell into this bucket two that fell into this bucket here and a bunch that fell into these buckets and we have a bunch of successful requests so that is how to add prometheus to your express rest api thank you for watching and i'll see you in the next video [Music] you

Channel: TomDoesTech

Views: 20,309

Keywords: prometheus, expressjs, typescript, javascript, metrics, monitoring, devops, sre, engineering

Id: HMQ-h3riqYU

Length: 21min 59sec (1319 seconds)

Published: Tue Nov 02 2021