REST API Rate Limiter in NodeJS using Redis and ExpressJs | REST API Rate Limiter | Redis

hello people welcome back to my channel and this video we'll learn that how we can create a rate limiter for our node.js application so rate limited is basically used to rate limit your API calls which can provide a safety measure to your API so that user can't hack it or can't hit large number of requests to your server so let's understand what we'll be building so this is a rest API interface and it can be connected to a micro service which is basically using ready server or it can connect to another micro service which is using database so this can behave as an orchestrator where it is called as API Gateway so this can be a API Gateway or individual micro Services can also expose their own rest API so we'll be having the client it can call to this API Gateway or it can directly connect to the micro service also whatever may be the scenario for your app so this client can call your rest server with large number of requests and it can hamper the functionality of your whole application so if the request is large then what happens it increases server load so more resources are utilized like RAM and CPU since the number of requests are increasing so the RAM and CPUs are also increasing nowadays we are using Amazon AWS or Google compute engine and this can increase the RAM and CPU usage there and finally your billing can also increase the third party Services cost should be in check so let's consider that this is Uber so what happens the Uber microservice calls this Google Map EPS to basically find that where the driver is how to book the cab at a particular location so basically they are using Google Map apis and this is chargeable Beyond certain throughput if the number of request increases on this micro service so what will happen the number of requests will increase on this Google Map apis and finally Uber will be charged more and hence more billing will be done to the Uber guys and this can create a problem so we should also keep a check on the third party Services by rate limiting our apis so what we are going to develop is like we will be developing a rest server in node.js will create a get API which will return dummy response so we'll mimic that there is a API present on our s server and we'll be calling that API and finally rate limiting it so we will rate limit this get API and we'll be using a fixed window algorithm so there are multiple algorithms like fixed window or sliding window algorithm which is used for rate limiting but we'll be using the simple one that in a given time a user can access the API only 10 times so let's say this is the timeline and the user gives or hits the server the first request comes here so since it is a fixed window algorithm so the timing is defined that within this time the user can do only 10 apis calls so this is fixed window algorithm this is the same scenario where we can say that the user hit the request here and since it is a fixed window algorithm So within this amount of time only 10 API calls are allowed so what are the prerequisitives so you need to have a ID to code in node.js and redis must be installed I have created a blog post here that is install red is using Docker compose and there is a video also like install radius using Docker intercore compose you can follow these two things to basically install redis and fulfill your prerequisitives I'll share the links in the descriptions below so once redis is installed we need to do the environment setup so we'll create a project in the ID for node.js development and I'll be using webstorm in webstorm we'll click on new project and here we need to select the project directory so I am using WSL of windows so I'll need to go to WSL and then home inside home we need to select the sandbox folder and inside the sandbox I will say rate limit using node.js click OK this will create a folder I'll select OK and hit on create this will create a project for me on webstorm and here you can see that rate limiting using Windows folder is present and it is currently empty in the terminal we'll go to that folder so it is inside sandbox and then inside sandbox if I do LS you can see CD rate limit using node.js I'll do LS it is empty click npm in it because we are creating a node.js project give everything as default and finally if you do LS you can see that package.json file is present here which basically holds all the dependency for your node application for doing the development purpose we need to install different libraries which can help you to create this Express server so this is Express servers which will expose API and for that we need the express Library we also need nodemon which is basically a node module which helps you to run your project in a watch mode so that if you change some file the server will automatically get restarted and finally we'll be creating the express server instance let us go to the IDE and here I'll create a new file that is called as index.js first we need to install all our dependencies for that let's go to terminal and I'll say npm install Express save so this hyphen iPhone save will basically save the module inside your package.json file and then I need to do sudo in npm install nodemon hyphen G this will install nodemon I have given a hyphen G flag because I need to install it globally so that it can be used for all my node.js projects so let's go to the slide again and here you can see that we need to create a ping API on the server so this ping API will return as a response as success true and message as found so whenever someone will hit this request so we'll get this response so this is the API which we are going to rate limit so in first scenario what will happen a user can come to the web server he can hit as many times thousand times million times and he'll get this response every time but then we are going to rate limited and finally we'll be starting the server using nodemon index.js so let's go to the terminal and we can say nodemon and enter so this will start the server in watch mode let us create the app instance and then the get API so we'll say require and then Express and finally we will create a app Express instance so we can say const and then Express and finally we'll say const app equal to express so this is the express app instance and we also need to provide a port so that we can start the server so we can say app Dot listen and then the port will be 7005 comma function this is a callback function which will be called once the server is started and we'll say console.log and then server started at Port 7005 so this is the port value 7005 so this is how the server will get started let's visit our terminal and say node Mon and it will say that okay server started at Port 7005 let's add the get API to the server so we'll say app dot get and the API route is Ping and finally we'll say async and then the request response and finally we'll create the place where we'll write all our code so this is the Ping API that has been developed and let's say we are returning the response as response dot status so we'll return a 200 HTTP status and the Json value that we are going to return is success equal to true and then finally we'll say message equal to Palm let us go to postman and hit the API so it is get API localhost 7005 ping and once we do send you can see it is Success true message equal to pound let's try to hit multiple time so you can see that every time I am getting a successful response so you can see here I can hit this request million times and I'll always get this response but when we hit it million times which means that the resources will increase on our server and that can bring down the server to avoid that we'll be creating the rate limiter function or rate limiter logic but before that we need to understand that where this middleware will fit so to understand the express request response cycle we can say that at the start of the request response cycle we have two objects that is request object and response object and it is a set of middlewares like we can have multiple middle layers in the between each middleware will call next and and then it will go to the next middleware so this is the architecture what Express follows and finally when it comes to the response layer that is the final middle layer it will return the response so we will fit our rate limiting logic in the middleware at the starting like in the front only so that every request can be processed and it can be rate limited based on the user ID so you can Define the rate limiting over all the apis or you can do individual apis also like individual apis can have their own rate limiting or you can apply it across all the API so what we'll be developing is like across all the apis so what will be our rate limiting logic so let's say that the first call for the user one comes at time T1 so we need to store some values like this like the user one is there and when did the request came so we will say that it is stored against created at and the time will be stored so this time will be a Epoch timestamp which is like in the integer format and and then we'll store the count value also as one since it is the first request so this is the T1 that is the first requested car which came and since we are using this fixed rate limit window and we are saying that it is one minute so what our rate limiting logic says that within one minute the user can hit 10 request 5 requests something like that that will be a constant that we will be defining so in our case we'll be using it as five times so this is the place where the first request came and this is the fixed rate limit final window that is T1 plus one minute which we have defined let's say that the second request comes after this threshold like the after this fixed rate limit window type since it is after this one so what we'll say that we need to again reset the bucket we don't need to consider this rate limiting we need to start a new rate limiting because the second request is coming after this T1 plus one if the request comes before this one so here we need to apply our rate limiting logic because it is coming within the this fixed rate limit window time so what we need to say that if the second call comes before the threshold so we need to check if the rate limit count is reached because it is falling in between this so we need to check for rate limiting so we will say that okay is the countries is the number of time which the user has been given has exhausted so we will check it from here that current account is one it can be two three four five anything so we need to check against that that what is the current count if yes that the limit count is reached so we need to return error that is rate limit error if it has not reached so we'll increase the count by one and it will allow the API to call the API controller where the business logic will be executed and finally a successful response will be written so since we are dealing with time so we will install the moment module and we will also install the i o radius module because we will be storing these data inside redis so let's install these two libraries first so let's go to the terminal we'll do Ctrl C npm install moment hyphen iPhone save and then npm install IO redis hyphen iPhone save so this will install both the modules and then let's go to the idea so here we will be creating our rate limiting logic so I'll click new file and then rate limiter.js so this is the file where you will be creating our rate limiting logic now this is a middleware so we need to write the contract for the middleware so we'll say module dot exports and then we will say rate limited is the function name which will be a async function with request response and next next thing is like we need to import our i o radius module and the moment module and we also need to create a redis connection so it is like we need to require i o radius and then finally we need to call new redis with the URL and this is the radius protocol and then localhost 6379 since our radius is running on our local machine so the redis client instance is present here we also need to Define two variables or two constants that is ready limit duration in second since we are saying that we are defining a fixed rate limit window algorithm so we need to Define that window and that is like rate limit duration in second that is 60 seconds which is equal to one minute also we need to Define that how many times the user is allowed to hit the API so this is the number of requests allowed that is equal to 5 if he goes beyond 5 within one minute then we will rate limit him since we want to uniquely identify the user so we will be sending user ID in every request so we'll say request dot headers user ID so this user ID will come in the header for every request and will say user ID and it will also be a key against which will will be storing all the information of the user like what is the current count what is the created ad and everything we also need the current time so we'll say current time equal to moment dot Unix so this is the way how we'll get the current Epoch time once these things are available so we want to go to the redis we want to check that what is the current information about that user so what we are going to say we are going to say await redisclient dot h get all so we'll be storing the information as a hash in the radius because we see that this is the user ID that will be a key and we want to store multiple information that is created at an account so it is just like a hash map so we'll do the same thing and the educate all will be stored against the key that is user ID so we'll say user ID and we'll get this information so this will be result once this is available so we need to check that this is the first call or this is not if it is a first call then we need to initialize our radius with the information with the current time and the count equal to 1. to check that this is the first call of the user so what we can say that the result that we have got will be empty because that user is not existing so we can say object dot Keys result dot length equal to equal to zero since there is no value inside that result data so we'll just say if the number of keys is blank which means that the this is the first request since there is no entry in the radius so we will call that client which this with this hset method which will basically set the hash against that user ID and it will have two keys one is created and and another one is the count so that is current time and this is the value is one since it is the first request now let's say there was some existing data in our radius so if there is some existing data which means that the already the first request has come then we need to check if the second request that has come is after this fixed rate limit window so what for that what we need to do we need to take the time difference between the current request which has been received and the existing data which existing data or the existing last request which has come and if it is after this fixed rate limit window time which means that we don't need to consider this rate limiting we need to start a new rate limiting fixed window time for that we need to check the result if result is already present there so we need to find the time difference so we need to calculate it by current time minus result and created it because we are storing this created at value for the previous instance so this will give a difference and we need to check if the difference is greater than the rate limit duration in second then we need to reset the bucket and return next so we'll say return next means what we want to basically send it to the next middleware and where the processing can begin so this case is what if difference is greater than rate limiting duration will allow the API to give a successful response now if the difference is less than the time which means that the request is coming within the this fixed rate limit window so in that case what we need today we need to check if the rate limit count is already reached in this case we'll say result count is greater than equal to number of requests allowed second request has come within this time frame so we need to check if the count has been breached or not so for that what we'll do we'll say result count which is there in the red s is greater than equal to the number of requests allowed so in that case we need to return error that the status is 429 which is basically for large number of requests success equal to false and message equal to user is late limited if this is not the case that the count has not been reached so in that case we need to increase the count value by 1 we we are going to just paste this else that is weight redis client at set user ID and we want to increase the count by 1 and allow it to pass to the next middleware so we can say return next so this is the whole rate limiting logic that is present here now to include this rate limiter as a middleware we will go to this file that is index.js and we'll say rate limiter equal to require and since it is a file which is present in the project so we need to use dot slash app instances here so we'll say app dot use and then this rate limiter dot rate limiter this is the instant so now this rate limiting logic has been applied let's go to our terminal here and you we will start the server again so it will say node mode and the server has already started let's go to the postman and here this time we'll send the request so it is saying true true 3 4 5 6 okay so see when the number of request is 6 so it is already rate limited like false user has been rate limited so you can see that after the fifth request the user has been rate limited and we are not able to pass the request let's go to our redis and we can just say ready CLI which will connect to our local radius instance here we'll say keys star which will give all the keys and this is the user ID that we are passing so this user ID is passed inside the header so inside the header we are sending user ID one one one one one one five time one and two which it can be anything so if we go here we'll say h get all and then we'll say the key value one two three four five and you can see we have stored these two information that is created at an account and let's do it again like let's CH get all this is 5 and again let's try to hit the request because one minute time has elapsed let's see so this time you are again getting true and pong and if I go to radius again you can see the count has pin again reset the discount is one previously it was five so let's go again and hit the request from the postman send two three four five when it goes to six the user has been rate limited so this is the rate limiting logic that we have developed in this particular project so hope you like this whole project hope you like this video please comment that what best you like in this project what Improvement needs to be done please share these videos with other people and if you want to thank me you can go to my YouTube videos select on this thanks and you can give me a credit that's all so thank you everyone

Channel: CodeWithRajRanjan

Views: 1,775

Keywords: rest api ratelimiter, nodejs ratelimiter, api ratelimiter, redis based ratelimiter, fixed window ratelimiter algorithm, redis, expressjs

Id: MChPvKTTIMk

Length: 19min 46sec (1186 seconds)

Published: Sun May 14 2023