Hide API Keys with a Node.js API Proxy + Caching, Rate Limiting and Slow Downs

so welcome to this content is that what this is this is cup this is YouTube content you're watching me on YouTube but you want content we're gonna give you content we got this question from kvitko who says how would you secure an API without any sort of user login for example your front end hits a third party API using an API key since you can't securely store the API key on the front end and you can't some people might think you can but anything that's on the front end is exposed to anybody that goes to your website so you actually can't hide things on the front end you move that API call to a node back in but is there a way to prevent others from accessing your API now there are a few things you can do but I will say ultimately aside from actually adding a user and password adding like user authentication to your own API which in turn is calling the other API aside from doing that there's a few things you can do to prevent people from like spamming your API and and potentially like running through your rate limit like you can actually rate limit users you can also increase the length of the responsive requests so like each request they make takes a little bit longer and it discourages them from overwhelming your API and then you could potentially create one-off API keys yourself on your own API there's a lot of things to do but I think the first thing that we're gonna do is we're just gonna set up a back-end that actually makes a call to an API and we saw here the we're gonna use the NASA API you can read about here they have rate limits on how often it can get called and basically the issue here would be if someone found your NASA API key NASA API API key they could potentially use that API key and use it so many times that you lose all of the potential remaining requests that you have and so if you hide that API key on your back-end your back-end can cache requests and basically prevent people from seeing the actual API key and what I demonstrated is we're using the demo key here and we have now been rate limited because the demo key is limited to 30 requests per hour so I actually need an API key to make sure that it works now technically you probably could just make a request directly to the NASA API from your browser like let's see really quick if we go to their image API which I don't think requires an API key and we look at this does it have cores headers which would mean you technically could just make that request yeah it does so because this images in point actually responds with access control allow origins star that means that you could request this code directly from your front-end there's no need to put this on it back-end but because we want to hide our API key which that's the reason we're gonna put this on the back end so first thing I'll do is just generate a simple back-end server for that I'm going to use create Express API we're gonna call this NASA API proxy and this is an example of a proxy so a proxy is something that essentially takes in a request does something with it or butt-licker potentially forwards that request to someone else and then forwards that response back to you so we're gonna build a back-end proxy server we're gonna build a back-end API that will make requests to get the weather on Mars it'll get back that response and then it will respond to whoever made that request so if we go into a NASA API proxy we have a basic Express app let's check it out yeah a man-in-the-middle on purpose but it's a man-in-the-middle so that it could do things like hide your API key or cache things or different stuff like that so I generated this is a very basic Express app generator that I created it comes with some tests and some some basic things but we basically just have an Express setup we've got our API mounted at slash API v1 and if we take a look in there we've got some emojis endpoints so if I start this thing up we're gonna get a basic API oh I have to change the the port because I have other things listening so we can see in our index that it defaults to port 5 5,000 or it looks at process env port so I'll go ahead and create a dot env and I'll set my port to be four five four five that's like NASA is it I don't know 45 45 and so now if I started up we now have an API running locally on port 45 45 if I go to slash API slash V 1 slash emojis it responds with an array of emojis very good but what I want to do is I want to create some custom endpoints I want an end point that's API v1 slash Mars whether something like this and that will respond with the Mars weather so let's get that set up I'm going to create a folder in here called Mars weather Jas it's gonna be very similar to the the emojis router but basically when we make a get request to slash Mars weather we're going to this is where this is where the proxy comes into play so step one is we will make a request to the NASA API and then step two is respond to this request with data from the NASA API so basically when someone hits our API we are then going to call the NASA API we're gonna get back the response and then we're gonna send that response back to whoever made that request so that's basically what we're gonna do in this route for now we're just gonna respond with a message that says hello Mars weather so there's that there that 6262 that's possible I'm gonna leave it with all this so I have I have this Mars weather router I need to now mount this router in my full express application so I'm gonna do that in my API index just like I did with emojis so we're gonna bring in Mars weather I'm gonna do bring in that specific Mars weather file and then we're going to use it so we're gonna say router use at slash Mars whether we want to use that Mars weather router and actually gonna do - case so what what I'm putting right here is whatever comes in the actual URL after slash API v1 so I'm gonna make this Mars that's what it like that and just like that we should now have an in point that responds so now if I make a request to API v1 Mars weather it's gonna respond with an object that says hello Mars weather great now what I need to do is I need to make the request to the NASA API and so we we tested this is the actual URL we need to hit but we need to put our own API key in and thank you Randy Orton for those hundred bits much appreciated okay so we will do that here I'm gonna store this URL we'll call this like the base URL and ran the again with the hundred bits thank you very much so that's the base URL and then we can actually wrap all of these params into some URL search brands so I'll say the params R will do this inside the request new URL search brands and Randy with another piece thank you let's try nd for all the bits and basically we're gonna create some key value pairs that have all of these things inside of it so we're gonna have that I'm gonna have feed type which is JSON and we're gonna have version which is Randy with another hundred bits Randy with all the bits thank you very much Randy awesome so these are the parameters that I need to call on this in point but instead of passing in demo key I actually want to pass in my specific API key with another thank you very much Randy so what I need to do is I need to I need to get an actual API key and load that in here and I'm alright this project is already using a package called dot env so technically what I can do is I can say process env Mars or sorry dot nasa api key and randy was a thousand bits thank you so much you're too kind I appreciate you so what this says is we're gonna read NASA API key from my environment and I'll show you right now we have a super basic dot env we're just specifying the pour but in here is where I will put my API key I'm gonna go get the real value but the real value is gonna go in here and this is stuff that I set up earlier but if you look in the app KS you can see that it's already bringing in dot E and V which is a package which will attempt to read in this dot env file if it exists and then put those variables on your environment someone in the chat asked about where API V you want to specify that's specified right here so this is like the root of my Express app and we're putting all of the API endpoints behind API v1 which is why when we look in here it's API v1 emojis API v1 Mars weather cool so I'm gonna hide my screen first put second Randi with more bits thank you very much Randy for all the bits funny dude with a thousand bits so much support so much support thank you everyone you are you were basically I'll say this I'm doing this for a YouTube video but the amount of bits that you've given me is more money than I would make in YouTube ads for like several thousand views way more and I appreciate you this is why I stream on Twitch and why I basically don't really make youtube videos because I people support me here and I appreciate you what am i doing I'm getting my actual API key so what I'm gonna do is I'm going to go into that dot E and V file wherever where I specified that like junk API key and I'm gonna put a real API key that's what I'm gonna do now and you won't be able to see it hopefully hopefully I'll keep it didn't so I'm setting my API in so in that dot env file I'm setting it and thank you very much Callum for those bits and now I think we're I think we're ready I think we're ready to make it happen Paul Beauty beard builder what's up dude who says it must be crazy windy outside cuz those clouds are moving yep that was the Denver skyline here cuz time lapse I don't know what is time okay so like I mentioned I put my real API key into that env file so now when this route gets hit it's actually gonna have something now from inside of this API I want to make a request to this other API and for that I need I could technically use some core modules to do it but there are some NPM modules that will make my life a lot easier so I could use Axios I could use node fetch I could use many other things I'm gonna use Axios because it's the one that I'm most familiar with but basically we just need something that will allow us to make HTTP requests so I'm gonna install Axios just like that and now that we have it we can bring it in so in this particular endpoint I'm gonna bring in Axios from Axios and then now we can make the request so when a client makes a request to our server at API v1 slash Mars whether it's going to go into this request Handler we're then gonna make a request to the NASA API so I'm gonna make this an async function and we're actually gonna put this in a try-catch because it's possible that their API will will fail so well the oh that the API call will fail so I'm gonna try and then catch it if there was an error then I'm gonna forward it on to my error handler which will automatically respond and set the write status code and all that good stuff okay so I will say V we're gonna extract out the data from Axios get we need to make a get request to that base URL base URL followed by these params and I can just do that that will automatically call to string and that will turn these URL search params into a query string separated by ampersands so that should make the request and then once I get back the response I'm actually just going to respond with it I'm just gonna say res days on data so basically a get request comes into my server I then make a request to the NASA API here I get back the response and then I send that response to the client that made the initial request now if I've done everything correctly this should just work now whenever I request from my local server API view on Mars whether that should respond with the Mars weather so behind the scenes it actually made the request to the NASA API gets back the data and then it responds to the client here but you'll notice there is no API key in the URL right so I technically could write a front-end application that calls this in point and that way nobody can see my API key it's hidden behind the scenes and this gets back to the actual question that was asked which is is there a way to prevent others from accessing your API so at this point we've protected the API key at this point it's behind the scenes nobody can see it I could put this this Express API on the web and people could make requests to it however people could technically spam this endpoint right you could write a script that just hits this endpoint over and over again and eventually it would exhaust my rate limit for the NASA API so there's a couple of things we can use to help with that and so I'm going to point you to the Express rate limiter and then there's another package that's very similar which will allow you to increase the timing of a response if people are making tons and tons of requests so the first one is called Express rate limit and oh and also yeah caching is another thing thank you unicorn hunter so right now and I'll show you this right now if i if i refresh the page technically that makes another request to the nasa api and then sends back the response but what if I mean this is the weather on Mars I probably only want the latest data like what every 10 minutes or something like that so we could create a cache I think I'll start there caching is probably the first step to reducing the number of requests to the API that you're calling and I'll just do this I'll say the the cache and and in this case it can be very simple I could just say cached data and that starts off as a having no value and what I can do is I can say at the beginning of this response here and also I could store like the cache time so I'll say if cash time is a thing and the cash time is greater than right now plus some offsets so let's say we want to refresh the cash every 30 seconds so I'll do 30 times 1,000 so now I'll do - so the current time mine I messed up - 30 seconds current time - 30 seconds if the cash time is less than that then I want to get the latest data so actually so if the cash time is greater than that then I'll return the cash a that's what I'll check because otherwise and because then I can just do return res days on with the cash data like that so basically this will first check hey are we in the cash and if we are just return that cash data but then if we're not in the cash we're gonna make the actual request right here I'll say cash data equals data and then I'll say cash time [Music] equals date now and that way the next request will use the cash data and technically to be consistent I should like to put a return here and put a return here so we have consistent returns and I think I do think I did this comparison right and let it be known that this is just a in-memory cache and this is okay right now cuz it's very simple but if you're calling a lot of different API endpoints you may not want to do this in memory because basically I'm just holding on to this variable and if the API ever goes down I will have to make a request again you can store your cache in something like Redis or MongoDB this is like the simplest way of doing it okay let's see if this works and also we're gonna just attach a property to it to let us know that it works so we'll say data cache time equals cache time hello wolfy so now if I make this request it has to hit the API because that's the very first request and if we go all the way to the bottom cache time so the cache time is Wednesday June 24th at 10:00 a.m. GMT minus six and if i refresh this because it was less than 30 seconds ago we can see that the cache time is still stored there and if i i'm clicking refresh down right now and it is it is instant like it's instantly happening but in about 30 seconds when i make the request again it's gonna have to request the api again and there we go it did it so now we can see that the cache time is like right now it is wednesday so this is like the the first form of protection is instead of hitting the api all the time you can now cache it in memory now you you also could use a database or something like that the next step is rate-limiting so express rate limit is a package that i've used and i like it there are other packages that will do a very similar thing and I'm also going to show you Express slowdown but Express late rate limit is very easy to set up you basically just say how many requests you want to allow a given IP address per time window and it will throw errors if you request more than the amount that you're allowed so I'm gonna install this in my back-end here Express rate limit and then we can set it up on this particular route so we can say rate limit we're gonna bring in and yeah this is definitely something you need to do if you're deployed like on in genetics I think you might even need this on Heroku app dot set proxy once so Trust proxy so if you set that in app j/s like it when your app first spins up what that says is when your app is behind a proxy it the the proxy will typically set the the header that says x-forwarded-for and that is actually the the IP address of the requesting client and this says we should trust that header because this code is actually running behind some sort of proxy like Engine X or whatever else or a load balancer okay so we're trusting that and when we deploy that should sure that we have were rate limiting the right IP addresses if we don't do this it's possible that where we rate limit localhost because it's behind the proxy and then every single request is rate limited instead of just the instead of each individual IP address cool so now we set up our rate limiter and we can set up a time window so we can say every 30 seconds I want to allow two requests per IP address now you can increase this you can decrease this but basically what this will do is it will keep track of the number of requests that have happened for the route that you set it up on and if someone exceeds more than two requests in the past 30 seconds they're gonna get an error they're gonna get a four to nine which is too many requests and I believe yeah we can just set it up like this so if you're not familiar with Express middlewares you can actually just put them right inside of your routes so we're saying we request the route right now we just have the handler but if I put this rate limiter right in between now we have essentially rate limited that endpoint so we've said that only two requests can happen every 30 seconds and if more than that happen then it's gonna throw an error so let's try it so now if we go back I should be able to request once we get the data I request again but if I request again too many requests because that's more than two requests in the past 30 seconds so basically the the rate limiter is preventing people from spamming your API so we have a couple of levels protection we're now caching the data we're now primitive requests that people can make and then the other thing that I like to do is Express slow down and so this really discourages people from spamming your API because it will essentially increase the timing of the response for every request that they make so if they send like 10 requests back-to-back each one is going to take a little bit longer to respond which is nice so let's install this Express slowed down and it works in a very similar way and I believe it's from the same person as Express rate limit yes in person it's great it works I've used it for many things and it works wonderfully so we'll set this up as well we're gonna add that here and then we're gonna need to set up our our speed limiter so I'm installing these in the backend so this particular project is an Express project and it has a package.json that's keeping track of all these dependencies so you can see there's Axios expiate Express rate limit Express slowdown and whenever I deploy this it will see that wherever I deploy it to it'll see that package JSON and know what dependencies to install okay so we need to create a speed limiter and so I'm gonna do over and actually I'm gonna allow ten requests every 30 seconds so that's the limit that I'm gonna do and then I'm going to allow well for each request over the next 30 seconds I'm going to increase it by 100 milliseconds each so and we're gonna do and we're gonna delay it after the first request so after the first request every request after that will be in the response time will increase by a certain amount so I'm going to increase the response time 500 milliseconds that that way it will be like really noticeable because after the first request it's gonna take 500 milliseconds more to respond with the next one then it's gonna respond with within within a second then 15 hundred milliseconds and then two seconds so it really really slows down the response sponses and prevents people from spamming your endpoint so I'm gonna first do the rate limiter so that prevents a certain number of requests and then after that I'm gonna do the speed limiter and so we're now going to allow 10 requests within 30 seconds and within 30 seconds each request each subsequent request from the same ip address will be slowed down by 500 milliseconds so now we should get back some data and then it'll take 500 milliseconds to get back some data and then a second to get back some data you can you can watch the thing here to see how long it takes and then 15 hundred milliseconds and then 2 seconds actually it'd probably be better to show this like insomnia because you can actually see the response time and essentially by doing this we're discouraging people from spamming requests to our server because our server will slowly and slowly get less and less responsive to that particular IP address so the response time here was 249 milliseconds then 500 then we should see one second you can see the total response time here and then two seconds and then two and a half seconds so basically if this was if this was a baddie if this was a bad guy who was just spamming our API they would slowly and slowly be slowed down but then after 30 seconds they're no longer limited then it goes back down to the original original response so I think these are the main things that you can do you can do rate limiting slowing down requests caching on top of this you could also create your own API keys and so I'll show you how you would do that really quick but all of these things I'm showing you you could basically get a database involved as well because that right now these are only limiting by IP address but maybe you want to limit by user account so that way if a user is logged in from two different IP addresses you still rate limit that user even though they're calling your API for multiple things yeah and let's see yeah last thing I'll do is all set up in my own API key I'm so I'm gonna do this I'm gonna create an object called API keys and this is again this is just like in memory of course you would like want to put these API keys in a database maybe you could associate them with a user or something like that but I'm gonna have when I create a rotating possibly you also could just do like grid IDs as your API keys as long as you could as long as they're not guessable and they could be looked up in your DB and technically if you're already using off you could use something like a JSON web token what's up flute welcome to the show but actually gonna do this I'm gonna create a new map we learned about a map earlier but maps are basically like objects but they're nicer well do app the API keys set one two three four or five I'm gonna set that to true so this is basically going to be a map of all valid API keys and happy Wednesday to you milky death thanks for being here but this is all valid API keys and then I could write my own custom middleware that checks to make sure that there is an API key in in the request so I'm gonna create my own little we're here with rec resin next to validate the header so I'm gonna say the API key comes from the header X - API key so I'm gonna require users that are calling my API to specify a header called X API key and then I'm gonna validate that so I'm gonna say if API Keys API keys that has this particular API key then we're just gonna call next then that means everything's good go on to the next middleware which is this which is actually going to make the API request otherwise we're gonna throw an error we're gonna we're gonna forward an error so we're gonna create a new error and we'll say invalid what's happening John is content content is happening right now so we have this error and then whenever you call next with an error or with a parameter that actually will will skip all of the the the request handlers below it and that'll go immediately to my error handling middleware which will just respond with an error here so what we've done with this is in this middleware we're checking for our own API key so we want to make sure that anybody that requests our API is using the API key one two three four five and if they're not we're gonna respond with an error so now it insomnia if I make this request it should give an error invalid API key but if I specify the X X API key header and actually if I put it to something wrong it should still say invalid API key but if I set it to that API key one two three four five we then actually get back the response and as you can see I actually put that API key validation middleware behind the speed limiter and the slowdown because if somebody is trying to just brute-force API keys they're gonna it's gonna take longer and longer for them to be able to validate if they are brute-forcing in the right way because this API key validation is behind the limiter and the rate limiter and the speed limiter cool now like I mentioned you would want to store these API keys in a database maybe you could also validate by user all that good stuff right now it's super simple it's just looking in the map but this gives you an idea of how you would do that as well I think yeah so the middle warriors in this case go in order first rate limit the number of requests by IP address then increase the response time based on the number of requests that that IP address has made in the last 30 seconds then validate the API key and if all is good we'll do this and if we've cached the data we're just going to respond with our cache data and then if we haven't we're gonna actually make the API request great no they wouldn't so if two people with different API keys try to access they're gonna be rate limited initially by their IP address so as long as they're calling from two different IP addresses then they will not be rate limited and the request will just basically happen yeah cool I think that's it thanks everyone for watching this and this all started from the question which was how would you secure an API without any sort of user login I think I think I would start here I would basically say cache your data for one because that will prevent you from running out of your your quota limit for some API that you're calling number two is do rate limiting by IP address and then also slowdowns by IP address and then beyond that you can start to implement your own authentication on top of your API which is setting API keys or potentially even checking to make sure that a user is logged in but ultimately this in point that we've created you can now create a front-end behind this you could create a react app or a view app that shows the Mars weather but it doesn't exhaust all of your available quota for the NASA API or whatever API you're hitting cool thank you everyone for watching I'm sure I missed a lot of questions and stuff but I think I'm gonna I'm gonna acknowledge the follows I

Channel: Coding Garden

Views: 83,568

Rating: 4.9685931 out of 5

Keywords: live streaming, live coding, lesson, html, twitch, beginner, learn javascript, css, full stack javascript, vscode, node.js, educational, full stack, web development, full stack web development, learn node.js, javascript, mechanical keyboard, games

Id: nCWE6eonL7k

Length: 31min 30sec (1890 seconds)

Published: Sat Jul 04 2020