Build an API Proxy Server - Hide Your API Keys, Rate Limiting & Caching

[Music] what's going on guys in this video i'm going to show you how we can create a basically an api proxy server or relay server using node.js and the main reason i want to do this is to show you how we can hide our public api keys by using our own server because in a lot of cases when you use a third-party api like the github api or twitter or in our case we're going to be using open weather map a lot of times you need to include your api key directly in the url when you make your request so here i have a very simple just client-side vanilla javascript weather app that uses open weather and we can just put in a city and it just you know gets the the temperature but i want to show you if i open up my source code here and i look in my main js you'll see that in this fetch weather function i'm making a request directly to open weather map and i have my api key or my app id directly in my client-side code which you really don't want to do so there's a few ways around this and in one way is to just have your own server your own api proxy server which is kind of like a middleman that you can make requests to and then store your api key on the server so that's what i want to do and in addition to that i want to show you how we can add things like rate limiting so people can only make a certain amount of requests per a certain amount of time and we'll show you how to add some caching so we're just going to create our own server so let's close this up and right now in vs code i just have i have a folder called node api proxy server and i just have a public folder with the front end files which you're seeing right here right now i'm just using live server to uh to display in the browser i'm actually going to cut that off because we don't need that so we have an index html some styles and then the main js so this is the front-end javascript this is what i just showed you in the browser and just to quickly go over it so you know what's going on we have a fetch weather function and we're we're we're requesting from directly from the api here and i have the api key right in the code which obviously we don't want that's that's the purpose of this video and then down here i'm just getting the data i'm formatting it and passing it into this add weather to dom which will then display the the data here the city and the temperature and then down here i just have my event listener for the form which will call fetch weather with whatever city i put in the form and we have an initial fetch with the city of miami passed in so that's the the front end javascript very simple now if you're following along which i would definitely suggest you do you can just create a new folder you don't have to worry about this public folder right now because we're just going to use postman to make our requests and you should also create your own api key from open weather map and you just log in create a free account and then go to my api keys and you should see your key here so make sure you use your own and not mine so first thing we want to do let's open up a terminal and we just want to run npm init dash y which will create a package.json file and then as far as dependencies we're going to be using quite a few right now i'm just going to install express let's install dot env that'll allow us to have environment variables that's where ultimately where we want to store our api key we're also going to install cores and we're going to install needle which is an http client a very lightweight client that we can use on our server so basically we'll the front end will make the request to our server we'll use needle from our server to make the request to the public api to open weather all right so now that we've done that let's um actually i want to install nodemon 2. so let's do npm install dash uppercase d because we're installing this as a dev dependency and that will just allow us to run our server without having to restart it every time we make a change and then in the package.json you should see your dependencies and let's go ahead and add a start script here so for start we're just going to say node i will say node index i'm going to call the main entry file index.js and then let's also do node mon and this one is going to be our dev script so npm run dev okay now let's create a file so say touch index.js and that's going to be our main entry point so from here let's start to create our express server and we may do a deployment as well because i know it can be confusing now that we have a back end when we have a server i think heroku is one of the easiest platforms to to deploy something like this so we might do that at the end if we have time so let's uh let's require express here and let's also bring in cores and then i'm going to set a port here port number we're going to set that to process.env.port so basically it's going to see if we have an environment variable called port first if we don't then we'll use 5000 and then let's see we'll go ahead and set up express or initialize express and let's just going to put some comments here we're going to enable cores so app.use cores and then let's go ahead and app.listen we want to listen on that port and we'll just have a call back here and say console.log put some put a template string in here and say server we'll say server running on port and then whatever that port is so we should be able to run this if we do npm run dev all right so now you can see it's watching on port 5000 so let's see what do i want to do now let's take care of just storing our api key so what we're going to do is we installed env right yeah we installed dot env so to use that we just have to go up here and require it so we want to say require dot and we just want to add on to this dot config and now in our root here in the root of our file structure we're going to create a file called dot env period env and the way that i'm going to do this i'm going to store a couple of things here first is going to be the api base url now i'm creating this in a way that is sort of reusable so you can use it with other public apis and not just open weather map so i'm going to do things a little bit different than if it was just for this api so let's put actually i'll just grab this it's going to be let's see it's going to be this right here right before the the q param here make sure we want https all right so that's going to be our base url then i'm going to have our api key name so what i mean by the name if you look here you pass in app id that's going to be the name and then the api key is going to be the value so the name for this is going to be app id and then let's do api key value that's going to be your actual api key which for me is this here okay so those are the three environment variables that we're going to have so we're going to close that up we'll get to those in a bit let's uh let's go ahead and create a route here so and if you're not familiar with express you probably should watch like my node crash course and then my express crash course first because i'm not going to go over what routes are and all that but i want this to go to slash api and then we have our function here this takes in request response and i'm just going to res.json so i'm going to return with jason and just put in for now success true and now i'm going to open up postman and you can use any http client you could use your browser if you want but there's some headers and stuff that i'm going to want to show you later so this is more helpful but let's do http local host port 5000 slash api and we should see success true should be a 200 response a 200 status now i'm not going to want to put everything in the route i'm not going to want to put in this file so i'm going to create a folder called routes and inside routes you could be very specific and put like weather.js since this route these routes are to deal with the weather api but again i want to make this kind of reusable so i'm just going to call this index.js and then to use the router we need to bring in express so let's require express and then let's say const router set that to express dot and we want to do a capital r router and then before i forget let's module dot exports because we want to use this in another file and then as far as the route we're only going to need one so i'm going to just grab this cut that put that in here whoops let me just cut that again and let's put that in here we just want to change app to router okay and i'm going to change this to just slash now i want to have slash api go to this route automatically so what we're going to do is include our routes here and let's do app.use so we're going to say whenever we go to slash api we want to require the routes slash index file now since it's called index we don't even need to put index we can just simply do routes so if i save that this should still work if i send you'll see we still get success true so we hit this route and it's going to look in this file at this route right here all right so now that we have that set up let's let's go ahead and bring in needle which is what we're going to use for our request so let's say needle set that to require and you you could use something like node fetch if you want but i think the latest version of node fetch you can't use common js syntax um you have to use es modules i had some issues with it um so we brought that in now let's see what we want to do next um yeah let's get rid of this and let's make our request now that we're going to use a sync away because needle does return a promise so we want to make this function async and then let's say const api response don't call it res because it'll get tricked up because of this so api res and that's going to equal needle or i'm sorry a weight needle we're using a sink away this returns a promise and let's say this is going to be a get request and then the url is what we have in our dot env so what i'm going to do is go up here and let's say environment variables i'm just going to put them in regular variables so api base url is going to be equal to process dot env dot api base url we might as well bring in the other two that we need which is api key name api key name api key value value and process.env this is just how we get our environment variables okay and when you when you add environment variables or you change them you do have to restart the server so i don't think i did that so i'm just going to restart that real quick and then we'll go back to our request here and the url that we want let's put some backticks in here and we're going to put in our api underscore base url okay so that will make the the request that'll put the response into this variable and then what we want is api res.body that's going to have the actual weather information so i'm going to have a variable here called data and let's set that to api res.body and then that's what we want to respond with so let's say res we can do res.json but i usually like to put the status anyway so set it to a 200 and then dot json and then data and you know what i actually meant to put this in a try catch so let's just add a try catch here and then we'll take this and put this into the try and then if something goes wrong then let's do a res.status 500 which is an internal error server error and then let's do jason and just put in whatever the error is and we can add a custom error handler later if we have time so let's just save that and let's see what we get so we'll go over here we just want to hit that route again and what we get is to be expected we get a 401 which is unauthorized and we get invalid api key the reason for that is we didn't include any api key on the we're just making a request directly to the url so we have the api key value here we also have the name so we want to attach those as query params now we could do that just manually like this you know we could have whatever app id equals blah blah blah or we can use url search params which is what i'm going to do so let's go above here and let's say const and we'll call this params and set that to new url search params and then this takes in an object of params that you want to add now what we want to add is our app id which is the the key name right so i want to use this as the key of this this object item so to do that we need to add brackets and then we can put our api key name which is going to be app id in this case because that's what we set in the dot env and then that's going to be equal to the value okay now to add that as a param we can go over here and just put a question we want to go outside of the i'm sorry we want to go inside of the template string here put a question mark and then we want to put params so that should attach the api key to that so let's go back here and send and now we don't get the you know api key message we get nothing to geo code and that's because we didn't it doesn't know what city we want to add because if we look at the format of the request we already we have this done right but we don't have this this q equals city name and what i want to do is make it so that whatever whatever query params we pass in to our server i basically want to forward onto the public url endpoint so there's a few ways to do that what i'm going to do is bring in up here let's say const url so we're going to require the url module which is part of node so we don't have to install it or anything and then let's see i'm just going to console.log just so you can see what this does let's do a console log of url dot parse and we can get the url with request.url and then is it url yeah and then true and then i want to do dot query because i want to be able to get whatever query params are passed in so if i were to go back to postman and right now i have nothing passed in as a as a query param and i send we're just going to see an empty object in the console down here but if i were to add let's do question mark and we'll do q equals boston if i send now you can see we get q boston and it'll it'll add whatever i put here so if i do like ampersand hello equals world and send that'll get added to the object okay and then that's why this is going to be kind of reusable because whatever you pass in is going to get forwarded on to that onto the public api so now we're just logging that but what i want to do is cut that and i want to add it to my url params here but i don't want to just add it like that i want to spread across the values so i'm going to add the spread operator in front of it okay so now what should get added on to the url to the you know the api url open weather is the api key with the key name which is app id and also anything i pass in including the the q equals whatever city so now let's save that and let's make this request again with this q equals boston and now we get the weather okay so we're hitting our own servers endpoint and then it's basically forwarding to the open weather api we're getting that response back and we're returning that data back from our server and in fact you know what we could do is in development mode we could log down here the actual uh request that's being made to our public api so let's let's just say if and we'll do process dot env and we want to check to see if we're we want to see if we're not in production so node underscore env if we're not if that's not equal to production then let's just do a console log and we'll put in our template string here and i'm just going to say request colon and let's get the api underscore base underscore url and then we want our params so it's just going to log whatever is being requested so let's send and now down here you can see exactly what is being called we have the open weather map base url we have our app id equal to our api key and we have q equals to boston you don't have to do this it's just to show exactly what's happening here so let's just say log the request to the public api okay now i mean we are pretty much ready to go we could just go ahead and use our custom endpoint here in our front end but i do want to show you a couple other things so the first is going to be rate limiting so let's stop the server for a second and just install something called express dash rate dash limit i'm also going to install something called api cache and then we'll go ahead and run the server again now for rate limiting i'm going to put this in the the main index.js file basically i want this to be for any any and all routes so let's go up here and let's say const and say rate limit and set that to require express rate limit and then we just need to add the middleware let's see we'll go right here and let's say rate limiting and this is in the docs if you want to check that out but we're going to call this we'll say limiter set that to rate limit and then this takes in an object of options first is going to be window ms so this is going to be the the time that we want the user to um you know let's say we want them to do a hundred within 10 minutes max so let's let's make 10 minutes here this is going to be milliseconds so let's do 10 times 60 times 1000 okay so that's going to be 10 minutes and you can set this to whatever you want and then max is going to be how many times in 10 minutes can they make their requests or how many requests in that time so let's make it something low so we can test it out we'll say five okay so that puts it in the variable now we just need to add the middleware with app.use pass in limiter and then we are using this as a proxy so let's do app.set and we just want to set trust proxy to one okay now this should give us limiting now so let's try it out so i'm over here i'm going to make a request that's one time two times three times four five and now six you'll see that i get too many requests please try again later so this will stop people from spamming your endpoint and you could even see i'm going to change this to a hundred and we could even see in the headers here let's see so right here x rate x rate limit limit we have a hundred because i set it to a hundred and this shows us what we have remaining so 99 remaining if i send that goes down to 98 97 and so on once i get to 0 it's going to basically just kick me off for 10 minutes okay and of course you could set that to whatever you'd like so now let's go ahead and add some caching so i'm going to go to my route here and we're going to use the api cache package that we installed so let's say api cache set that to require api cache and then we want to initialize our cache so we'll say let cache we'll set that to api cache dot middleware oops and this is in the docs if you want to check out the api cache docs and then all we have to do to use middleware we just add it as a second argument here so in here we can add cache just make sure you put a comma here and then you can put the time you want let's say two minutes so it's going to cache it for cache the response to this route for two minutes should be minutes all right so now if we go over here and we make our requests we get our data back but let's take a look at the headers and you'll notice now we have this cache control it says max age 120. so at the time of making this that was our first request so we have 120 seconds which is two minutes because that's what we set here so if i send again now 104 so it's just it's going to give us the cash response for this amount of time 104 seconds 97 so it just keeps counting down once it gets to zero it'll give us you know the it'll make the actual request and it'll store that for two more minutes and you could add you could put this to whatever you want and it really depends on the api on how long you want this if it's something that's constantly updating you're gonna you're not gonna want this to be too long all right now i think we're all set to actually use this with our client-side application which we have in the public folder so with express we can make a certain folder our static folder which i'm going to set the public one to so let's go uh we'll go right above the routes here and let's say set static folder and to do that we can just do app.use and in here we want to say oops we want to do express dot static and we're going to make the folder named public our static folder so we should be able to go to local host 5000 and it should load our index html so let's open up a browser and let's go to http local host 5000 okay and if we open up our console down here now we're still trying to make to make the request to the public api if we look in our public main js so right here and we don't want to do that we now want to make the request to our own api now there's different ways you can do this we happen to have this like you know we have our public folder we have our front end on the same server you might want to just deploy your api somewhere else with a specific domain and then you would add that you know you could connect to that domain here but in our case we're just going to simply do slash api not slash we want to add the q param here or q string to whatever the city is that's passed into this function so like that all right so let's save that now and let's reload and now we should be able to get our weather but if we look at our source code unlike before if we look at our main js and we look at our fetch weather there's no api key here okay we're just we're we're interacting with our own server and there's the api key is stored on the server side it's not on the client so someone can't come along and steal our api key all right and and you can do this for working with any public api it's not just open weather and i tried to make this so it's kind of a boiler plate that you could use with other other projects as well so i think the next thing we'll do is actually deploy this so to get ready for deployment let's go ahead and create a dot get ignore and we're going to use heroku which get not got we're going to use heroku to deploy and we're going to use git to do that so we don't want to add the dot env file right because that has our api keys in it we want to keep that out of our repository we also don't want node modules so we're going to add those and then i'm just going to create a a dot env dot example just for my github repo for on github so that people know um you know what they need to put in here so i'm gonna copy this put that in there and then just keep these blank all right so we can close all this up next thing we want to do is let's go to heroku and you just if you don't have an account just go ahead and create one uh we'll do that later all right so all your projects will show here if you don't have any obviously you won't see any and you want to install the heroku cli which i already have installed but you can see right here if you're on mac you can use homebrew windows you can install it ubuntu you can use your package manager so i already have it installed so i'm going to just go ahead and close this up and we want to just do heroku login press any key it's going to open up in the browser and we'll just click login okay so now i'm logged in with my heroku account next thing i want to do is create a heroku application so let's say heroku create and we can name it i'm going to call this i'll just call it node oh let's call it something to do with open weather we'll just say i don't know call it traversie weather app so it's going to create that application and if i go back to my dashboard here and reload i should see it okay there's not going to be anything there if i click on it and i go to open app it's just going to be a welcome page we haven't pushed anything to it yet so what we want to do next is go to deploy and we want to add this git repo so we're going to copy this right here yours will be called something different and then let's first initialize the git repository get in there we haven't done that yet and then let's go ahead and get add everything let's get commit say git commit am initial okay and then i'm gonna add i'm gonna paste that in i'm just adding the heroku repo and then we should be able to get push heroku master okay so now this should push everything to heroku okay so that should be all set now let's go back over here and let's check out the app now it's not going to work right now because it our environment variables remember we store on our development side we stored those in our dot env which we didn't push we put those in the git ignore so the way that we can add environment variables in heroku onto the server is by going to settings and go down here we can say reveal config vars and we just want to add those three to this so we can just grab that so this is going to be api base url right api base url and let's paste that in all right so we get that and then api key name is going to be app id and then let's grab the actual value paste that in this is going to be api underscore key underscore value okay so we're going to add these to our server as environment variables and our our app by api key is stored here so it's on the server it's not accessible to the client and then let's go ahead and whoops that's localhost let's get rid of that and it might take a couple minutes after you enter those those environment variables but now as you can see this is working we see our weather in miami this is our live application let's try a different city if i put something that you know isn't a city we get city not found let's do new york and again if i were to look at my main js here when we look at the request there's no api key or anything like that we're just requesting it from our own server where we have the api key stored all right so that's how you can hide keys and you could add things like rate limiting and caching and there's other things you could do as well so that's it i hope you guys learned something from this and i will see you next time

Channel: Traversy Media

Views: 134,312

Keywords: node.js, hide api keys, api keys, proxy server, node.js proxy, rate limiting, caching, node.js caching, express, expressjs

Id: ZGymN8aFsv4

Length: 32min 19sec (1939 seconds)

Published: Wed Oct 20 2021