API Key Authentication Best Practices

hi in this video I want to share with you some API key authentication best practices we have spent a lot of time at zuplo designing our implementation of an API key authentication Service which is part of our Gateway and in doing so we also spent a lot of time researching how others had gone about solving these problems and so what we've done is we've collated some of the the things that we think are best practices what if you're going to implement API Keys yourself these are some of the things where you should really think about these are the decisions and some of the trade-offs that you're going to need to make the first question though is like why why would you use API Keys why not we've written about this on our blog in the past and there's a couple of reasons why API keys are attractive the main thing you're comparing that against is probably jot tokens JWT and oauth and I think oauth or we think rather oauth and jot are a a great choice especially if if your API is doing something on behalf of an individual user so you remember back in the day when everyone was building their own Twitter apps that was on behalf of an individual user that was calling the Twitter API oauth is great for that in some cases you have um GitHub apis that actually support they have API key apis and they have apis that use jot tokens the apis that use jot tokens are the ones that are doing something on behalf of a user the ones that use API keys are the ones that are doing something on behalf of a sort of a system or an organization or a back end that's not a real human being and so their oauth typically makes less sense um I think actually what GitHub decided to do is an excellent they're an excellent example of how we think about it and uh and I'll explain some reasons for that if you look at the the world's best API first companies in the world the people that come to mind are people like stripe like twilio sendgrid I read really like our table they all have one thing in common and that is they all use API Keys once again their apis are not typically doing something on behalf of an individual user they're doing something on behalf of an organization you know so when we call the stripe apis we do so as zuplo not as Josh twist and I think that makes API Keys a great fit and some of the reasons for that are well one uh they're much easier for engineers to use and with engineering and making developer experience work it's really about avoiding Death By A Thousand Cuts you know the best experience is they might seem like trivially better but that adds up to just everything being easier so some examples of things that are easy with API keys I don't have to think about complicated oauth flows you know I just get my API key I stick it on the header and I send it in testing the API that really makes everything much easier you know a curl is like one curl command it's not client ID client secret building this complex a wealth dance of redirects and then pulling my jot token out of something and then sending an API call that's actually quite painful it's important when you're doing on behalf of a user but if you're not it's questionable in terms of value and this is why these great companies use API keys they really care about how easy it is for engineers to be successful using their API so they spend a lot of time reducing or focusing on optimizing that time to First Call how long does it take to invoke this API so that they feel successful and they're making progress one of the biggest complaints against API Keys is that it's not a standard and I think that's true you know the nice thing about jot tokens is that they're fairly standardized at this point and you just go and follow the protocol and it's well written that's one of the reasons we've put this guide together though I don't think you should discount API Keys you know if it's good enough for stripe if it's good enough for twilio it's good enough for you the other argument against API Keys is security this is also pretty debatable uh there's no you know this pros and cons here actually and again if you know if the likes of stripe and so on uh where it gives you sort of access to The Farmhouse and the barn um through their API because if they trust it then I think you can as well and there's some actual benefits to API keys in terms of security uh jot tokens they're you know they're Json packets that's what the J stands for they're Json data I think base64 encoded and stored in a signed token that means anybody like you can take any jot in the world actually um if you go to job.io you can take any jot and you will see that you can see the contents of that jot token even if you don't have the secret that what it was signed with it's kind of public information that might contain information about your claim structure it might give people some better understanding of how your system works and potentially use that to form some kind of attack so there's some value in being in being just an opaque string like an Opa a API key is just a guide typically or some kind of like high-end trippy random string and that's valuable because only you know what it means only you the system owner when you go and match that to our record on your database have any sense of what is actually in inside of this key the other thing that's important is the API keys can be self-serve managed so that means me as a developer I can go to my stripe dashboard I can get my key that's nice but I've got to also roll it so if something happens with that key I can go in and say oops I leaked this I accidentally recorded it in a video like like this one I'm making now you know we've all done that when we're making demo videos I can go in and I can roll that key instantly it's revoked it's gone that's harder to do with Yachts jots have an expiry and you know unless you pick a very short expiry which then actually makes the development workflow even more complicated because um you know the refresh token flow comes into play and you have to think about that whole dance um the idea that I can revoke a key it's much more powerful whereas with a job token to revoke that that's a very involved process often what will happen is you'd have to revoke the whole client the whole the whole Space of people using those jot tokens which would impact all of your tenants so that's pretty sub-optimal as well so there's actually a number of feathers in the cap of API keys and why they're arguably in some ways more secure we feel super comfortable using them we recommend them and actually there's been advances recently and we took part in this with the with the the GitHub scanning program so the form of our keys which I'll talk about as we go into the best practices of API Keys hours are formatted such that GitHub can recognize them and if they get checked into Source control by accident they notify us and then we can notify you as a customer anyone can participate in that you can write your own secret scanning code um uh you have to to do it in sort of the way we've done it you have to go through an onboarding flow and a program with GitHub you know if you zoopla we take care of all that for you but you can absolutely do that yourself and it's one of the best practices we'll talk about in a minute when coming to decide about API Keys there's one major decision that you need to make in terms of like how are you going to design this and it's whether you want the keys to be what we call retrievable or irretrievable and I I'm assuming as an engineer you've probably seen both kinds of these an example of an irretrievable key is I go to some dashboard some developer portal and I um I create my key my key is shown to me I can copy it out of that product and put it into ideally into some kind of Vault what some people do is put it in notepad or text edit which is bad and that's the last time I'm going to see that key in that dashboard like immediately once I go away from that screen that key is irretrievable and the reason they've done that is because they don't want to store the key even so if their database is compromised then someone will only be looking at typically at sort of um hashes of the stored key and those hashes are one way so they can't be reversed so no one else can get the key out of the system businesses that do this are stripe Amazon AWS people who tend to be a little bit more on the secure sensitive side of the domain you know payments Amazon AWS is your whole Cloud infrastructure they tend to use irretrievable Keys there's lots of examples that don't you know twilio ER table rapid API they use retrievable keys and what that means is I can go back at any time and get my API key without having to revoke the old one I can just go in and see what it looks like copied away I think there's actually a bit more to this trade-off than just like one is secure and one isn't um and that is one thing to bear in mind is that if you have irretrievable keys people have to copy them somewhere immediately if your customers are disciplined well-behaved audience that are going to do the right thing and put that in a secure encrypted Vault then this is good practice if they might not be then it's actually pretty risky because what that customer is probably going to do is stick it in notepad or text edit on their machine which is a very unsafe place for a key to be but they might even email it to themselves you know this happens really um in fact I'll put my hand up I've been guilty of this you know when I've when I've deemed the API not to be crazy sensitive to me I absolutely do this kind of thing in those cases I actually think retrievable is a better option because now I don't make that mistake of pasting that key I know I can go back to the dashboard and get it this is your decision to make if you feel you fall on the fall on the higher bar of the the the security spectrum and your customers are going to be sensible and and going to put that in a vault then go with irretrievable if you think the the use case is like a little bit less sensitive or your users are you know perhaps like hobbyists at weekends who don't have access to a vault then you might want to go with retrievable because actually that will be more secure because they're not going to put the key in somewhere that they shouldn't or email it to themselves or send it over Facebook Messenger or something so that's one of the biggest decisions to make when it comes to API Keys that's your call I think I've explained the trade-offs as we see them um and zuplo our behavior is excuse me our default behavior is retrievable um uh for the reasons we've outlined so you would get that for free if you use upload as well let's talk about um best practices so the first best practice we've kind of covered when we've talked about retrievable or retrievable and make your decision and do that but then the second is that you need to support a rolling transition period so all API Keys should be revocable actually that's almost an implicit best practice if you're going to go with API Keys you have to design your system such that a key can be revoked to not have that would be just glowingly insecure and unacceptable for any system what you do need to support is you need to support when the key is revoked that the customer can set some kind of rolling transition period and what that means is let's imagine they go into your dashboard and they decide they want to roll the key what's going to happen is if you instantly revoke that key all of the systems that have that key logged into them or entered into them are going to fail and so you that's probably bad unless this is like Sev zero super critical situation um and what most people do like you know I'll show example of stripe here where they when you roll an API key you get to set what is the expiration date of the ALT key you get to say you know this should expire in like five minutes that probably gives you time to take the new key cut and paste it and stick it into all the systems that have a dependency on it so the key things to there is that it's you support revocation and you support a rolling transition period another thing that's important is that you show the creation date of the key in your UI this is how people know has this key been rolled when was it last updated a lot of people want to roll their keys on some sort of cyclic basis you know like I might roll it every six months just to just to feel good and healthy about how often I'm sort of pruning the tree and keeping things fresh and it's hard to do that if you don't show the date the key was created if like zooplow at you which I'm showing an example of here on screen we allow people to create multiple keys so they can manage their own revocation you can create a new key and then delete the old one on your own time it's really important to show when it was created so people can disambiguate between the keys The Nest sorry the next best practice number four here is check some validation um so we do this and this was actually guidance that came out of github's best practices on API Keys uh for their GitHub secret scanning program that I'll talk about in a second and what that means is that at the end of the key we actually put a checksum and that allows us to to when uh secret scanning is taking place we can verify that this key looks like it was sort of at least somewhat legitimately created it's not proof like anyone could create a checksum but it allows us to to to do sort of a very quick false positive is this just just a random string here that we're playing with is it just some random guide or someone at least gone to the lens of creating a checksum out of this this is also useful on on uh GitHub secret scanning as I'll talk about in a second you could go further with this you could actually make that check some assigned hash similar to a jot token um we don't do this today on on zupo we we talk about it where I think if we get a customer interested in it we'd add it as an option potentially um and with the signed hash that means that the the signing Authority can have absolute confidence when presented with a key that it was generated by their system so this allows you know in the case of an API it allows you to reject invalid Keys now you still have to check for revocation but it allows you to reject like just completely invalid keys without doing a database check without calling any sub Services you can just check that you can just create a copy of that signed hash see if it matches up and if it doesn't reject it that's how jot tokens work so it's highly performant it's a common approach and it would be absolutely fine to do so um it adds a little bit of you know a little bit more work a little bit more complexity so we don't do it for that reason at this time but but it does it can provide additional protection and potentially additional performance if you get a lot of fake API Keys into your service most people don't deal with that a ton so it's you know it might be premature optimization to do that until you see that being an issue best practice number five support secret scanning so API Keys you know I think are defensively a secure solution but we all make mistakes and one of the downsides of these Keys is they can sometimes if you don't put them in your environment file they should go into sort of environment variables and not into source code they could get checked into into a source control repo fortunately GitHub has a secret scanning program you can run this on your own repos without getting their permission uh just search for this so I'll provide some links in the in the video in the video comments um you can also enter the GitHub secret scanning program and what that does is you give them information about the structure of your key like essentially a regular expression that matches your key and that will go into um that will go into GitHub and they will scan source code and look for matches and if they find a match they'll notify you and say hey we found this thing that looks like a key in a public repo it will also run on private retail repos that people have opted into and we do this at zuplo so if someone's using a zooplo API key and it gets checked into a source a public Source repo or a private one that's opted in we will get notified we in turn notify that customer we automatically revoke the key and in depending on their settings we'll notify their customer as well uh really important thing to do to make sure that API keys are secure and we really you know I think what we're trying to do here with these best practices is sort of these are what we think are becoming the standards of API keys or what should become the standard so it's exciting to try and start that conversation but uh this is a must do for for your API key and that's why you want to think about the format for us we start with zpka underscore and then if we get a key reported To Us by GitHub before we do anything about it we quickly check the checksum to make sure it's real and not just an unlucky random string that looked like a zooplo key um and if it is then we'll go and search our database and find the key and find the customer and do the notification um best practice number six so on an API you're going to be receiving the API key typically as a header um uh you know I think people are pretty open that can be an authorization header with a Borough and the API key it can be an api-key header that's up to you I don't you know I think that's pretty flexible at this point um but what that means is your API the first thing it's probably going to do is do a check and typically that your keys are stored in some kind of Secure Storage so then hopefully encrypted or as a one-way hash if you're using irretrievable and that means a database call of some kind you probably don't want to do that on every single read it's going to add latency to your API it's going to add load to your database so you probably want some read through caching strategy on the API keycheck so that means every time a key comes in you go and check it you pull out the metadata associated with that and then you might want to store that in a mem an in-memory cache in process with your API if that's how you're building it with zuplo we have a much more complex system because we're a distributed Gateway we run at the edge at you know over 200 data centers around the world and highly scalable on a serverless architecture so what we do is we actually store these um in a in a in a secure cache at the data center level that means um we don't that we can do the API key checks extremely high performance and um are reading quite local to where that that machine is even in a distributed manner now we have a Timeout on those so it means every 60 seconds or so we have to go back to the database and get the key we've actually designed that to be so high performance we do that asynchronously um so we go and make the call we almost never block an API on an actual database call once the system's warmed up so you can look at strategies like this to like reduce the latency and reduce the impact of API key checks best practice number seven hide the keys until needed you know we live in an age where everybody has a high quality high resolution camera in their pocket and uh people are recording things all the time and so if you if someone goes to your dashboard and the API key is displayed that could easily be captured should buy somebody with a quick photograph or someone recording a video and they now have your API key so you really want to be pretty sensitive about how you show that and I give an example here of Super Bass who they have a secret that they hide until you press the reveal button um I think you can go even better what we do in zuplo is we have a reveal button but we also have a copy button so if all you want to do is copy the key to the clipboard you don't even need to reveal it just click the copy button we'll put it on your clipboard and then you can paste it in and hopefully the thing you're pasting it into has a password mask on it and your your key your API key was never seen on screen by anybody so that's what we think is like best practice actually there is like hide the keys ideally provide a copy button so they never need to be revealed but obviously it's useful to reveal it to people in some cases in case it needs to be read out or or inspected visually best practice number eight this is a real attention to detail thing um when formatting your API Keys it might be tempting to put dot separators in or Dash separators that's what you get with a guide for example we'd recommend against that change it change it to the underscore so that it's in snake case why because if someone needs to cut and paste that key you can just double click anywhere on the key and it will select the whole thing if you break that key up with a period or with dashes you can only select that section and it makes copying and pasting the key a pain in the butt so let's let's be thoughtful let's pay it forward and be nice to Future developers who just want to cut and paste this thing into their clipboard and so let's use a snake case as the format for API keys and then finally best practice number nine consider labeling your keys uh stripe do this we do this at zuplo and that means give your keys some kind of name at the beginning this fix so in the case of um uh stripe they have SK live which tells you this is their secret key so it's very sensitive and the live tells you that it's a live key it's real it's not a test key if you uh they also have keys called PK test and PK is a publishable key that means this is shareable It's not that sensitive um and the test obviously tells you it's a test key not a live key so why is that useful that's useful for a couple of reasons uh one is someone in support So working on your support team or if you're helping someone debug an issue you can very easily ask them can you tell me how that key starts does it start with SK live or SK test and if they if they've accidentally put a test key into production then you know that's probably where they're going wrong if they've actually put the SK key in something that expected the PK key again it can really really help with debugging also if the key is discovered in GitHub secret scanning you instantly know well if it's the publisher key that's fine you know with the publishable key is publishable by design it can go on a website that's kind of what it's designed to do if it's the secret key then oh that's more like Panic stations someone has published this sort of keys to the house here and we should really help them so it doesn't cost anything to do this it's a really nice neat little practice and uh we do it at zuplo and we found it we found it useful um so essentially that is uh what we think are nine best practices if you're gonna design your own API key implementation so I'm going to walk you through sort of a canonical check now of what that might look like if you actually go and Implement that so the request comes in to your server the incoming request you check first of all is the API key present and is it in the correct format if it is the next thing to check is is the checks unvalid that's very cheap so we know if we're just getting garbage API Keys we can just reject that with a 401 right away very fast microseconds of compute time the next stage is to check whether we have assuming the checksum was valid is to check whether we have a valid key in the cache if we do and we're happy you know that we set a deliberate cache expiry on this then uh we can just let the request proceed so actually at this point move on to the business logic or whatever and you can be happy that the API key is valid and use the metadata associated with that key but if there is no key in the cache then it's time to go and call your API key store so in that case you'll be calling the database where your keys are stored hopefully either encrypted if it's retrievable so that'll be like symmetric or asymmetric encryption that allows me to reverse the encryption and get the original key value back or if you're irretrievable then that really should be a one-way Hash a fast one-way hash like shar256 you don't really want to do B Crypt here because it'll be too slow um and we'll add latency to your API um once you call that API key store if you find the the key to be valid then you should store it in the cache if you find the key to be invalid you probably want to store it in the cache as a as a false key as like something to be rejected because if a person keeps hitting you with that key you want to be able to just 401 them very fast and sort of say no thank you we're good also you probably still want to do rate limiting on our per user basis even if the key is invalid um even if it's not a recognized user you know your rate limits are going to recognize that they're not they're not an actual user and going to kick in um one thing about the the cash it does it does obviously have a bit of a trade-off though actually as you think about it as we talk about the cash one downside is if you cache these keys in memory it does slow down revocation um that's why we'd recommend not putting them in cash for too long so that if a key needs to be revoked the key is only live like maybe 60 seconds or something like that um you can have strategies to proactively go and flush the cash or flush that key from the cache it just gets a little bit more complicated so you know we have some techniques like that um uh zuplo but if you're building this yourself you might not want to go quite that far and it's probably fine to have a key live for an additional 60 seconds you know if if it's totally not fine then you're probably just gonna have to like turn off the cache you need a kill switch on it um if that's something that you couldn't live with but yeah that's the flow through a kind of canonical API key implementation and some of the best practices that we've learned in designing our own API key solution and looking at how other businesses what we think are the best API first businesses have implemented it so good luck have fun um uh if you do decide you'd like up with this and of course zuplo can add this to your API pretty easily and you might want to give us a try but otherwise thanks very much

Channel: zuplo

Views: 9,161

Keywords:

Id: ooyOmiczY1g

Length: 25min 56sec (1556 seconds)

Published: Tue Dec 13 2022