Certificates gone bad | The Backend Engineering Show

in this episode of the backend engineering show i'd like to talk about certificates you know those things that show up when you click on the padlock uh icon on your browser when you visit a secure website and mainly what i'd like to talk about is why do they exist and the problems that they introduce as a result specifically to the revocation of a certificate when it goes bad you know certificates can go bad because the private key can get leaked or the certificate authority itself becomes shady or doesn't follow the rules anymore and we want to mark all this certificate as bad so we we kind of fell into the pet of this problem of revocation and we try to solve it with many many solutions but most of them are are really hacky and and really just puts a lot of stress in both the front and engineering and the back engineering community so i'd like to talk about all that right so i'll have some time stamp for you guys to jump into effectively i'll be mentioning a certificate uh revocation list i'll i'll mention uh authority or vocation list i'll mention ocsp protocol online certificate status protocol and then i'll mention the online certificate status protocol stapling and then i'll finally mention what is really the best solution when it comes to protecting and having a fully secure certificate and what does that mean how about we jump into it welcome to the backend engineering show with your host jose nelson this is the long-form content show that is also a podcast that you can listen to on apple podcasts on spotify google play all that jazz go to hussein also dot com slash podcast to listen and make sure to read the show thank you so much and uh there'll be some time stamp guides uh both in the show notes and on youtube if you're watching on youtube you can jump into the interesting part of the video why do we need certificates the main reason to keep the to keep from going into kind of the weeds or the details is the problem with connection management if i want to connect from a client to a server we use the ip address and the port to connect to an application right however these these two pair is not enough to establish that we are absolutely sure that we're connected to this particular server because someone in the middle could be looking out at our packets right and this is regardless whether there are encryption or not because if if there is a request which transform to a bunch of tcp segments in case of tcb which goes into a bunch of iep packets all this packets go through your router which goes through your router's isp which goes through many many many other routers in the internet so they pass through many many hops until it reaches the final destination how can you make sure that nobody tampered with your data right how can you make sure that when a response came back no one actually changed it right you can't and the reason is this lack of identity we don't we don't know that this response that we have received is actually from the original server or not from someone who did a man in the middle attack using uh arp poisoning in a shady starbucks wi-fi or a hotel guest right you can't know that and this also includes encryption even even if you do encrypt the content the request the first request to establish the encryption using tls the tls hello is unencrypted most of the time there are solutions to do the an encrypted client hello that might solve some of this but still not 100 but yeah the request to encrypt goes through the attacker right so the attacker can just say okay i'll just response back to you with an with an encryption response as effectively create two channels two encryption channels right so i'm gonna terminate your encryption and then i'll serve you back with my own diffie-hellman parameters effectively terminating the encryption and looking at everything so if you send me an encrypted request i'll decrypt it and then send it back encrypted to the backend server and the server doesn't know that the client is the one connected right it's just it just knows that uh someone is requesting this information it doesn't know the identities this this goes back to both the client and the server we don't know the identities so what is the best way to introduce identities effectively authentication i want to know who am i to connect it to what is something that the server or the client only have and no one else have private key public key infrastructure so if you generate a pair dedicated for you on the server a private public key you can encrypt with the private key and only the public key can decrypt it and vice versa as well you can sign something with a private key and you can verify it with the public key the private key is kept secure the public key can be public obviously and this is the trick when you use private public key infrastructure the problem is kind of solved now let's use the private key as a way to establish the communication so no hey this this is a good thing let's let's do the same thing again so when i send you a message right i have your public i have the service public key here and then i'll send a message so when the server responds before it responds it adds a padding digital signature to the message saying hey by the way i'm gonna encrypt this thing that i'm about to send you with my own private key nobody else have it but my public key will be able to decrypt it right so if the attacker receives this is oh it cannot do anything about it it can't try to change the data but now the data doesn't match the signature it will and it cannot create its own signature because if it creates a new signature it needs a private key which it doesn't have so now it passes back this way it passes back all the way to the client and the client will take the public key and then verifies that the data is on tamper but there's a big problem with that how does the client know the public key of the server you probably already asked this question while you're listening to this the problem is they can't they have to be delivered right so how the moment you deliver the public key in the same message the attacker can just grab it and change it right because this the client doesn't know that this public key belongs to the server we have the same problem again we're in the same loop how do i know that this public key belongs to the server there is no link right so then then this becomes a problem so how do i deliver the public key that is me to the client so that they know this is actually me meet certificate so the the basic gist of certificate is the the server generates this private and public key they take the public key put it in the certificate and then signs it with a lot of other metadata like the domain and stuff like that and then signs this public key with its own private key and creates a signature right and then they ship this information right but you might say i'm saying that that's just dump that's the same exact thing that you just said because the certificate now hey the attacker can do the exact same thing right this is the certificate itself that i can take it oh it's a certificate the public key is signed boohoo i'm going to generate a new certificate i'm going to put my own private key here and i'm gonna self-sign it just like the server did with the private key and i'm gonna ship it to the client the client doesn't know that this certificate is from the server or from the attacker so what do we do meet certificate authorities we need someone third party to trust unfortunately we need a third party certificate authorities are are entities that are trusted and so the server negotiates with the certificate of authority and says okay here here's my certificate with the public key in it hey i want you to vary i want you to to kind of what is this put your stamp on it so the certificate authority does exactly the same thing id has its own private key in the public key and it signs the certificate right from the server which has the server public key with its own private key and now you need a way to trust the certificate authority now there is something higher than the certificate authority called the root certificate and the root certificate does exactly the same thing with the certified authority until that's it the root certificate is globally trusted and it's self-signed nobody trusts nobody signs the road certificates it signs itself but you might say how do i trust this self-signed certificate root certificate go to your computer you'll see a list of all trusted road certificates can you trust them no you cannot no you cannot because if you ship a laptop from china it will be shipped with some shady road certificate how do you know what's good or not you're at the mercy of whatever whomever these people decided to decide that these root certificates are trusted you cannot trust anything but let's say i'm being facetious is that all facetious facetious yeah i think that's the right word but you get my point right let's assume we trust whoever the windows who installed the operating system or linux so install the operating system with this list of road certificate and we say okay we trust these guys okay and now when when when this certificate travels which is signed it has a chain of trust right it will travel and the attacker cannot do anything about it because even if it did create its own public key and private key it needs a certificate authority to sign it which nobody will sign because they know you're an attacker they know you're not facebook.com they know you're not google.com right there's a way to kind of how you might say how do i generate this certificate there's a dns way the dns way to generate certificate and there's other ways as well but yeah so this this looks fine this looks great right right what was the problem this all of a sudden now the client and server trust each other and the client can do something like that right the client can also provide a certificate called a client certificate so the server truster client right if you want a mutual tls mtls that's what they call it if you want both identity if you want to trust this is this is good for microservices specifically in the cloud you you don't want just to trust the server you want to also trust the client effectively this only these clients can connect to me as a server all right we talked about why we need certificates why they're good right they solve a problem that we genuinely have kinda because certificates just like uh yogurt they go bad and what does that does does that mean i'm saying do you mean the certificate expires yeah they have an expiry date like why don't you put an unlimited certificate would not expire well you cannot do that because if the private key are you guaranteed that the private key is going to last forever what if the server got hacked and the private key got leaked oh that's a problem right why is the problem i'm saying it's a problem because if the private key is leak and the attacker got access to the private key they can effectively keep the certificate they don't need to change the certificate yeah keep this attack as is they have the private key all this message that is included with the certificate they can change it all over together they can put their own message they can put put their own different parameters they can put anything and then sign it with the server private queue because they have it and this client will say oh let me verify this message the messages sure the private the message says hey this i must be talking to the server because no one else have the private kicks at the server bad so private key leaking is a problem so we need to put an expiry date so there we put expiry date it used to be three years we shorten it to at least one year uh let's encrypt put it i believe two three uh three months i believe by default shorter the better what if your one year or three year certificate didn't expire but your private key got leaked before it expired all of a sudden the client cannot refuse this certificate they look good they didn't expire but they are bad right you need to revoke these certificates and people found a problem with that because how do you tell that this certificate is now revoked right even the server itself so the server itself can just update it but most servers don't know that their their private key is actually lead so so how does the client now trust because that's the client is the victim in this case so the client needs a way to know that the certificate is revoked the first solution that people came under i think just shot of the dark let's build a list let's build the list of and all the certificate revocation list we're going to call a c lr so for the refrigeration crl it's a list of all the revoked certificates and it turns out to be a huge list because it turns out a lot of certificates are being revoked and not just because private keys are being leaked and i don't know if you heard about this attack that was called this this flaw in openssl which is this this library that helps in this crypto stuff right the cryptographic uh algorithm right and then generate these tls libraries and functions uh it had a problem called heartbleed where attackers were able to read the memory of the server and what's in the memory of the server pressures beautiful things including the private key and not only just private keys you want to revoke their certificates because of private keys or link but certificate authorities are run by people and people are inherently some people are bad some people are malicious so certificate authorities go back so we need to a way to not trust certificates generated by bad certificate authorities right or certificate authorities are run by human human make mistakes so oh i accidentally created a certificate for someone to that claim to be google.com accidentally well we know google.com or not this shady person living in the middle of nowhere i need to revoke that so mistakes happen so you need to revoke it so people created this idea of certificate revocation list it's a huge list so the client need to download this list and every tls session right i didn't mention that but certificates are served with the tns server hello the server responds back with that so every tia session i have to check this huge list so now my my my stuff is slow right and people now uh needed not only and they found that this this is a growing huge and if a certificate authority turns out to be bad all their certificate thousands and thousands and millions of certificates now are also bad what do you do you generate these fine revocation list entries for every certificate that this shady certificate authority like i was was it called superfish i believe the certificate authority did not are so they came up with another list called the authority certificate revocation list so another list just for the certificate authorities we know it's the solution didn't work didn't scale okay because of those problems like i don't want to search this huge list and i don't want to consume bandwidth and then how do i update my list it's like this this list keeps growing by by the minute so i have to download the client have to download this list and check okay so how do we solve this problem a new solution was proposed and people said okay let's put this list on the server let's let the client ping ask the um centralized server if this certificate that i just requested from a server a third-party server might may i tell you let's ask this middle server third party if this thing is revoked so now this is less bandwidth but it's more efficient bandwidth efficient and this is called the online certificate status protocol so there's a server that manages the state right off and you you the client will ask hey um i've just visited google.com and they gave me uh this certificate can you check if it's revoked all right they create an ocsp request and they asked this third party hey can you check this report hey i just visited reddit.com can you check if this certificate is referred hey i just visited facebook.com is the certificate of work hey i just visited um hussein also dot com is this visit is this certificate revoked uh you get the idea what do you notice now i just visited the nastygrammas.com can you check the certificate i don't i i try to come up with a name can you check if this certificate is as it is would have worked what do you what do you notice that your entire history of the of of what website you visited are in this third party kind of defeats the purpose what is the privacy you don't have a privacy anymore this third party have a log of everything effectively that's not good so the ocsp the online certificate status protocol people didn't like it because of this privacy so the so people invented a new thing called ocsp steep link what is that so instead of the client doing the instead of the client doing the ocsp request let the server do the ocsp request this way the ocsp request let the server do it and let let it attach the ocsp the status of the revocation of its certificate as a proof that it wasn't it wasn't revoked in the tls response and there is a response back to us so this way i know that oh not only i got a certificate i also got a proof that was stapled it is stable literally staple it with a proof that it has not been revoked and this looks fine this is nice but now how does the server know when to do this does it do does it do it synchronously if it does it synchronously that means with the tls hello the server will pause with wait let me do let me let me before i give you the certificate you asked for the ocsp stapling okay let me check with this third party i'll give either small okay it wasn't revoked attach it and send it back that slows down things definitely right because now the back end has to do this extra work right that kind of sucks the backing has to do work it was the front and now the back and has to do it that slows down the connection and that's definitely going to slow down connection establishment and that's not good so people now do it asynchronously right so when the client asks for an ocsp stapling tls extension because that's what it is it will asynchronously always do it on the back end and have it ready right and and and when the when a customer on a client asks for it it will attach it to the tls server hello and then ship it back to zest to the client right and this seems to be solving a problem yeah so we don't have this delay anymore but still you're asking the server first of all the server has to do more work that's cpu that's memory that's bandwidth and more all you're asking the server to be connected to the internet what i don't want to serve my server why do we need to connect to the internet my server is a pro this is a private thing the whole my backing is hidden behind a firewall i don't want to expose it to the internet you might say i said why do you what are you talking about what internet well what is this ocsp stabling thing it's somewhere on the third party in the internet right you need to acquire this list right you have to open the port you see the complexity where we're going why because of the certificate so now now you might say hosting i'm gonna i'm gonna run my own uh ocsp stapling server internally right you can and people do that a lot of a lot of you guys actually are cloud experts right you might have running your own certificate servers and you don't really need all that crap right you're all internal the whole thing is internal for microservices you really it's not really public but when it's public then it's then you get into this situation where the public internet needs to know about this thing that right there are vocationalists but yeah it gets just more complicated as you as you see so what's really the solution huh saying well i think everyone agrees in the browser community that you need shorter certificates man i mean cloudflare is is really the best when it comes to this thing cloudflare generates like two weeks certificates that's it recycle your private key all the time because these things the private key itself changes keep it changing sure because if you change your private key not only you you don't run into this revocation problem as often because you don't really need to check for notifications like who's gonna get hacked in two weeks right i'm not saying it doesn't happen but the chances that it can happen very unlikely right and plus you don't get the problem with a forward what is it called uh you don't get the problem forward secrecy right when it comes to certain sessions i know the tls 1.3 doesn't have forward secrecy because it generates a femoral symmetric keys right now probably doesn't matter really because uh the the tls 1.3 has the it's fully uh supports forward secrecy so uh full forward security i believe it's called so even if you if you change your private key if you don't change your private key it generates ephemeral uh symmetric keys all the time so unlike rsa right rsa the problem with ours is uses the private key as um a vehicle to generate this symmetric keys which causes obviously problems because once the private key is leaked not only you have access to the future but also the past conversation if you ever recorded them all right guys uh that that's all for me really just keep your certificates short that's encrypt yeah how player thinks that the shorter the better and then be pragmatic when it comes to this revocation things because uh if you enable this i mean iis have this this all these options you can disable or enable ocsp status protocol stapling you can have it enabled or disabled it's up to you in genex hi proxy all these have this option to enable or disable ocsp stapling based on that but now i think now that you know the fundamentals and the reasoning behind all of this crap you you get to make your own choice at least you understand what's going on at least thank you so much for listening slash watching i'm gonna see on the next one you guys stay awesome

Channel: Hussein Nasser

Views: 5,912

Keywords: hussein nasser, backend engineering, certificate, certificate authority, OCSP, OCSP stapling, CRL

Id: ZlTjwVYnbLw

Length: 27min 21sec (1641 seconds)

Published: Thu Oct 07 2021