GitHub SSH key is Leaked - How bad is this?

SSH private RSA key has been accidentally pushed to a public repo might as well consider it leaked because the moment anything exposed even in a tiny fraction publicly it's as if it's clicked because it could have been sniffed the Wayback machine might have read it and cached it you know cached that page someone might be reading it so when I read that news I asked by myself like what does that mean what does it mean for the private key of the SSH [Music] e session that belongs to GitHub to be leaked so I tried to unpack that and try to understand this a little bit more and that's the that's the goal of this episode just to understand what does that really mean what's the implication how does SSH really work when it comes to host keys and public key cryptography so I'll try to unpack this in this episode let's jump into it so this comes from github.com let's read a little bit and then dive into the discussion here we updated our RSA SSH host at approximately 5 AM UTC on bra on March 24th out of abundance of caution we replaced our RSA SSH host key used to secure get operations on for github.com we did this to protect our users from any chance of an adversary impersonation of GitHub and if eavesdropping on their git operations over SSH this key does not Grant access to GitHub infrastructure or customer data it's very important this change only impacts git operations over SSH using RSA web traffic through which is https is not affected you know that this will happen if you give this particular message if you connected to GitHub and try to push something you know push the commit or even Kelowna repo you might not get this message for those listening on the podcast I'm going to read this out loud warning remote host identification has changed it is possible that someone is doing something nasty someone could be eavesdropping on you right now man in the middle attack it is also possible that the host key has been changed so GitHub in this blog walks you through how to remove your old key the old public key that belongs to GitHub and then replaced with their new public key because they don't share the Privacy they share only the public key that matches their private key so let's go ahead and actually discuss what this what all this really means you know athesh works very similar to TLS when it comes to negotiation the first step is they exchange the algorithms for the key exchange okay it says all right I support this version of SSH I support these algorithms and here's my key parameters let's go ahead and exchange the key right and they use the element to do that so that both parties the SSH server and you at the SSH client um can derive this in symmetry key encryption and then you start encrypting with that key right so that nobody can seize that but just like TLS there is the authentication part how do I trust that the server I'm connecting to is the server that I'm connected to and it's not an imposter right in TLS we have the whole thing with certificates on certificate Authority with the server will sign will create a certificate a public key and then use a third-party certificate authority to sign the certificate and create a chain and this chain is returned to the client you know this full chain from the leaf certificate to the certificate authority to the root right and the client side will have either root certificate that is trusted in their operating system search store or the certificate authorities list of this they're public key so if once the client receives that certificate chain it will do a hierarchical check to check if the parent either one of these certificate Authority is in the store or the root is in the store if it's in the store I trust it if it's not I don't trust it and you get that error right in SSH it doesn't work like that you know there is no search store what there is is there is a file called known hosts right and SSH and you can go to it so versus telda tilde.ssh Slash known host and the list of the public key the raw public keys of all the servers are there you can go and manually add a public key that you trust so you can exchange a public key of the server you know off Channel and save it manually that's the best approach to stay absolutely safe right so yeah you can go to like get help and download that key and then literally just add it manually yourself to the client machine and this way you now know and trust that server no so the other way is when you connect first time to a new server it will warn you it says hey I know I didn't find the server public key in any of my known host list do you want to add it if you say yes it will just add manually so you you become it's up to you to trust that or not right so this takes us to the SSH handshake when you first connect through SSH whether whether you're just directly shelling right or you're cloning a GitHub repo for example through uh through the SSH endpoint right what happens is you're actually connecting to the TCP port 22 on github.com okay after the connection you'll establish a SSH session which it says okay here's the clients here's my server parameter and then the server send you something called The Host key and the host key is literally a public key that the server has and this doesn't change right this host key doesn't rarely change and that is the identification of the server right so what what happens is once you take that host key which is literally just a hex of that value you check the client checks the non-host and say are we are we in this list yes I trust you immediately and then you go through the login credential right now you're prompted for a password or if you're using public key authentication you can exchange your public key to the server and the server will check uh is this client in my authorized Keys list then you will log in without a password right that's the password listing so that's what's happened here so now what happened with GitHub GitHub the private key of GitHub always has a matching public key and that public key shared to I'm not exaggerating you know because I don't know how long this public key was alive but it's not exaggerating if I say hundreds of millions devices you know it's everywhere it's like every time you use an a client a necessary client to clone a repo you connect to GitHub and that known host will use that old public key to store it so and I omitted something here that I recently learned as I researched this topic that RSA is just one algorithm right to get the host key and that's also exchanged during this negotiation with the SSH configuration so you can say that client hey I prefer elliptic curve difficultman key guys if you have it hey server if you have elliptic curve give me that key right but if you have only RSA I'll take that right so that is even the key can be different so what what got leaked in case of GitHub is actually the RSA key okay so still that thing is everywhere right so there are millions and millions and millions of devices whose known host have the old public key of github.com you might say what's the problem with that well that's that's that's that's the question I asked myself how bad is this because they don't tell you this stuff right if someone's if someone took that private key right like with the way machine the way Mac machine maybe has archived it someone sniffed it and got that because there was like a brief period of time where they pushed their private key accidentally if someone got it then they can create an SSH server and they can put their private key on the RSA SSH key as a host key right so they can place it there right it's literally under SSH config right you can put it as an index as a server you can put that private key and then because you have a private key you also you get the public key right because there is always the private key and the public key is known the private key is now known the public key was always known you put it there and then you are pretending to be GitHub but there is one problem you now have to get the second one the first step is is done your private your machine your your attacker SSH server is now done right it's now GitHub it's it's an identical copy of GitHub the second problem is like you have to somehow trick clients to connect to you to your IPS because you're not going to have github.com it's impossible right so one trick is to do a DNS poisoning while while it's not trivial it can be done well if someone took the laptop right the development they went to Starbucks and you somehow let's say you own that Starbucks router right and then you injected a specific entry to hijack DNS queries for github.com and instead of returning of instead of just bypassing the DNS query to to the recursor to the Upstream recursors what you do is just you reply back with your attacker SSH copy of github.com the IP address if you do that then GitHub now from this developer machine will point back to your server okay now what do you do next you're gonna do a GitHub clone for example right you're cloning a report or you're doing something you're trying to connect your journey to push something to GitHub the fact of git add or get pull or get push will establish a new TCP connection to github.com but not the actual original server it will connect to your attacker server nice the next step is we're going to exchange the client SSH keys right the server which is your attacker server will reply back by with the host key of the old public key of github.com right your client machine has never connected to GitHub after that before that point so you still have the old public key that belong that matches the private key which got leaked GitHub didn't get a chance to update you yet because you you are one of the millions that you didn't get updated right and because the attacker server responded back to you with this public key your SSH client innocently will look up your known host and we'll find that the public key is there and it will completely trust you the next thing is what is the Authentication is the authorization I guess now you're authenticated now you trust the server which is bad the next step is if you put passwords then enter you type in the password you just uh leaked your password to the attacker server because now that password will be encrypted with the symmetric key that has been negotiated between you and the attacker server and then that password is now sent to the server and the server will of course decrypt it and store it and the server can do so many things here the server can act like an actual git server and forces you to push they can causes you they just push the stuff it will say push your code and if your code is private that's a really disaster because you just now pushed private source code that is not supposed to be about look to the attacker server so this is really bad as far as I uh that that's how I analyzed it right I think this is pretty bad but if you got this message okay which says Hey ramoto's identification has changed now let's explain what happened here now nobody's doing a man in the middle attack on you you just connected to the GitHub the next time right what the SSH client is doing here again this warning remote host that this video has changed this is the SSH client being cautious that hey I connected to This Server right the GitHub pointed you to to the same IP address the server responded back with a different public key then the one you have in non-host so it looked up the github.com as a website in your known hosts but I found that the public key that is returned is different from the public key that stored in our house so this sh will always throw this message like hey something changed either someone is man in the middle in you right and because that's the normal path right if if if if someone didn't if the private key didn't leak and someone tries to do this DNS poisoning cache they will never have the private key to begin with right so they will have to use a different problem key and the moment they use different public key for the same host they will get this error and that's where SSH client trying to be cautious right to give you that message but they also tell you that hey by the way it might be just that the host key has actually changed right and here the good thing about this message and I don't know how to how this happens how to force the verification to fail you know I couldn't find that out yet you know because every time I connect to a new a different host I get as much do you want to add it no I don't want to add it it seems like GitHub is doing is having an option to force the verification to fail if you ever get a different public key which is a good thing I'd like to have something similar there that's probably I'll probably look it up later but probably there is an option probably so yeah so what happened here is ironically is the client thought that get someone is doing a man in the middle but it's not really right what happened is the server host key has changed and now you need to update it now the good thing about this is they tell you don't just say yes go and actually do it manually and the steps here is the first step is SSH Dash Keygen Dash R which means remove anything that says github.com remove all the public keys that are mine on host that's the first thing or it says go ahead and do it manually if you want just edit it and then go ahead and add this entry that's the new public Key by the way we're looking at the largest public key here and this is by the way it's a base64 because the public key is as well is like it's a binary right and you can't really paste binary it's not really text friendly so you change it to a base64 so it becomes like a a B3 NZ one y c two AAA b b z okay I'll stop and then they'll say okay you can also automatically update the RS RS assh key here blah blah blah good there uh and that's just another way right what is affected GitHub actions and approach if everything that uses automation right if your default preference for SSH Keys is elliptic curve signature right the other one you're fine because that didn't get leaked right if you're using https right that's another path right too like you can clone through https right this is just like more a lot of people prefer this because it's consistent with https like there's a search store there's sometimes you require I don't know I found https to be like at least it was like 2016 when I used it was always like shaky very shaky to use but maybe it's like more stable right effectively if you're using ecdsa you're fine if you're using ed25511 and you fine uh does that mean like I want to profess this by saying that using ecdsa doesn't mean you're immune right it's like someone will say oh is it stock more secure of course it's more secure but this is not about security and certain in a sense of breaking Odyssey nobody broke out of say it's like if you took the AC DSA the elliptic curve diffie-hellman signature algorithm and you publish the private key that's pointless right the security of that algorithm is pointless you just publish the private key for God's sake so it doesn't matter which algorithm you use so this could have happened to any algorithm right so don't think you're safe right the moment the private key is pushed right that's a disaster right I suppose you can you can argue that https is the most secure because well unless someone also pushes the the private key of the of the certificate and that that's something I talked about right if you if you if you accidentally leaked the private key of the certificate that's also a disaster right all right guys uh so that's all what I wanted to talk about here there is so much happening at GitHub right right these days like they I think they they have like over 12 outages on March 2023 and I think they just laid out like 40 people from India their entire engineering team so there is so much happening and uh wish all the engineer our engineering uh brethrens all the best and uh this something is going on definitely it has to do with AI probably there's a big shift in Microsoft and what's happening but this could be like a significant infrastructure change that caused them to accidentally make silly mistakes like this like pushing the RSA I don't know like pushing the private key of it SSH RSA okay that's kind of bad but I guess so that's what I wanted to talk about again I found this interesting and to unpack this I always like to when I read something I always ask like what does that mean right and I I rarely find the answers so I try to find my the answers myself and I might be wrong and some things some of the things so let me know if you like this stuff talk to you another one bye-bye

Channel: Hussein Nasser

Views: 64,606

Keywords: hussein nasser, backend engineering

Id: WUNu24bM9sE

Length: 21min 57sec (1317 seconds)

Published: Thu Mar 30 2023