What Everyone Missed About The Linux Hack

we've just experienced the biggest exploit in the history of Open Source software I've never seen something that touches on everything from the social engineering side to really well off you skated hacks and just taking advantage of the entire open source ecosystem in order to build a very well-hidden exploit so well- hidden that it only got found by some random Microsoft engineer because he was benchmarking his SSH connections previous similar security issues honestly weren't that similar if we look at things like log for J or even stuff like heart bed in the open SSL World those mistakes were honest mistakes made by maintainers who were just trying to write good code and things slip through code review that weren't built in exploits they were just honest mistakes those people made what happened here wasn't that a good faith maintainer was exploited he was harassed he was manipulated and he was tricked into building trust with an individual who then built a very well off you skated hack and then distributed this across the world if you're running Linux or Mac OS there's a decent chance that you have the things that are exploited here in your systems and if you're running a new enough version of Debian you might even have the exploit to this is Absolut chaos and I want to do my best to cover this responsibly so rather than just being a soy JS Dev pretending I know what security is I'm going to call somebody who's a lot smarter than me talk about the security side now you're probably wondering how did an open source project a project where everyone can go and read the source code get compromised by a malicious back door that's a really good question the way that it was done was actually pretty ingenious and it was done via these two binary object files files that were committed to the repo not as source code but as just Blobs of data these these two binary files are injected into the build process and when they're deop discc turn into bash scripts now what this bad script actually ends up doing is taking these layers that hide the obis data inside of good large compress lzma and extracting the inner evil object file and then making that a part of the build process so that the evil object file is now depended on by the Linker at compile time now as far as affected systems this only matters if a couple cases are true one the repo that you have is not from gits Source control is not from gits Version Control it's a release tar ball that was from GitHub they did not include this code in Version Control to keep it to keep it hidden also you have to be using x86-64 and a Linux gnu variant for this to get compiled into Li lzma as a back door also important to know that the back door only triggers if the following if the following things are true if term is not set as an environment variable and if the binary that is running is user Espin sshd very important even though there is a back door in a widely used compression Library it only matters for sshd now what does this back door actually do so here I have the object file put into gidra the disassembler by the NSA and there is a function here called get CPU ID get CPU ID is a function that is normally just an inline it's a oneliner that get inlined by the compiler inside get CPU ID which gets invoked by the Linker it runs all of the malicious back door functionality now the details of this back door are still getting worked on by the reverse engineering community trying to figure out what all of these named functions that normally would do compression things but actually do backd door things what they do and how they function but if you want to follow along with me I'm working on a video right now for this topic uh it should come out as soon as we find out more about the back door but that's it for now thanks for hanging out if you couldn't have guessed from that little piece low level knows a lot more about the security side than I do and is interesting as that stuff is you might notice there's a lot left to this video and I don't actually plan on talking about the security side like at all anymore this awesome diagram was put out by Frogger and I highly recommend this if you're interested in the security side the thing I want to emphasize here is that in this diagram we have this huge chunk at the beginning that goes from 2021 to where the xlo was introduced in 2024 that is a very small portion of this diagram most of this is focused on how the exploit Works how it was introduced how the back door and the bash file for and all that goes together there are much better people to cover that than me what I want to cover is the part that I don't think is getting enough attention which is the craziest hack I've ever seen the social engineering part here this individual didn't just sneak evil commits into a project that somebody else was running they exploited the existing maintainer in order to take over a project with a lot of users in order to do horrifying stuff so how did they do that I read this phenomenal article by Rob mening that goes deep on the manipulative side of here in the experience of Open Source maintainers and this this is genuinely horrifying let's talk all about how open source's Nature is able to be exploited in this way Rob mening posted a pretty cool article that was originally like a Twitter thread talking all about how this is a failure of Open Source itself to some extent and I think this is a really interesting take that really shows the risk here originally thre it on Twitter about the XC liel CMA vulnerability when I finished typing it I realized I had a real world slice of Open Source interaction that deserved more attention there'll be lots of analysis of the XE lib lzma vulnerability however I found most skip over the first step of the attack again this is why I'm making this video this is a really important piece the original maintainer burned out and only the attacker offers to help this is the key there was one maintainer and then there was two maintainers one of which was exploiting and was waiting for the opportunity to take over and eventually the first one burnt out and now only the bad actor is left this is unprecedented planning and execution like we've never seen before in open source amazingly someone found an archive with an email thread that captured the state of the world just as the Step Zero was taking place let's read their words first we start with a reasonable request asked reasonably the question forces the maintainer to address his failings I use failings in quote here because a the maintainer doesn't actually owe anything here so he hasn't actually failed and B I know exactly how this feels it feels terrible to let down your community this is straight from that email is XC for Java still maintained I asked the question here a week ago and have not heard back oh I hate these messages so much I hate these messages so much I I the number of times I've gotten things like this about random I built or work on this is the worst feeling to be like too busy for like a week to respond to things and the response isn't oh I hope you're doing okay maybe we can chat soon it's oh are you not working on this anymore it's the most passive aggressive and it's the worst feeling to get messages like this this is absolutely a key point of the start of the burnout of this maintainer the maintainer acknowledges that he's behind and is struggling to keep up this is a cry in pain this is a cry for help help will not be coming in this thread again very very real and painfully common yes by some definition at least like if someone reports a bug it will get fixed development of new features definitely isn't very active frownie face again very understandable oh here we're introduced to our Li May attacker in the very same message is not the help you were hoping for certainly not Gan has helped me and he might have a bigger role in the future it's clear that my resources are too limited so something has to change in the long term this is when the attacker offered to help instead an UNH helpful consumer says unhelpful things this is exactly where these types of email threads go progress will not happen until there is a new maintainer the current maintainer lost interest or doesn't care to maintain anymore it's sad to see for a repo like this aside given that this exploit appears to be a purposeful attack by Gan should jagar Kumar be considered an accomplice by actively encouraging the original maintainer to give it up not sure we'll see this unhelpful customer again soon interesting I like the implication here is it possible jagar Kumar isn't a real person and was doing some nuts social engineering to try and burn this maintainer out faster to make it more likely he was willing to give up the project in the first place there is some 200 IQ going on here not just on the exploit side but on the social management side this is a people exploit first and foremost as cool as the security is inevitably the maintainer tries to defend himself maintainers handle the stress of burnout differently I tend to get angry which ends up coming across snarky however this reaction is heartbreaking yep I feel this I definitely am the snarky type but I've seen other really good maintainers just straight up burn out and it's the worst most painful I haven't lost interest but my ability to care has been fairly limited mostly due to long-term mental health issues but also due to some other things this seems like this particular poor Dev was targeted because the package was simple and should have been easy to maintain and he probably assumes such when he built it but because of struggles that existed well outside of the work he was doing he didn't feel like he could maintain it and some other party poked and proded until he gave up and the maintainer also reminds everybody how the world software is built now it's also good to keep in mind this is an unpaid hobby project and as always the xkcd dependency comic is more relevant than ever all modern digital infrastructure all the crazy that we're building is half being held up by some random project by some dude in NE Nebraska maintaining it thanklessly since 2003 yeah this is the first time we've seen at this level somebody look at this chart look at this comic and say you know what I bet I can get that person to give it up and let me hold this up instead and that's exactly what this attacker did and again this is all within 2 weeks somebody filed an issue got no response for a week made a really rude comment a week later they come back and make another rude comment sadly there are definitely real people who do this on one hand I think this is the attacker doing it but on the other hand I've seen people be this rude in Open Source before that I wouldn't be surprised this a real person you ignore the many patches bit rotting away on the mailing list right now you choke your repo why wait until 540 to change maintainers why delay what your repo needs okay now I'm convinced it's the attacker just the tone of this one the attacker is this other person what purpose does this serve I can't tell you how angry this makes me feel for the maintainer yeah I honestly part of why I'm pretending this is the attacker is probably because I can't imagine a human doing this and I don't want to believe they would I'm probably in some amount of denial right now where like obviously a real person that isn't an attacker could do this but in order for me to to be okay with Humanity I have to pretend this person is intentionally acting maliciously to make this all happen another really good point just made from Nick from moderation in security perspective it's now dangerous to not ban rude people like this shout it from the mountain tops absolutely in the future cite this example as the reason you're Banning people because now you don't have to just say I don't want to deal with you now you can say dealing with you might cost the security of our package goodbye be kind if you want to be talked to and I hope we can get that one little positive piece out of all this so back to this uh reasonable requestor he decides to come back in and make demands I sorry about your mental health issues but it's important to be aware of your own limits I get that this is a hobby project for all contributors but the community desires more but the community then Fork it then Fork it if you're not happy with the speed it's moving at move it yourself it's open source you can Fork it at any point he's freaking of the same thing Rob killed it with this article definitely give him a follow if you haven't this is even better than I had hoped read the last sentence again Community desires more consumers must be fed the needs of the maintainer of which there are clearly a few important ones are ignored yeah are no longer reasonable requestor also offers a suggestion notice that there is no offer to actually help there never is they're always bitching because they want you to do the work for them why not pass on maintainership for XC for C so you can give XC for Java more attention or pass on XC for Java to someone else to focus on that for C Instead try to maintain both or trying to maintain both means neither are maintained well I don't love the suggestion here I think there's a way to say this that is kind and thoughtful but is not in a mailing list it's in private conversations after you built some trust with the person but I this one is so far the one that that feels the most well- intended but it's still a really dumb thing to say in a chain mail list that everyone can see like this then the maintainer explained the reality finding a co-maintainer or passing the project completely to someone else is been in my mind a long time but it's not a trivial thing to do for example someone need to have the skills time and enough long-term interest specifically for this also a great Point everybody seems to think at least people who aren't real open source contributors seem to believe that you can just grab some random person and help them maintain your project one of the things open source maintainers ask me about the most is how the hell did I find so many great people to build things like create T3 app with me I shouldn't even say with I should say for they're doing all of the work the reason I am able to do that is I have an incredible community of awesome people like you who hopefully has already subscribed to this channel by the way hit that button below subscriptions are free you should consider it but I have this awesome community of people who do dope this community doesn't have a lot of noobs because being a noob kind of makes my videos hard to watch because I'm not going to teach you the basics of stuff I'm not even going to tell you what the definition of a word is I'm just going to talk about the I'm interested in because of that I have a community of people who are on average much more technical than the typical Community member might be and on top of that I keep a close eye on who's doing the most interesting stuff and pull them in in order to build a tighter little nit Community inside of the chaos that we're doing that's only possible because I have this Mass M platform with hundreds of thousands of subscribers and millions of views a month and even then I can only find like 5 to 10 of these people if you're this random block you don't have that platform you don't have those people to rely on for those types of things which is why the people who are in this position surprisingly often come to someone like me to ask hey Theo I need help maintaining this project can you help me find people to do it it's really really hard to do if you're a random Dev just using the package it might seem like oh just grab someone else it is not that easy yeah so uh if you pressure somebody to just finding someone their bar is not going to be as good it takes skill and knowledge to write software and while many skills and some knowledge will transfer working on a new software project inevitably requires developing new skills and more knowledge some devs are not fungible cogs that you can swap in and out all the time yep like you can't just swap most devs in and out of things especially if they're not being paid they have to care they have to understand they have to be productive and they have to know how to manage a community to do open source maintenance most devs aren't one of those things much l all four so it makes sense that even like the 1% of devs probably aren't cut out for open source maintenance the email third with the complaining consumers offering no help while continuing to make demands only the attacker is left Gian may have a bigger role in the project in the future he has been helping a lot off list and is practically a co-maintainer already smiley face yep and I want to be very very clear not only do I not blame the maintainer here like the original maintainer whatever the opposite of blame is here is how I feel I feel genuinely so sorry and horrified that their mental health was exploited to do something as terrible as what happened here and if anyone ever talks any on this maintainer for what happened here you I need you to hear how bad of a person you are if you blame them for this because they were taken advantage of for doing free hard work for everyone to use they did everything they could in more and they were just trying their best to make sure this thing that people depended on was maintained well they did nothing wrong here they did absolutely nothing wrong here I love the summary here this is really good I totally agree this is a microcosm of things that if you're a maintainer you've experienced and you how bad it is that's why I'm getting getting so heated because I've been a maintainer in the past and I still help maintain a bunch of stuff some of the most thankless work I've ever experienced we just don't get it it's actually funny going from open source to YouTube because I'll do a small thing and get a ton of Praise on YouTube I'll do a big thing and get nothing in open source and I want to really shout out the original maintainer here I know I was just cursing about anybody talking but he did everything he could in more and as a result his GitHub account got suspended G's the attacker but he was following lass so I could see from his following that lass was suspended and I go to the opposite so was Gia so the attacker was suspended that makes sense lost getting suspended no GitHub if anybody there is watching and listening if you have no really good reason for this account to be suspended free him now this is horrifying that somebody who got their mental health exploited and did nothing wrong and has no harmful commits on their account ever is getting any for this at all free L this is nuts he actually updated his blog and wrote some details here and it's mostly just a fact list but at the very least I want to cite this because he deserves to be shown here cuz he's doing everything he can and more huge credit to him OG Prodigy just found a message from the maintainer that I think is really valuable to read here hello I've read the open wall post I've been on holiday and happened to check email I've spent time with friends and they're at my place at the moment too but I thought I have to spend some time on this since I happen to check the emails I'm really tired but I suppose I should do something right now longer investigation by me likely can only start on Monday or Tuesday this sounded too serious to ignore I I feel so bad for this maintainer holy he was literally on vacation hanging with his friends this and is doing his best to jump on it is still being super honest about it the whole time I this is breaking my heart like straight up this sucks God that's so sad I again if anyone gives this individual any you're on my list forever this is his brief overview cuz he just wanted as an official source to give some info here and I have Master respect for him for doing this in finding the time even when he's doing a bunch of other stuff and trying to enjoy his Easter with his friends yet he's still out here talking about it huge credit to him for that also the get repo for the actual project has been removed from GitHub which again if the repo was removed his account should be reinstated the fact that his account isn't reinstated is terrifying to me I I really hope GitHub makes the right decision and brings him back soon this page is short for now but it will get updated as I learn more about the incident most likely it will be during the first week of April again it's not just getting more info it's he's trying to take a vacation but seems to be focused on this as someone who's been in the middle of some crazy drama in the past it's hard to even just like sit with your friends and eat food without it being on your mind constantly it's the worst feeling in the world something so much bigger than you that everyone wants your input on that you know is like some amount about you but you can't really do anything it's the worst feeling in the world and I have so much sympathy for lass for what he's gone through here the git repos are on this URL here because again they've been taking down other places so he's making sure they're accessible XC the t.org the DNS name has been removed the XC projects currently don't have a homepage this will be fixed in a few days facts this is a cve for it XCS 560561 released tarballs containing a back door these tarballs were created and signed by Gan tarballs created by gan were signed by him any tballs signed by me were created by me good to call this out that his signing credentials weren't exploited cuz there are there's been some skepticism on things like Twitter that an account was exploited and people were doing fake commits and someone even said this is a reason to sign your commits you should sign your commits but this is entirely separate from that get up accounts of both me laru as well as Gian have been suspended this sucks this sucks so hard x.org the dnsc name was hosted on gith pages and thus is down to that's why it's down well if you're watching this I'm more than happy to help in any in all ways with the hosting of this if github's not going to reinstate you I'll personally make sure this can be hosted it'll do it out of pocket if I have to you you deserve all of the community support you can get in more and I'm happy to put my own money and time on the line for that cuz I'm I'm genuinely mortified as I keep reading this only I have had access to the main tu.org website get the tu.org repositories as well as related files Gan only had access to things hosted on GitHub which included this site cuz this was like a subdomain that was going through GitHub and this is the only thing he ever had access to on the domain this is a really good call it to make too so that we know what we can and can't trust I'm amazed at how useful this tiny little post is and I'm really thankful that he made this even if he's not taking the break that I wish he would take so again L you're a Legend you're doing this better than anyone would be expected to much l somebody who was like bullied off of the project good this in my opinion is textbook how to handle when something like this happens he he has now written the book of how to do this right and I hope other maintainers that are seeing this and are mortified at the very very least can learn lessons from how well he has handled this and how to keep this from happening in the future as a result this is another IRC message that came from the original maintainer and there's a really good quote in here I wanted to highlight the crazy thing is how much Gia helped I still need to get more facts to exclude that it wasn't his account being compromised Etc although the evidence I've read is heavily tilted already Gia actually helped so he was playing all sides of this this is one of the craziest two IQ manipulative Warfare social and software engineering hacks I've seen ever I can't fathom anything else coming close to this and I feel so bad for the poor maintainer who was exploited in this way to laru on behalf of the open source community and software as a whole I hope you know how sorry we are this sucks and you did nothing wrong here and if anyone here or elsewhere talks any on you do not let that be your problem let that be ours the community need to do better here and we as a group need to stand up for what happened and do our best to build a culture where this can't happen in the future because this was not an individual problem this was not an engineering problem and this certainly wasn't a code review problem this was a community problem and we failed this maintainer and we need to do better that's all I have to say about this one I am horrified so uh yeah I'm going to go send an email to maintainer and let him know that we respect him a lot and do not harass him do not spam him with stuff but if you do see him around let him know that he did good here and until next time peace nerds

Channel: Theo - t3․gg

Views: 40,278

Keywords: web development, full stack, typescript, javascript, react, programming, programmer, theo, t3 stack, t3, t3.gg, t3dotgg

Id: 0pT-dWpmwhA

Length: 20min 24sec (1224 seconds)

Published: Mon Apr 01 2024