Unveiling the xz Utils Backdoor which deliberately opens our SSH connections for RCEs

hi welcome back to my channel geeking Judy and I'm here to talk and try to explain as much as I understand the super shocking news about the recent backd door in many Linux distributions although they haven't reached our major server distributions but you can see them in the bleeding edge once like Arch deian seed and these kind of distros rolling ones with a bleeding edge software uh this is discovered and explained by Andress FR front if I pronounce correctly the fun fact is it is discussed in Mastodon another guide talked about it in blue sky so X or Twitter is losing its Edge and we like it not anything against X but having more options is better having a fediverse is much much better and madon is our choice anyway the news is about a back door in many Linux distributions or possible and it comes from the XZ Library this comes from Andreas he says I accidentally found a security issue while benchmarking postgress changes he was checking the speed of postgress runs and found out there is a problem somewhere here was benchmarking different versions different configurations and found out somewhere it's suddenly becomes slower uh later he says from 0.3 seconds to 0.8 seconds this is huge difficult to observe for normal human beings but he's an expert and he was using the software to check the performance he went forward and found out this is not about post gruel it is about the SSH connection what he found was when he was trying to SSH from one system to another it normally had to took this but on a Debian seed it was taking this much why he started to uh Explore More and R his finding in the open wall uh email open wall is a where many Security Experts discuss the security of their open environment this is from 29th of March he says I found some odd symptoms regarding leap lzm a this is a compression Library part of the XZ package you know about compression when you have something you compress it it becomes much smaller and you can make this larger back and get back your main stuff SSH has not has nothing to do with the comparation so what is going on first he says I uh assume this should be a problem in Debian because I can see it only on Debian seed if you are familiar with Debian you know that seed has the latest software it's like a rolling release but he says afterward I found that this not this does not come from Dean this is is coming from exet library although when checking the codes in exet GitHub which is not there anymore it's disabled I cannot see these problems very strange so he found out that this solely happens in the distributed tbol and not the source code itself again it's becoming more and more stretch this was one of the I wanted to say most complicated but best planned supply chain attacks some people claimed in the main GitHub of XZ there is no problem if you get the code and compile it yourself you won't see any problem but when GitHub creates distribution tarballs the final product the problem is there how can this happen for example it points out to the Debian package which is is shipped this is a M4 file this is the build to host so this is an M4 M4 is a language created by kagon himself if I'm not mistaken used in the original Unix environment now it's in many places it's in autocon system so you write a configuration kind of a program it's a program you write a program and this can be used to create distribution specific builds or anything you want with the auto conf it's a very very versatile system but he says that I observed this if text X some variable is not equal to X so if some variable is there has a value do this what it does it checks for something does some replacement evaluat something does something it's kind of cryptic other lines are not like this other lines are much easier to understand but cryptic things have not started yet this is a very complicated attack he says I observe this and this line is not in the upper stream source of the build to host so he goes forward points to some codes these codes are not there anymore this Roos has been disabled although this is related to the very very INF famous iy package for uh compressions this says this injects an obfuscated script to be executed at the end of the configuration and it's obfuscated so a normal human being cannot read it what does this mean when you write your program you say if blah blah for blah blah print this what happens if you replace if with something strange with a Define if you change your for if you rename your variables to strange things after some time normal people won't be able to understand your code anymore but a compiler can so there are even competitions check for upus skated SE competition you will see lots of fun codes which are impossible to run but compiler can run run it and create fun outputs so somewhere using that line you just saw an unreadable code is added to the iy library what is that it goes forward uh he has tried to follow what is happening this ends up to this one at the end this happens to the code you can see it's unre able does strange things and it has a hello V here and at the end it leads to the attachment which is here in injected txt this is the injected txt see he creates some values this is steps and steps and steps of clearing the code making code more readable what happens is it's a program which is added to the code does many things for example here it checks to make sure that the architecture of the computer running the code is a normal 64bit computer not a arm or something for example or later it checks to make sure that the system is using GCC compiler and also it checks to make sure that this is using a g version of linker you know you write your program you you compile it to something but this using different libraries so you have different libraries here different parts of functions which you you need for example if you are creating an SSH connection you need to sign your code somewhere sign your signature somewhere check your signature somewhere those are coming from different libraries so you write your own program and you say use this Library use this library and use this library now you can call this function that is the work of this Linker and it checks to make sure that you are using GCC and you are using the G Linker and then does all of these still not very easy to understand very difficult to understand and you can see that it loads the libraries from lzm a and everything so what happens later uh the Andress FR person who found this while working on postgress he's an expert on databases has linked to many places these are not here anymore but he have shown the most important parts he says the point is on the version 567 and 561 we have these two lines which adds whatever what I showed you to the iy library there strange thing is people started to check what is going on and it's like a mystery or a trailer a user called G t75 has done this how can someone add something to the library as important as iy which is running on practically all of Linux distributions how can someone inject some mysterious unreadable code and no one uh blinks an eye or objects to it this user joined uh GitHub on 2022 did some uh contribution to one to another uh compression Library it was accepted and it contained some strange code with some print later it was removed then he approached ity project it started contributing here another person start pushing to iy people that you need to add more developers you have to add more developers you have to add more developers and this is a new guy who's working on this project so let's add him as a developer so now he has access even his email is regarded as one of the main contacts for exz and now when he adds some unreadable OBC up Aus gated code to this project no one needs to check it he just adds it because he is trusted so that's how code is added on version 56 0 but what does this code does it's very difficult to reverse engineering something like this the writer of the article says the committer we know G t79 is either directly involved in this attacks or there was some quiet severe compromise of their account this is very unlikely other people tried what he has done and he was suspicious also people saw some changes on the code making parts of it more unreadable from this user so it seems he knew what he was doing his name is kind of like Judi I'm not the guy so and also in some parts people started asking him questions not accepting him as the contributor and another user suddenly appeared defended him and then disappeared so this for me at least this looks like a very very planned attack uh it affect some of the systems and our friend have found it on the open SSH servers as I've told you on a normal system the time for the connection if you know about SSH it's a protocol I'm in this machine this is my server I connect to my server using SSH I issue commands here see the results on my terminal issue commands and see the results it's like sitting behind that specific terminal and working practically all the Linux administrators are using SSH to do their every everyday job it's a secure shell on a remote machine so what he saw was on a normal system establishment of this connection which requires some key exchange uh signature verification and these kind of stuff takes three point uh 0.3 seconds but on a Debian seat which now we know that using ex the library version 5 60 or 1 this was taking much much longer he also tried to examine where this happens and he found out that this only happens when the term environment variable is not set we are using default terminal Arc zero which is the program being run is user Espin sshd also LD debug and LD profile are not set you can use these variables to profile and debug programs when you are doing this you're checking line by line what is happening checking your logs and everything so when in this state this program doesn't do anything yeah Langs need to be set don't know why some debugging environments like RR uh appear to be detected so if you're are debugging the the program and in some cases with GDB which is a more common debugger this doesn't do anything so it's very difficult as soon as you start looking at it it doesn't do anything act super normal in other cases what it does uh this researcher have found it using perf with record so he observed that uh when the back door is working it's taking much longer uh it's replacing IUN resolvers crc32 resolve and CRC 64 result so what is happening is when you are running your program it calls different functions from other libraries this should be resolved where is this these are saved in some tables and different places this is manipulating these ones looking at this replacing this with its own back door versions it changes the symbol tables in memory it's a slow step and this is the one which is detected there is one function it's called RSA public decrypt it's called decrypt but technically if you know about it it doesn't decrypt anything it signs your verification it verifies your signature sorry and it is the PLT which is the uh procedure linkage table we have another thing which is go which is global offset table these are the places where your Linker looks into to find other procedures this uh manipulated one was replacing this so the function CRC resolvers you saw above are their task is finding these functions and create creting a table so whenever in your program you want to call this uh you look into that table and say okay I'm calling this let's run this and then return back to my own program what is happening here is the broken XZ is loaded by SSH although it is not needed but it's a very fundamental Library so when you run SSH this is this runs this is manipulated and back to so when you run SSH you are practically running this specific version of the resolver so this resolver replaces the RSA public decrypt function when you want to run the SSH RSL public decrypt is used to verify that you are the person you are claiming you are so this is manipulated when you run SSH XY is loaded first because it's a fundamental library or whatever reason I haven't checked exactly into it but it runs before any other thing most of the things it changed the resolver so now the resolver manipulates your table and says RSA public decrypt is here instead of the main one which is here from your library so now when SSH runs the RSA public library because this has run beforehand uses this one which is the back door one it signs one specific signature now your SSH accepts whatever comes from here and then goes through the normal path and this works so this is in the middle SSH calls RSA but RSA is already manipulated using the iy library so practically you are using a fake RSS public decrypt which signs your specific version you're a specific person and then calls the main one and this works so it's transparent for you but you have one extra trusted party here very very very strange this is the flow he describes uh you are calling this but practically first you are calling a bad version from here and then going back to the one from lip Krypto H he reported the bug to the dros because obviously he didn't trusted the iy itself and later other people started adding more information for example filipo vorta on the blue sky as I've told you it's fun it's a masteron blue sky and a fediverse he says I'm watching some folks checking this and he has this important piece of information that this new hooked RSA public decrypts verify a signature on the server host Key by a fixed key and then passes the payload to system D so you run SSH ssh goes through calls the fake one he accepts whatever comes from a bad actor and sends it to your system call so tactically this is not a attack on your uh authentication it's a RCA attack remot code execution practically someone with this key can send whatever to your system call H very very very strange attack we can be sure that this gr t75 comes we can be sure I mean I I think so you cannot be sure but comes from a long long plan creating an account getting Trust on one of the important libraries which is installed everywhere and then adding something to it and then disappearing now the whole XY is disabled what they did was this was the affected version and one they reverted back to 540 and checking whatever comes from this user but for sure this actor can have different users countries will start doing this kind of stuff but we are in a free software W uh important code from Eric Raymond this is called lenus laws but this comes from Eric Raymond says given enough eyeballs all bugs are shallow when you have enough eyes watching to all the code what was this is like a room like this when you have enough eyeballs watching to your code like testers code develop Vel opers uh programmers code reviewers at the end you will see all of these box this happens because we are on a free and open- Source software World okay someone can come to the I project and GitHub Get Trust do something bad but on the other side there are other thousand eyes watching the project so this is a new era for sure we can be completely sure that the same thing is being happened with other projects government actors bad teams whatever are trying to send people to the projects Get Trust have trusted developers it's good to pay I don't know 100 PE good developers in your country and they will work on important projects and get trust and everything for the day H but on the other side the good news is this is free software the issues will be seen because exact same thing can be happening with any cloth Source it's enough to send some employee to blah blah company and compromise their OS and nobody will see this now it was possible to check the code and the other hand it's a call for maybe a change there is a joke which shows a very huge system and it's all relies on one brick here and this is your blah blah developer working on a spare time on a library so it's time to give more support to these people who are working on free software if a free software is being used in a Fortune 500 company it's good for that company to pay the guy or G or what anyway have fun I'm glad that we reviewed this we will continue watching it some fun things may happen again dangerous one I repeat nothing happened for major Linux distributions this only appeared in the uh very bleeding edge Dr draws like Arch Linux Debian seed and these kind of things which people use the latest ones normally for example your Debian may ship with 540 and this 560 may come only to Debian seat after 6 months one year it may go to the stable dist so you are safe if you are using an stable this dra on your servers if you are using AR or something do an upgrade and it will upgrade this broken one with the older one have fun enjoy technology and learn everything

Channel: Jadi

Views: 22,017

Keywords: liblzma, hack, xz, ssh, backdoor, JiaT75, security, new, gnu/linux

Id: gyOz9s4ydho

Length: 24min 47sec (1487 seconds)

Published: Sun Mar 31 2024