March 29th 2024, a day that will live on in
infamy as the time that nearly every server in the world was this close to having a hidden
backdoor installed, and we may never have known it. And this was not the typical scenario of
someone creating a virus or exploit and using it to infect systems. It was a carefully
crafted plan that took years to execute, likely by a foreign government, but we
don't know for sure. The story started when a Microsoft developer named Andre Freund
noticed something that 99.99% of people wouldn't. He describes in a post how he was doing some
micro-benchmarking, you know, like we all do, and noticed the SSH process was using more CPU
power than it should have. He also happened to remember some odd error messages in debugging
software he was using a few weeks earlier, shortly after updating certain software. And so
he began to investigate. And what he discovered is mind-boggling, not just because of the backdoor
itself, but even more so the way it was added. So first I'll relatively quickly go over the
backdoor itself, and then how it happened, which I think is even more interesting. This
backdoor was in a software called XZ Utils, which is used for uncompressing certain types of
archive files. It's more of a behind-the-scenes thing, so it's not super well known by most
people, but it is widely used. And I'll give some context on why this was so serious. First
I'll point out that fortunately the backdoor was found while the version of that software was still
on a development branch, basically a beta version, and had not been pushed to regular users yet.
But if it had, it would have been bad to say the least. XZ Utils is used in a ton of Linux
distributions, including Fedora, Ubuntu, Debian, Arch, and more. This backdoor would have allowed
root remote access to any vulnerable system, and it even required a special key that only
the attackers knew in order to trigger it, so it might have never been caught. Real quick
though, on the other hand, I've got a way you can catch and stop threats to your own computer,
today's sponsor, Bitdefender, and their latest new security feature, crypto mining protection. Which
is now available for Bitdefender Total Security, Premium Security, and Ultimate Security plans
at no additional cost. There's a huge variety of malware out there, and increasingly common
is a kind that uses your computer's resources and electricity to mine cryptocurrencies for
hackers, called crypto jacking. But not just that, some websites even have embedded code that makes
your computer mine cryptocurrency for however long your browser is open to that page, that's known
as drive-by mining. This can happen either with or without the website owner's knowledge, like if the
website has been hacked. In the past, even US and UK government websites had crypto mining software
injected into them. Now of course, because some people actually do crypto mining for themselves
legitimately, the feature is totally optional and customizable. You can choose whether or not to
enable the feature at all, and if enabled, choose whether it blocks crypto mining activity, or just
notifies you if any is detected. And of course, you can add specific exceptions if you want. So
definitely check out the link in the description to learn more about how you can protect yourself
with Bitdefender's new crypto mining protection feature. And with all that being said, let's
get back into the story about that software backdoor. It all started two years ago, and
this meme basically explains the context. The XZ Utils software, despite being so ubiquitous, was
maintained by only one guy. He's apparently pretty burned out at the time, and at one point says that
he still got a lot of unanswered emails, and he doesn't plan on developing any new features for
the tool, only bug fixes. However, fortunately, really seemingly so, one guy named Jia Tan showed
up and made a couple small contributions to the source code here and there. That's kind of the
point of open source software, anyone can chip in and help, and then the so-called maintainer,
who controls the project, can ultimately decide what code actually gets added. A few months later,
other accounts start making occasional requests, asking the maintainer if he's still working
on the project, and they mention how it hasn't been updated in ages. At this point, the
maintainer talks about how he knows he's behind, and he even mentions how Jia Tan has been
helping. Then another person, Jigar Kumar, shows up and responds to the original poster by
making a jab, saying "progress will not happen until there's a new maintainer. You're better
off waiting for a new maintainer or forking it yourself," meaning to add the feature himself. The
maintainer tries to defend himself, and explains that it's an unpaid hobby project, and he has
other stuff to worry about too. A week later, this same guy, Jigar Kumar, who was almost
certainly actually part of the whole attack, comes back even more insulting, saying "you ignore
the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait
until 5.4.0 to change maintainer? Why delay what your repo needs?" The original person asking
the question then makes a seemingly empathetic comment, "I'm sorry about your mental health
issues, but it's important to be aware of your own limits. I get that this is a hobby project for
all contributors, but the community desires more." Then he says how he should pass on the
maintainership of one of his projects instead. But given the context we know now,
definitely some, and probably all of these people, were teaming up to wear down the maintainer, and
convince him to add another maintainer, who of course would be that other guy who's been helping
a bit. The maintainer even mentions that Jia Tan has been helping a lot, and might have a bigger
role in the future. For the next six months, nothing really happens. Jia Tan keeps making
contributions, and near the end of 2022, he is officially a maintainer on the project, and added
as an official point of contact. So he is able to make direct contributions to the code himself at
this point, but the attacker was patient. For over a year, nothing really happened. Jia Tan made some
contributions, though we know that a couple of these, while maybe innocuous on their own, we now
know some of these were actually added to lay the groundwork for the backdoor to be able to be added
later. Then finally, in February 2024, over two years since Jia Tan made his first contribution,
the attack began. Very quietly. Jia Tan added a few additional test files that were binaries to
the project, that had hidden binary malicious code in them. Apparently these types of things are
pretty common, and notably, there is no source code for these provided. But even long before Jia
Tan showed up, the description of similar files said that there wasn't any source code, it's just
for testing. So it's not like this was unheard of. A day later, he added another malicious file meant
for building the source code, basically to install the software. Notably though, these files, while
included in the release archives you download, they aren't in the source code repository
online. So while not secret, they aren't really ever inspected by anyone. And there are lots of
other legitimate files like this along with the malicious one, so it's not like it's suspicious
in itself that this was added. Several days later, Jia Tan made another change that in very simple
terms broke a security check in the configuration script, which further contributed to the backdoor
being able to work. And here's the actual code, where the green text is what he had added at the
time. Can you spot the extra character that was added that broke the script? Yep, it's that single
period all the way off to the left. But because technically it was inside this function, it would
mess up the check for whether to add the so-called "landlock" feature, kind of controlling what
access the program has access to, put very simply. And now the backdoor was officially in. But it
wasn't over, because it would be some time before the new version of XZ Utils with the backdoor
would be updated by default in actual operating systems. And it actually did make it into the
beta versions of Fedora and Debian. Fortunately, it was just a couple days later that Andre Freund
discovered the backdoor, and just in time, because it was on track to be added to the next versions
of those operating systems and many more. Now, interestingly, it turns out that a package that
the backdoor relied upon called liblzma actually had a proposed update in the works that would
have broken the backdoor. This was just by chance, no one knew about the backdoor, but the attackers
likely did see that contribution coming and knew it was going to be implemented. And this
probably put pressure on them to get the backdoor implemented as soon as possible. The race
was on to get their backdoor into the operating systems before that other software's update. In
fact, at this time, another person who had made a contribution a year prior that in hindsight was
a bit suspicious, came back and made a bug request to Debian to update to the latest XZ version. And
Jia Tan made one for Ubuntu. They really wanted to get that backdoor in fast. From what I've seen,
we still don't know what the backdoor was fully capable of. We know that it would have allowed
the attackers to remote into affected servers and have root access, aka they literally could
have done whatever they wanted. We just can't be sure because we don't have the source code of the
binaries containing the bulk of the backdoor. Now, you might be wondering how archiving software
would have allowed remote code access. Turns out those malicious test files would be extracted
during the XZ install process, and replace some files used in the installation of that other
software I mentioned, libLZMA. So basically, like many programs, XZ also requires it to install
other software along with it called a dependency. But the backdoor installs a corrupted version of
libLZMA. And it turns out that libLZMA is also indirectly used when certain operating systems use
another feature, SSH, a typical way you remotely access servers. So indirectly through this
backdoor, they kind of added a vulnerability where they could get in through this other SSH feature.
Now, because that other change to libLZMA was in the works, this means the backdoor wouldn't have
worked for very long, but the attackers wouldn't really need very long. They probably had malicious
payloads ready to go to immediately infect any possible servers they could, which would give them
further continued access even if their original backdoor stopped working. Given how sophisticated
and dedicated this attack was over so long, it is very likely that the attacker was nation state
sponsored, aka some government agency around the world did it. And I would bet that they had some
very specific targets in mind, and that having a backdoor everywhere else was probably just a
bonus, because the more you use a backdoor, the more likely it is to get caught anyway. But now
this has got a lot of people wondering, we just happened to catch this one in time, but has this
already happened in other software that we don't know about? I would actually say that it's pretty
likely, and that's not even including the untold number of unintentional bugs and exploits that
hackers may know about but keep for themselves. It's all very scary. I'd be really curious to know
what you guys think about this, and especially if you had even heard about it. I don't remember any
major news outlets talking about it, except for maybe some tech news specific websites and blogs.
Thanks again to Bitdefender for sponsoring. Be sure to check out the link in the description
to learn how Bitdefender's new crypto mining protection feature can protect you from malicious
crypto miners stealing your computer's resources. Anyway, if you liked the video, give it a big
giant thumbs up. And if you want to keep watching, the next video I'd recommend is where I talked
about 11 cool command line programs that you probably want to know about, even if you don't
usually use command line programs. I'll put that link right there. So thanks so much for
watching, and I'll see you in the next one.
