- Hi, and welcome to our talk. My name is Guillaume Fournier, I'm a security engineer at Datadog. And today Sylvain and I are
going to present the rootkits that we implemented using eBPF. If you don't know what
eBPF is, don't worry, we are going to present this technology and tell you everything you need to know in order to understand the talk. So let's start with a few words about us. We are the cloud workload security team. We usually use eBPF for good. And our goal is to detect
threats at runtime. Everything we do is added
to the Datadog engine, which is an open source project. So feel free to check it
out if you are interested. That being said, for DEFCON, we decided to use everything
we knew about eBPF to build the ultimate rootkits. So, as I said before, we
are going to start the talk with a brief introduction to eBPF. Then Sylvain will take it over to talk about how we
implemented obfuscation and persistent access in our rootkits. After that I will come back to present the command-and-control feature along with some data
exfiltration examples, and then I will talk about
the network discovery, and RASP bypass features of the rootkits. And finally Sylvain will
present a few detection and mitigation strategies
that you can follow to detect rootkits such as ours. All right, so let's start with eBPF. EBPF stands for extended
Berkeley Packet Filter. It is a set of technologies
that can run sandbox programs in the Linux kernel without
changing the kernel source code or having to load kernel modules. It was initially designed
for nitro packet processing, but many new use cases
were progressively added. So for example, we can now use eBPF to do kernel performance tracing along with network security and
runtime security in general. So how does it work? So eBPF is simply a two-step process. First, you have to load your eBPF programs in the Linux kernel and then
you need to tell the kernel how to trigger your programs. So let's have a look at the first step. EBPF programs are written in C. So it's not exactly C, it's
more like a subset of C because of many restrictions
that eBPF has to follow. But I'm gonna talk about this later. So once you have your C program, you can use LLVM to
generate eBPF bytecode, which you can then load into the kernel using the BPF syscall. EBPF programs are really
made of two different things, eBPF maps and the actual program. So there are a lot of
different types of eBPF maps, but all you need to know is that they are the
only way to persist data generated by your eBPF programs. Similarly there are a lot
of different program types and each program type
has its own use case. However, regardless of the program type, each program has to go through the same following two phases. So the first one is the verifier step. So I will talk about
this later, but for now, just know that this ensures
that your program is valid. And second, your eBPF bytecode will be converted into machine code by a just-in-time compiler. And when those two phases succeed, your program is ready to be executed. Step two is attaching eBPF programs. So in other words, this is
when you tell the kernel how to trigger your program. So there are many different program types and I can't present them all, but I'm just going to
talk about four of them. So for example, you can use kprobe to trigger any BPF program whenever a specific symbol
in the kernel is called. Tracepoints are similar to kprobes, but the hook points on
which they can be attached have to be declared manually
by the kernel developers. Those two programs
require an older syscall in order to be attached. And this is called as the
perf_ event_ open syscall. So the other two program
types I wanted to talk about are TC classifiers, so it's
sched_cls and XDP programs. So those program types can be
used to do packet processing. So whenever some network
traffic is detected at the host level or at a
specific network interface level, those two require the
netlink to be attached. And the only thing to remember here is that each program
type has its own set up and does not require a
different level of access. Another very important fact about eBPF is that the BPF maps can be shared between different programs regardless of their program types. All right, so the eBPF verifier. So the verifier is used to
ensure that eBPF programs will finish and won't crash. To do so it's really just a list of rules that the verifier checks and your program has to
comply with those rules. So for example, your
program has to finish, it cannot be like an infinite loop. So your program has to be
a directed acyclic graph. You can't have unreachable code, you can't have unchecked dereferences. Your stack size is limited
and your overall program size is also limited. And finally, one of the
most infamous features of the verifier is it's
very cryptic outputs. So basically if your program
doesn't pass the verifier step, you will have a huge log of everything that the verifier looked into and eventually some kind of
error telling you what happens. But yeah, basically you are in for a very painful beginning session. Last but not least, eBPF
comes with a list of helpers that will help you access data or executive operations that you wouldn't be
able to write natively. For example, you have context
helpers, you have map helpers, a lot of things that you
wouldn't be able to write in C and that you would need
external instrumentation to do. In short, you have about 160 helpers. And most of the heavy
lifting of your eBPF programs will be based on those helpers. So that concludes this
introduction to eBPF. And I will hand it over to you, Sylvain, so that you can kick off the
presentation of the rootkits. - Thank you, Guillaume. Before we get into the details, let's see why eBPF is an
interesting technology to write a rootkit. First, the safety guarantee. Broad eBPF mean that a bird in our rootkit cannot crush the host. (indistinct) will not cause
any low message to be emitted. The user therefore has no way to know that something actually went wrong and notice the presence of the rootkit. As we saw earlier, the eBPF bytecode is converted to native code. And the number of instruction is limited, which limits by extension,
the performance impact that our rootkit can have on the machine that could otherwise be
detected by the user. On the commercial side, the eBPF is used by an
increasing number of vendors in various use cases, network monitoring security, for instance. With eBPF becoming widespread, returns of one product being abused, too many issues programs also increases. The safety guarantee
we just told you about should not give the
security and (indistinct) before filling of security. There's a lot of activity around eBPF and each new version of the Linux kernel comes with a new set of eBPF helpers, bringing new capabilities. As we wanted our rootkit to run on a widely used distribution on Linux such as RHEL/CentOS Linux,
or the latest Ubuntu LTS, we used a limited number of
helpers using RHEL/CentOS or a feature like KRSI
will have probably made the development of the rootkit easier. One of the primary tasks of a rootkit is to ask itself, what
does it mean in our case? EBPF programs are bound
to a running process. If this process gets killed, all the attached eBPF
programs will be unloaded. For that reason, it is essential that we
boast either our program and protect it from being killed. The eBPF programs and
maps used by the rootkit should also be hidden and we
should forbid other programs to gain access to them, so
there are five descriptors. So let's see a rootkit in action. So let's start with kit,
would give us its PID. Then we can try your PS kernel in order to see if we can
detect it from the output. We can try using its
preface entry and nothing. We can even try using your such file or even a relative pass. And we still have the same
issue, no such file or directory. And finally, we can try to send a signal to see what happened, and we
get no such process error. Your prescription
capabilities of our rootkit mainly rely on the use of two BPF helpers. So BPF provides user helper
arose our BPF program to ride into machinery of the process that issues the syscall. This can be used for instance
to alter the other data that is returned by your syscall. It's also possible to alter
the syscall arguments. So it's one caveat with this helper, the memory to be modified has to be mapped into the kernel space. Otherwise a minor or major
pitfall will be triggered causing the BPF provider
user call to fail. User BPF helper is to ease
the BPF override return. This one allows you to change the return that you have a syscall and
as an interesting property. If this helper is used
at the syscall exit, it will simply change the return that you will do at the syscall. But if we use it at the
entry of the syscall, the execution of the syscall
will be completely skipped. It is important to note that this helper can only be used at the entry
of the syscall or at the exit. So let's see how obfuscation
of a file actually works. At startup, the rootkit
will populate the map with the pass of it's PID folder. Now the user space issues a
file with syscall such as stat. The syscall usually come in two forms, one that accept the pass
to the file as a string, another one that accepts
the VFS for the file, but the user space program
must have previous (indistinct) using an open syscall. So let's consider the former. To properly identify the targeted file, the rootkit needs to do an
accurate resolution of the pass as the pass specified
could be a relative pass. At the entry of the syscall, that is not in context to the resolution. So we need to go deeper
in the kernel in our case, in DBFS code. So we are at the region, but at this point we
cannot block the syscall as we are outside of
the override of points for the eBPF override return helper. So the only thing that we can do is to change the return value so that the user space believes
that the syscall failed. We also need to scrub the
content of the switcher that could have been filled by the kernel. Now let's consider the latter, the ratio that accept the file as return. We do the same passage version as before, but instead of just pretending
it was the descriptor file that the syscall failed, we
still have the file descriptor that we can edit located into an eBPF map. If the same processes issue the syscall with the same file descriptor, we can add the syscall
entry return an error and block the syscall. In this situation, the
user has no way to know that the file descriptor exist. And as we control the read syscall, we can also hide all the references to the file descriptor in proxy phase. Blocking the syscall that
accept the PID as an argument is trivial if you're
using BPF override return, same for in kernel modules. Now let's see your demo. Let's demo the obfuscation
of our BPF programs and maps. So we still have the rootkit started and we've released the
maps and the programs, thanks to the eBPF tool kernel online. We can't see anything
related to the rootkit. Now if we (indistinct) kprobes and maps, and we list, again, the programs, we can see the program
related to the binary, but still nothing related to the rootkit. Then we can even try checking the kprobes and still nothing related to the rootkit. The ID is to hook into
the eBPF syscall itself, two things are required here. We want to avoid the real space to utilize the two IDs of our own duties using the BPF_prog_get_next_id command. And we need to prevent the user space from getting a file descriptor
to the program or map from the ID using BPF_prog_get_fd_by_id basically to prevent ID guessing. When these operations are
handled by the rootkit, the user space is not able to view or modify our programs and match. BPF provider user is a
crucial part of our rootkit but it comes with a pitfall. While an eBPF program that
makes use of it is loaded, the journal printer pretty
scary, but legitimate message in the kernel ring buffer. So let's see how the rootkit handle this. First, let's check what
is the current statues of the kernel ring buffer
using the kernel dnesg. So nothing special here. And we can even start
a cache on the device used by the ring buffer. Then we start the rootkit. So we can see that the cache terminated with a pretty legit looking message. We can recheck the kernel ringer buffer, thanks to the message kernel online, and we can only see
legit-looking messages, but I need more. And then we can stop the
rootkit and we can run again, the message kernel online in order to see what was
overridden by the rootkit. So let's see how it works. In this diagram, Ronald is
waiting for a new message and is blocked on a red syscall. Two BPF programs are loaded
secondly by the rootkit. The first one, which doesn't
choose eBPF provide user work on our red syscall
and we make it written zero and we'll write legit-looking message. This will guarantee
that the warning message is related to the BPF
ProLite user won't be read. Then the second program using
BPF ProLite user is loaded. At this point, the red
syscall can be unblocked and we can override the
content of the warning messages with legit-looking messages. Another important task for the rootkit is to set up a persistent access across reboots, for instance. The rootkit can copy
itself to a dedicated place in either it's binary file
with the same mechanism that we already saw. For the persistent access to the system, we can use a generic method
close to what we described in the obfuscation path. We can replace the content
of some critical files read by the root demands,
such as crotab or even sshd. Let's see an example targeting sshd and using the reader override approach. So the approach here is to open an sshd to the authorized key files. Only sshd should be impacted, meaning the file will remain the same for the user point of view. And we want to have it available
to the command-and-control. So let's see this in action. So let's take the authorized
keys content first. So we can see that only
one key is present. So let's start the connection and it's in that password is required. So now we're going to start the rootkit and we are going to specify that we want to inject an SSH
key to the authorized keys but only for SSH. So we can try your connection again, and it seems to be successful. And now we can check what is the content of the authorized key from
the user point of view and nothing changed apparently. Persistent access to
an application database can also be set up using
another type of eBPF program. Uprobes eBPF programs attached
to user space function. In addition to being safer
and easier to use than ptrace, they offer a valuable advantage. The kernel will automatically
set up for us the hooks on every instance of the program. Let's see a uprobe
demonstration using (indistinct) So first, let's try to
connect to (indistinct) using the word bonsoir as password. This one seems to be the
good one, then trying hello, and this one is rejected. Now we start the rootkit and we'll get the opposite treasured. Now the value password is hello. So the idea here is to hook on the md5_cript_verify function of a progressive creator that check whether the
user provided the right MD5 for its role, passwords, and the challenge sent by the server. Overwriting the expected
hash contained in shadow_pass with a known value makes
the comparison succeeded and give persistent access to
the database to the attacher. Now I will hand over to
Guillaume that will show you the command-and-control
capabilities of the rootkit. - Thank you, Sylvain. Let's talk about the
command-and-control feature of the rootkits. So what exactly do we want to do? We want to be able to send
commands to the rootkits to exfiltrate data and
to get remote access to the infected hosts. Unfortunately, there are a
few eBPF-related challenges that we need to face in order
to implement those features. First, you can't initiate
a connection with eBPF. Second, you can't open a port. However, eBPF can hijack
an existing connection. So in order to show up this feature, we have set up a very simple
infrastructure on AWS. A web app was installed on any C2 instance and we used a Classic Load Balancer to redirect HTTPS traffic
to our instance over HTTP. In other words, the TLS termination is done at the Load Balancer level and its eBPF requests are sent to our instance and encrypted. So our goal is to implement CNC by hijacking the network
traffic to our web app. First, we need to figure
out which eBPF program types we're going to use in order
to implement this feature. Although eBPF provides a lot
of options to choose from, we decided to go with
two eBPF program types, XDP programs and TC classifier programs. So both of those programs are usually used to do deep
packet inspection use cases. And while XDP only works for ingress, TC works on both ingress
and egress traffic. Another difference between
the two program types is that XDP programs can be offered to the network interface controller, which essentially means that
your program will be run before the packet enters any subsystem into the network stack. On the other hand, TC
programs have to be attached to a network interface, but
much later in the network stack, which means that they are
triggered later in the kernel. With both programs, you can drop, allow, and modify your packets. And with an XDP programming, you can also retransmit a packet. This option is actually
super interesting for us because it means that you can
essentially receive and answer to a packet even before it
reaches the network stack, which in other words
means that you can do this even before it reaches any
kind of network firewall or monitoring on the hosts. Skipping the network side
also explains why XDP programs are mainly used for DDoS mitigation and TC programs are
usually used to monitor and secure network access at
the pod or container level. So what you need to remember
about this slide is that first, XDP programs can be used
to hide natural traffic from the kernel entirely and TC programs can be
used to exfiltrate data on its way out. First, let's see how we used XDP programs to receive commands with the rootkits. So we implemented a client for the rootkit and this client communicates
with the rootkit by sending simple https requests with custom routes and custom user agent. So after going through the Load Balancer, the request eventually reaches the host and triggers our XDP programs. Then our program pass the
requests, the http routes and understand that this request is not meant for the web
app, but it's meant for us. So after sending the other, sorry, after reading the user agents, the rootkit executes the
requests that come in and moves on to the final step. So this final step is probably
the most important one. It overrides the entire request with a simple health check request, and we do this for two different reasons. First, we don't want the manager's request to reach the web app or any kind of user space monitoring tool that might be bringing and that might detect the unusual traffic. And second, we want the
client to receive an answer in order to know if the
request was successful. So, as I said before, we could also have drop
the packet entirely, but since we're using TCP, the Load Balancer would have
retransmitted the packet over and over again until
the request times out. And this would have generated noise and increase our chances
of getting discovered. That said, if you were
working with a UDP server, this would be a totally valid strategy. So let's have a look at how we can send
Postgres kernels remotely. All right, so on the left of the screen, you can see two different shells. Those shells are connected
to the remote infected hosts on AWS and on the right,
this is my local shell and this is the attacker machine. Okay, so let's start with trying to log into
the Postgres database using the normal password. And again, the rootkit is not running yet. So as you can see, the
bonsoir passwords works fine. And then they start the rootkit
and restart to log in again and as expected and as you've seen before during Sylvain demo, it doesn't work. So you have to change into, hello, and this time it will work there you go. Okay, so we're gonna try
to do the same thing, but instead of hard
coding the new password with the rootkit, we're gonna
define remotely through C&C what the new password should be. So, as you can see, we
have a custom client that will make a request to its CPS, it needs to be a CPS
request to defcon.demo.dog and then we will provide both the role and the secret to override
the normal secret with. So the request that will go
through is a very simple one with the custom routes and the user agent will
contain the new password that will be used at runtime. So as expected, we get the 200
okay from the health check, which essentially means that we know that the new password now is
defcon and not hello anymore. So as you can see,
hello that doesn't work, but if I'm change it
to defcon, here you go. This time it does work. Okay, so this is how we send
the command to the rootkits. Now let's see how we can exfiltrate data. So to exfiltrate data, the client has to send an initial request to specify what kind of data
and what kind of resource we want to exfiltrate. So the XDP part of this process is basically the same as before, but this time the XDP programs
stores the network flow that made the request along
with the requested resource, sorry, in an eBPF map. And the reason why we do so is because when the web app
answers the health check, we want to be able to detect the packets that are meant to be
sent back to the clients. So when the XDP answer reaches
the TC egress classifier, our eBPF program looks at the network flow and overrides the answer
with the requested data. Now, the question is, what kind of data can you
exfiltrate with the rootkit? And the answer is, well,
pretty much anything that is accessible to eBPF. And the reason for that
is, as I said before, multiple program types can
share data through eBPF maps, regardless of what those
programs are supposed to do. So basically you can exfiltrate
things like file content, environment variables,
database dumps, in-memory data if you start looking at
the stacks of the programs. Anyway, you can pretty much
exfiltrate whatever you want. So let's have a look at a simple demo that we can exfiltrate
Postgres credentials along with the file
content of etcpasswrd demo. All right, so again, the
two shells on the left are the ones connected to
the infected hosts on AWS and on the right, this is my local shell. So the first request that I make here is to do Progres list, which basically means please
list all the credentials that you have detected so far
since the rootkit has started. And as you can see, the answer was, sort of
the health check answer was overridden with the content of a map that we used to store the passwords that we've collected at runtime. And again, remember that with Postgres, you don't need the
clear password to login. You just need the hash password that is stored in the database. Here you go. So now we're gonna try
to do the same thing to dump the contents of etc password. So to do so, this is a two-step process. The first thing you want
to do is tell the rootkits to start looking for this specific file. And as soon as a user space
process tries to open the file and read the content of the file, our rootkit will actually copy the data as it is sent to the
user space application and say that into an eBPF map so that it can be retrieved later. So this first request
will tell the rootkits to start looking for etc password. And now let's go back to the host and trigger some kind of pseudo operation so that our user space process
tries to open the file. Here you go. And then this time,
instead of saying, add, we're gonna say, get, and this will dump the content
of the etc password file. Here you go. All right, so the cool
thing about this technique is that it applies to any
un-encrypted network protocol. So for example, we also
implemented it for DNS, which means that you can actually
use it to do DNS spoofing. So the only difference between
the normal way of doing this and the DNS spoofing is that
instead of using a TC program to override the answer of the request, you will actually switch
to see an XDP program because DNS requests
are made from the host instead of received by the host. All right, so let's move on to our network discovery feature. So I know everybody knows what it is, but I have to say it anyway, network discovery is the
ability to discover machines and services on the network so that you know where you want to go next in the infrastructure. And also discovering services
is a super important step when you are trying to
pivot between hosts, because it will tell
you what kind of attacks you might want to try. So the rootkit has two different networks
discovery features. One of them is passive,
the other one is active. And you can control both of them through command-and-control. So I'm gonna get into more details later, but basically the only
difference between the two is the kind of networks
scanning you're looking for and also the level of traffic that you are willing to
generate on the network. So first, let's have a
look at the passive option. So the passive option is simply a basic network monitoring tool. So it will do pretty much the same thing as any other eBPF-based
network monitoring tool, which means that it will listen for any ingress or egress traffic and then generate a graph from all the collected network flows. It will also show you the amount of data that was sent per network flow. And to implement this feature, we used our TC and XDP programs. So the TC programs were used to monitor the egress
traffic and the XDP programs were used to monitor the ingress traffic. So for this version of the rootkits, we are limited to IPv4
and TCP UDP packets, that said support for
IPv6 and all the protocols could have been headed easily. So the reason why the
passive option is pretty cool is that it will not generate
any traffic on the network. In other words, it is basically impossible to detect that someone is
tapping into your network and more specifically at the network that creatures this
specific infected host. However, this doesn't work for services that do not communicate with the infected host. And so in other words, the graph will definitely not be complete. And that's also why we
implemented the active method. So the active method
is a simple ARP scanner along with a SYN scanner. So we implement it using
only our XDP programs, which means that the entire process is done without involving
the kernel stack. And although this will
be a slower process, you can use this method to
discover hosts and services that are reachable by the infected host, but that are not communicating usually with the infected host. And again, the rootkit client will generate a nice network graph for you once the scan is complete. So on a technical level,
this feature of the rootkit is actually quite interesting
because, as I said before, eBPF cannot create a
connection from scratch. So in other words, we
had to figure out a way to generate hundreds of SYN requests while dealing with this
limitation of eBPF. So let's see how we solve this problem. So in order to send a SYN request, you first need to know
the MAC address of the IP that you want to scan. To do so, we use the same trick that we've been using so far, which is to override the requests
from the rootkit clients. So when our XDP program
receives a scan request for a specific IP and
a specific port range, it will override the entire
request with an ARP request for the target IP. And then instead of returning XDP pass, which is what we've done so far, and also which would send the
packets to the network stack, our eBPF program returns XDPTX. So what XDPTX does is send the packet out to the network interface
controller it can inform. In other words, our HTTP packet was transformed into an ARP request and broadcasted back to
the entire local network. So eventually the target IP
will answer the ARP request and will be able to store the MAC address of this specific IP in an eBPF map. However, during this entire process, the TCP packet that was used
to send the HTTP request was never acknowledged by the kernel. And that is simply because
it never made it's way to the kernel in the first place, which means that the Load
Balancer or the client itself will eventually try to
retransmit the packet. And when this packet is retransmitted and when it eventually
reaches our XDP program, we will do the exact same thing. But this time, instead of,
because we know the MAC address, instead of overriding the
request with an ARP request, we're gonna override the
request with a SYN request. And more specifically a SYN request with the first port of
the provided port range, the target IP and the MAC
address of the target IP. And assuming that the
remote IP or the remote host doesn't have any kind of
prediction against SYN request, sorry, SYN scanning, it
will answer either resets or SYN plus acc to this first request. So reset would mean that the
port is open and SYN PLUS acc would indicate that there
might be a set of sprinting running on the host. And this is where basically
the network loop happens and the reason why we were able to generate
hundreds of packets while dealing with the limitation of eBPF, that is the inability to create packets. So whenever we get an
answer from a SYN request, we override the received
packet with another SYN request on the next port. And we also switch the IPs,
switch to the MAC addresses, and send it back again to the target IP. And we do so in a loop until we go through the entire port range. So eventually the clients
will try one last time to retransmit the initial HTTP requests. Because once again,
during the network loop, we never answered the second retransmit. So eventually when this third retransmit reaches our XDP program, we
will override the request with the usual health check requests so that the 200 okay
answer will make its way back to the client after
the request was handled by web app and user space. All right, so let's see it in action. So on the right of the screen, this is a shell to the
infected host on AWS and at the bottom here, it's another one. And at the top, this is my
local shell on my machine. So the first thing you want
to do is to start the rootkit. Then second is to start dumping
the logs of the rootkit. So an eBPF can actually generate
logs using a trace pipe. Obviously you would not want
to do this in a real use case for a rootkit, but this is a great way of visualizing the scan
as it goes through. Yeah, so that's why I'm doing this and that's why you will
see what the rootkit does at runtime. And then let's make the scan request. So what I'm saying here is
please scan the IP 10.0.2.3 from port 7990 and for the
next 20 ports after this one. So the first thing you can see is the request is immediately
changed into an ARP request and we already got the
answer for this ARP request. So next step, when we get a retransmit, we will change this into a SYN request. Here you go. So the SYN request went through and then you can see
the loop that happens. And we increased port one by one until we reached the final port requested by the port branch. And then now we are waiting
for the third retransmit and this retransmit will be the one that we override with
the health check request, which means that we will
eventually get here, go the 200 okay, in other words, the answer from the user space web app. All right, so now what you want to do is retrieve the output of the scan and exfiltrate all the network flows that were detected at
runtime, and here you go. So you would say network_discovery_get
and eventually ... So it actually requires a
lot of different requests because there's a lot
of data to exfiltrate, but eventually you will get the
entire list of network flows that were captured by the rootkit. Here we go. So you have all the
different individual flows and then more importantly, you will have a graph generated for you. So this one is the passive,
sorry, active graph. So as you can see, range there,
you can see the ARP requests and replies between those different hosts. And then in gray, those
are the SYN requests and the reset answers. And in red is the only SYN plus acc, and so from the remote host. All right, and then you
have also the passive graph, which is the one that we saw before. Okay, so now let's move
on to our RASP bypass. So RASP stands for runtime
application self-protection. So in a few words, a RASP is a new generation
of security tools that uses runtime instrumentation to detect and block
application level attacks. And more importantly,
it leverages its insight into the application in order to make more
intelligent decisions. So simply put, it is some kind of advanced input monitoring tool that can
detect malicious parameters and can understand if a malicious input will successfully exploit a
weakness from one of your apps. So the textbook example of a RASP is usually a SQL injection. So the RASP route implement
multiple functions, instruments, sorry, multiple functions in the libraries that you use such as for example,
the HTTP server library or the SQL library. And it will check at runtime that the user control
parameters in your queries are properly sanitized. If not, the RASP will stop the query before it reaches the database
and redirect the client to an error page or some
kind of error message. In other words, a RASP
relies on the assumption that the application runtime
has not been compromised, which is exactly what we can do with eBPF. So just a little disclaimer
before I move forward. I want to stress the
fact that we are playing outside of the boundaries of what a RASP can protect you from. And more importantly, this bypass does not apply
to one specific RASP, but to all of them because this is one of the core principles of how a RASP works. So let's have a look at how RASP products a good
web app from a SQL injection. So let's say that you have a web app with a simple products page and get parameter to specify
the category of the products that you want to see. Chances are your web app uses the default go
database SQL interface. So this is a generic
interface that you can use to query your database
without having to worry about the underlying driver and the database type that you're using. More importantly for us, since it is such a generic interface, this is usually where the RASP
tools instrument your code because simply it's much
easier to hook at this layer rather than having to hook onto
all the underlying drivers. So when your request is
handled by the web server, the query will be formatted with the provided category parameter and eventually the web app will call the query context
function of this SQL interface. This is when the RASP checks the query and makes sure that everything is normal. And if it is, the execution
will resume its normal flow and the underlying driver will be called. So in our example, we use SQLite, so the SQLite driver is called. Eventually the query makes
its way to the database and the answer is sent back to the client. However, if the RASP detects
that something is wrong or detects some kind of SQL injection, it will block the query
and redirect the client to an error page. All right, so now let's see what we did to bypass this protection. Well, the answer is
actually pretty simple. We added a uprobe on both
the database SQL interface and the SQLite driver interface. What this allows us to do is
call one of our eBPF programs right before the RASP checks the SQL query and trigger another one right before the SQL query is executed by the database itself. And thanks to the BPF ProLite user helper, we can override the input
parameters of the hooked functions so that the RASP sees a benign query and the database executes
our SQL injection. And the cool thing about this is that we can even do it conditionally, which means that we can bypass the RASP only if one specific secret password was added to the beginning of the query. Perfect, so let's move on to the demo. So as you can see, we have
a very simple web app. So it's a shoes retailer. So you have a lot of different products and you can filter by category. So let's try to do a SQL injection using the get parameter. So the injection was simply be select star from user and because the
RASP is not running right now, the SQL injection should work. Here you go. And if you scroll down this time, you will see the users along
with the passwords, perfect. So now let's restart the
web app with the RASP, which is what I've just
done and try this again. So let's go to the shop and then override the
category parameter, perfect. And this time the RASP blocked the request because it detected that someone tried to do a SQL injection and the SQL injection would
actually have succeeded. Great, so now let's start the rootkits by providing the path to the web app and then try to refresh the page. So again, this should also
be blocked by the RASP because we haven't provided
the secret password for the bypass to work. And the secret password
is of course, defcon. And when we say defcon, the entire process that I described before will be triggered. And as you can see, the
RASP did not detect it. So that's all for our RASP bypass. I hope you had fun. Just before I hand it over to Sylvain for the detection and
mitigation strategies, I wanted to say that,
unfortunately, we won't have time to talk about the container breakouts that are (indistinct) into the rootkit. However, they have been presented during our Black Hat call this year. So if you are interested,
feel free to check it out. That being said, Sylvain, take it away. - So let's talk about
detection and mitigation. How could we detect and protect ourselves from this type of rootkit? We could do this at different level. First, if a vendor
provided you eBPF programs, you should go through an
audit and an assessment phase of their programs. Some changes that the code has to be GPL. It probably uses some
internal candid symbols, so you can ask for it. What should we be looking for? The program types that are used, but also the eBPF helpers used. So communication. So maps between programs may
indicate the control risk in the case of that the
vendor program is compromised. We developed a tool to
assist in this auditing phase by inspecting the health files
containing eBPF programs, it is able to list the use
entities, programs, and maps, and compare to graph of the
interactions between them. The tool was run on our rootkit
with the following result. We can identify on the
graph that the XDP program has stored information into maps that are also used by some kprobes, which correspond to the
command-and-control capabilities of the rootkits. It is also possible to mitigate at runtime the loading of such
programs by monitoring calls to the eBPF syscall and
logging (indistinct) It would even be possible to
protect the eBPF syscall itself by either (indistinct) syscall to it to need some trusted processes. As the programs inspected
before then and rejected if they contain suspicious patterns or make use of some dangerous helpers. We could also compute and validate the signature of the
programs before (indistinct) exist to the kernel itself. Using TLS everywhere for network traffic also helps mitigating the
risk of a rogue eBPF program that intercept network data. Now if we are not able to block the loading of such a rootkit, how difficult would it be
to detect its presence? Even if it's possible,
though very challenging to write anonymous perfect rootkit, we should concentrate on the
action the rootkit will have to block and lie about the
result of such actions. For instance, our rootkit (indistinct) we are loading kernel modules because such a module
would have the ability to lead the eBPF programs
and the active kprobes. Now let's imagine that we insert a module that executes a specific
action only known to us. The blocking of the module by the rootkit will then be easy to detect. Monitoring the network traffic
at the infrastructure level could that detecting hijacked connection or strange back into our transmission. Our rootkit being far from
complete and far from perfect, it should be relatively easy to detect it. That being said, we hope
it will bring to light the potential and the risk
of such an eBFP base rootkit while presenting some
interesting technique. The code of both the
rootkit and the monitor is available at the
servers, please have a look. Thanks for your attention
and have a great conference.
