Hunting Webshells: Tracking TwoFace - SANS Threat Hunting Summit 2018

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
(angelic synthesizer pad fades in) (low-pitched hum crashes and then fades out) (audience applauds) - So I'm Josh Bryant, like you said, I'm a Director of Technical Account Management at Tanium. - And I'm Robert Falcone, I'm a Threat Researcher over at Palo Alto networks on Unit 42. - And today we're gonna talk about web shells. Why web shells? Well, this is what U.S. Cert had to say about it a few years ago. They've been around a really long time, probably since the creation of the web, and the tactics are just getting better and better and we're finding more and more of 'em. - Yeah, so, this is what we're gonna be talking about today, is these web shells, okay? There's so many out there, it's unbelievable. So you can go to Github repositories and find hundreds of these just easily accessible. And they range dramatically in capabilities. But the reason why they're really important is because of the access that they provide to the web server. Whether it be running commands specifically, uploading, downloading files, that kind of thing. And like I said, they're very easy to obtain, so in a lot of cases, actors don't have to build their own. So, the one web shell that we're gonna be talking about today is not one of those web shells that you can freely grab off the Internet. This is a custom-built web shell that an adversary was using, they developed it to be used in their campaigns in the Middle East. And me and my colleague named it TwoFace, because of the unique design which they used to develop it. - And as Rob was mentioning, this is the one that I showed at this very conference last year, however, at the time, I couldn't find a name for it. I could've sworn, I can't possibly be the first one to come across this, still probably wasn't, we just happened to be doing the same research from a couple different angles at the same time. And we're able to come together to kind of update this and give you guys some new information. - Yep, so I named it. It's a silly name, but it was named because there's two layers to it, okay? The first layer is a loader. It does, its whole big responsibility is to install the second layer, which is the TwoFace payload. And we're gonna talk about this in detail going through this talk. So, what's the loader? Like I said, its purpose is to install the second layer, which is the payload. But, what they did was, the actors actually took their loader code and appended it to legitimate content on that web server. In this case, I chose one that was a news article on an IIS server, but what Josh and I are gonna talk about throughout this presentation, they also appended it to error messages on Exchange servers. So they're really focused on Exchange servers. Okay, so the loader, the actor, when they wanna start interacting with the web shell, they send a specific POST request to the loader, which installs the payload, that second layer I just mentioned. As you can see on this slide, this is where all the capability exists for this payload, for this web shell. It's pretty capable, upload, download files, execute commands, timestomp files. You can even run SQL queries on remote SQL servers. But one thing I should focus on here is that you have to authenticate to this web shell to actually be able to use this thing. So you can't just find this web shell and start interacting with it without that password, which is really important. So, how does the actor actually interact with this web shell? How did they construct the POST request to the loader to install the payload? Well, the actor will provide a 64-byte key, which is fairly massive, right? The loader will actually manipulate that key with a key salt, so that's pretty clever, and then the result of that is gonna be used to decrypt the embedded payload web shell, and then that's saved to the web server. And, so that's the whole process. And I felt like that, you really need to see it to believe what that process is. So I made a demo video, okay? Like I said, we're gonna talk about Exchange primarily, but they also had one sample that I'm gonna show you right now that ran specifically on IIS. So for simplicity, I just chose that sample, and I set up an IIS server just to show you how it works. - Yeah, it's important to note that IIS is a dependency of Exchange, so there's not too much of a difference here. - Yep, so here I'm just showing you in this video, I just set up a simple web server, and I'm showing you that this web server, the only thing that's hosted on it, is the TwoFace loader that I named TwoFace.aspx, right there, okay? We're gonna hop over into the browser and just browse to it, okay? And what you're gonna see is something very similar to what you saw on the slides there, you're gonna see a news article that was meant to be on that web server. There it is. But, if the actor wanted to actually use the web shell, they're gonna have to instruct the loader to install the payload, so I wrote a script to be able to do that myself. So here's the key that's required to install this particular payload. So I'm gonna run my script, which is gonna craft the correct HTTP request, which is gonna send the key over with the filename to drop it to. There it goes. The TwoFace loader responded back, saying, yep, I installed the payload. And we're gonna hop back over to the IIS server. I'm gonna show you that this new file called drop.aspx is gonna be on the IIS server as well. There it is right there. So if we navigate to that in the browser, that's where you're gonna see the functional web shell for TwoFace right here. It's a little bit different than the one you saw in the slides, but very similar variation. So right now I'm trying to just run a command, whoami, in this web shell. It didn't work, 'cause I had to authenticate. You can see that that bar at the top is red, and when I authenticated it turned green. And now I run the whoami command and it worked perfect. So we're gonna talk about TwoFace in-depth coming up, and exactly what the actor was running command-wise. So when you see us talking about running commands and doing things using TwoFace, think of the actor actually interacting with something like this, okay? So, from a hunting perspective, how did we go about hunting TwoFace and the web shells themselves? As a threat researcher, I rely a lot on YARA rules and other kind of detection signatures, so we had to write them, right? But the whole purpose of the two layers for TwoFace was that the loader layer's very obfuscated. It's difficult to detect. So, what did we have to do? We had to figure out how to detect the payload that's embedded within the loader. So, how did we do that? Well, we took a look at the actual decryption mechanism within the loader, and we found, this is how they applied the key salt to the 64-byte key, and then they used that resulting value as the key to decrypt the embedded payload. And I don't know if there's anyone in here that studies ciphers, but this is probably the weakest cipher in the history of cryptography. They're using addition and subtraction, so we can easily just reverse that, and we wrote a simple script to do a crypto attack on it, and we were able to find the 64-byte key, and then we were able to write some YARA rules for the payload, because we now have the cleartext, okay? So, from a threat researcher, that's how I went about digging through our content, by writing YARA signatures, or YARA rules, to hunt for these payloads. - Alright, now that we have seen a little bit about what web shells are, how they work, and how this attacker was using it, it's time to go hunting. And if you've maybe been living under a rock for the past few years, or haven't gone to a few other conferences, you may have missed the fact that attackers think in graphs. When I go hunting, I wanna think like a bad guy, so I need to think in graphs to be successful in my hunt. So let's look at how an attacker might get the web shell on to the target, on to the Exchange server. So here I've created a couple different attack graph scenarios. The first one I call here is the Attacker's Hope. This is how the attacker wishes it worked. It's plausible but not likely, it may also be how your management might think this type of attack works. So here our admin uses Domain Admin everywhere, he's never heard of credential hygiene. He just logs in with his Domain Admin cred, so everybody gets in, in fact if you all look under your seats right now, you've all got Domain Admins. So now our attacker is going to attempt to directly compromise the Exchange server. He's gonna try a few different techniques. This is standard for implanting any kind of web shell on any server, however, these techniques tend not to work on Exchange servers. There's no SQL in Exchange, so SQL injection's not gonna work. There might be a remote file inclusion, that's probably not gonna work. You might have some cross-site scripting. Again, plausible, probably not gonna do anything. Might have local file inclusion, they're gonna try it, probably not gonna work in this instance. Our most likely culprit is gonna be remote code execution. I could not find any remote code execution vulnerabilities in Exchange itself, there have been some in the underlying IIS in the past, but if you have a fairly up-to-date system, unlikely here as well. But here they're gonna use the remote code execution, get access to the Exchange server, and because our Domain Admin uses creds everywhere, he's gonna go ahead and run Mimikatz, not a virus, and dump creds and then use that to gain access to the rest of the environment. So now let's look at a more likely scenario. Here our attacker starts by sending an email to their target. Our user here loves email, loves attachments, needs to enable macros, and because they just love to run macros, they go ahead and run it. Now our attacker has gained a foothold into the environment, but they need elevated permissions. This scenario, we're a little bit better at our credential hygienes, we haven't exposed our Domain Admin creds to this endpoint. So it's gonna cause a little havoc, induce a Help Desk call. Our Help Desk just wanting to do good here logs in with their admin credentials on to this box, allowing our attacker to gain access to a file server that the Help Desk just happened to be admin of as well. However, he was not an Exchange admin, so we're kinda going in a roundabout path here to make our way to the Exchange server. Here our attacker finds a misconfiguration in Exchange, he didn't have to exploit any vulnerabilities here. He's able to directly access the file system, and drop the web shell payload on to the Exchange server. So from there, because Exchange is 99% of a Domain Admin, he doesn't even need to bother trying going after that. Remember, Domain Admin is never the goal of the attacker, he's after the data. Exchange is a high-value target, it stores a lot of information in it, and from there he can modify other groups and objects in AD to give himself access to these other sources of data that they're truly after. So some time later, our security team gets a virus alert because one of the tools got uploaded to VirusTotal and we finally got some signatures on it. So our security team comes in, and he just cleans up everything that he got a virus alert on, because they're not really doing any true threat hunting here, they're just doing that reactive stuff where they're just responding to alerts. Now you'll notice that there's one thing on this slide that our security guy didn't find, and it's that web shell hiding on the Exchange server. Which brings us to our last scenario here, which I call Return of the Attacker. Because we missed the web shell on the Exchange server, our attacker can immediately come back in, and now he's a little mad and wants better access than he had before, so he goes after the Domain Admin creds this time and regains complete control of the environment. - So, Josh gave some really good scenarios there with the attack graphs. I'm gonna actually talk about a little bit of the commands that we saw during my team's research efforts on this, in regards to what the adversary did with their access to that network. So, I really, really wanted to make a pew-pew map, so we're gonna start talking about, geographically, where the actors were carrying out these activities from. And it just gives a little bit of a high-level insight into what they were doing. So, going back to June 16, 2016- - Pew! - I know this is, I know this is timely and pertinent information, but back on June 16, we saw the first activity from this specific adversary on this network. And what they did was, you can see that little orange server down in the bottom-right corner here. That is our Exchange server, okay? And what the adversary did on June 16 was, they actually used legitimate credentials to log in to Outlook, specifically OWA. And they actually used an IP address from Iran, from all places. - So this kinda lines up with the scenarios that we're just going to. Obviously they had already obtained credentials into the environment, they weren't exploiting something on the Exchange server first, they're using what they got to get in. - Yup. The next day, they also logged into the Exchange server from the same IP address, and then the next day, on June 18, that's when we start seeing this adversary using the TwoFace web shell to interact with the Exchange server specifically. Fast-forward a couple months later in September, we saw this adversary log in, I'm sorry, use TwoFace to run six commands from an IP address in France. And then again, from France, running 16 commands in March. And then, I really wish we had pew-pew noises. - Pew-pew! - 61 commands they ran in April, and then finally in May 2017, we saw the last activity, which was 10 commands run from an IP address in Germany. So you can see that this threat group has at least the foresight to know that some sort of forensic analyst is gonna see where all these activities are coming from, and to try to cloak where they're originating from. So, let's kinda dig into each one of those dotted lines there, what they did with their access, okay? So, like I said, on June 16, they used legitimate credentials to log into OWA. So what did they do? So they immediately opened up the address book, and they looked through 15 different distribution lists, and over the course of a couple hours they looked through 81 different user accounts, okay? So that's pretty interesting, why did they do that? Well, what they did was, they used one of those distribution lists that at this organization was called Network Support, and they sent an email to them. The subject was Network Team Support. Now, we don't know what the attachment was, and we don't know what the content of the message was, but if you think about that distribution list and the people that are probably on that distribution list, they probably have some pretty decent access to the rest of the network. - [Josh] What better way than to phish your target by making the email come from with inside the domain? - [Robert] Yep, pretty clever. So unfortunately, we don't know how they did it, but that is likely what they did right here. So, I said again they logged into the Exchange server the next day, but this time they didn't do anything with emails or anything like that. They actually went and downloaded what they called the offline address book from Outlook. So they used Outlook to just download this file that contains the address book, probably for some offline processing or data retention, so they don't have to keep interacting with the Exchange server to look through these- - [Josh] Lot of valuable information they can get out of here, so it's gonna have not only the email addresses of everybody in the organization, it's also gonna have any external contacts they had stored in there. But phone numbers, addresses, office locations, and possibly some of their organization structure. They can see who's a manager of who depending on what attributes they've populated on these accounts. - [Robert] Yeah, and imagine doing data exfiltration as an actor using something like Outlook, that's pretty awesome. So, the next day is when we saw the first activity using this web shell, okay? So, remember I said that there was a loader and a payload layer to TwoFace. In the top-left corner of this slide you can see this URL, error2.aspx is the file that they're interacting with. - [Josh] That's a legitimate page in Exchange, that's the actual error page that Exchange 2010 uses. So they're appending the code to a legitimate file on the server to act as their loader. - [Robert] Yep, and then what they're doing with that loader is they're using it to install the payload at error3.aspx, and, I mean, if there's an error2.aspx that's legitimate, there's probably an error3.aspx legitimate file as well. - [Josh] Right, if you're not paying close attention, this is something that you might miss. Especially if you're not familiar with the application itself and what's supposed to be there. - [Robert] Error3 doesn't exist, does it? - It does not exist. - Okay, good. So, with this access, they run one command, and one command alone, and it's this whoami command. So we actually think that they just did this to see what account they had access to, or if the TwoFace payload- - Fun fact, that account would be system. - Yes, it is definitely system. - 'Cause this runs in the system contacts. - [Robert] So, now, they know that their web shell is successfully deployed, and they don't do anything else, they just kind of, they don't do anything. So fast forward about two months, two and a half months, we start seeing this adversary use the web shell to do some more interesting stuff, okay? So the first that they do is they upload this executable, m64.exe, using the web shell, and then they run it with very specific parameters. Now, if you look at those parameters, you can probably tell what this is. This is Mimikatz. - [Josh] Yep, and if you've never seen somebody run Mimikatz on an Exchange server specifically, if run and running OWA, it's pretty much like cracking open a credential pinata. There's credentials spilling out all over the place. - [Robert] Yep, so, and what they did was they outputted this information into a text file, 01.txt, and then they exfiltrate that data using the web shell by using the type command. And in Windows, that just displays the contents of a file, they copy-and-paste it from their web shell to their own system, and they've exfiltrated the credentials, so it's pretty clever. Then they clean up after themselves by just deleting the text file, the executable, and the TwoFace payload from the server to clean up their tracks. Okay, jump six months into the future to March 3, 2017. This is where we start seeing them doing some pivoting activity using the web shell. So the first thing that they do is they run this net group command looking for Exchange Trusted Subsystem. - [Josh] And that's a really powerful group. That is what makes an Exchange server pretty much 99% of a Domain Admin, that group has rights to do almost anything in Active Directory. It typically is only gonna have the Exchange server computer objects as a member. If you see anything else besides an Exchange server computer object in that group, that means something's wrong. Sometimes organizations will put people in there by mistake, but definitely worth investigate. - So what that command does, is it gives you a list of all of the hostnames for Exchange servers on the network, okay? So what the actor did after that was they just started trying to find where Outlook's content is being hosted for OWA and those kind of services on these remote Exchange servers. - You can kind of tell that they knew exactly what they were looking for here, because they're enumerating a very specific path. This indicates that their target was Exchange 2010. - Yep, and so what they did was, when they found that path, they uploaded another web shell to the current web shell called Exchange.aspx. Then what they did was, they just timestomped it to match the same timestamps as a legitimate file in Exchange. And this one specifically was Exchange.asmx. - Yeah, asmx is a legit file, that's where the Exchange web service is, kind of like an API interface for Exchange. And, pretty clever, it's one letter off in the extension, so again, something you might miss, especially if you're just looking through some log files. - Yep, so, after they timestomped it, what they did was that's when they started doing their lateral movement. They used the copy command, that's all they did, they just copied that new web shell over to the other Exchange servers that were, which were discovered using that net group exchange Trusted Subsystem. - Yep, and that Exchange Trusted Subsystem has to be a local administrator on every Exchange server in order for Exchange to function, so that's how they're very easily able to laterally move between all the Exchange servers in the environment. - Yep. Finally, before they ended their activity here in March, they just delete the payload, the TwoFace payload, and they move on. They come back a couple months later in April, and what we think happened was, they lost access to one of the web shells that's hosted on the Exchange server. And what they did was, they came back, they ran that same net group Exchange Trusted Subsystem command to find all of the Exchange servers on the network, and then they run the hostname command to figure out where they're at. Which one of those Exchange servers that they're currently interacting with. - [Josh] Yeah, this could've been because of like we showed in the scenario earlier, that maybe our security personnel at the target organization found something and cleaned up the original web shell, but more than likely it could have been just that they had to replace the server because they had a hardware malfunction or something else that caused them to replace the Exchange server. - [Robert] Yep, so, after that what they start doing is they start trying to list the contents of that specific path within Exchange on the other four Exchange servers on the network. And it's telling that they probably lost access to one of those with these commands. They immediately try to ping server number three, and we don't see the output of this obviously, but it's highly likely that that didn't succeed. 'Cause then, the next thing they do, is they try to list the contents or try to access the C: drive and D: drive on that server, and they weren't successful, okay? So, what did the actor do in response to this? They uploaded yet another web shell to this web shell, and this global.aspx that they uploaded was just another variant of TwoFace. And they used the attrib command to hide it, and then they copied it over to all of the Exchange servers they still had access to. - [Josh] That global.aspx is not unique to Exchange, it's actually a part of IIS, it's typically a hidden file that holds configuration information. It's not always present, and it's typically not accessible, it's not served by the web servers. You can't access it through a browser, generally only through a file system. - [Robert] So while they're on the web server, they uploaded another executable, mom.64, which, again, is Mimikatz. They exfiltrated the output by using the type command, and then deleted a bunch of files to clean up after themselves, including the text file, including the executable, and the TwoFace payload web shell itself. Okay, so this is the last activity that we saw, was in May 2017. And we believe we lost visibility on to this because the organization that was impacted started doing remediation, and they actually caught these guys and was able to successfully kick them off their network. But what we saw them do before that happened was, they immediately tried to ping an external IP address, 4.2.2.4, and we think that they were doing that to see if they could establish outbound network connections. And immediately after that, they tried to find Domain Admins on the network by running this net group command. And again, they upload another executable, and this one's MicrosoftUpdate.exe. - [Josh] Yeah, obviously, these are good guys now. They're trying to patch this server for them, because they feel bad for their target. - Yeah, and they're also dumping the credentials while they're doing that. One other thing that I should mention, though, and it may have been kind of lost over, is the names of the Mimikatz tools that they're uploading kind of blend in to other Microsoft executables as well. So they're trying their best to hide while they're doing all this. So they run that executable, and you can see that the parameters change a little bit, but again, this is Mimikatz. They're outputting the credentials to mic.txt, and then exfiltrating it with the type command. So you can start seeing that there's a clear trend happening here. Every time they access the web shell they're dumping credentials to see if they got anything new, and to expand their presence on the network. - So now that we've had our bad guy hat on, let's put our good guy hat on, and take a look at how we would examine this type of stuff on the Exchange server itself. The first thing I need to do is find out where the logs are at. If I'm coming in and I'm not familiar with the Exchange environment where I'm doing my hunt, I can whip up the Exchange management shell here, just a PowerShell module, and run the command here in the first example. That's gonna tell me where all of the Exchange servers are, specifically the ones that have the Client Access role on it which is gonna be what runs OWA. So I know how to focus my hunt. The second command here is gonna show me where the IIS logs are stored. Now, they're generally gonna be in the default location, but this will tell you where they're at if they've been moved, just quick to run and find out. So once I've gathered my logs, I need to look for some indicators within those logs. And our first big giveaway that there's something fishy going on here, is POST operations that have a low RequestCount. There are very few pages on an Exchange server that are gonna have POST, period, and those are gonna have a very high count because you're gonna have normal day-to-day users accessing it on a regular basis, issuing that POST to it. - [Robert] And if you think about from my pew-pew map, there weren't a lot of commands that were being run each time they visited the web shell, so it's likely those would be kind of exposed. I mean, 61 commands is gonna be 61 POST requests, right? And that's probably gonna stand out when you're looking at other legitimate files that are handling POST requests in Exchange, that are gonna have lots and lots and lots and thousands of requests. - [Josh] Now, you might run into a few false positives here and there, because it's case-sensitive when it logs, so if you have somebody that types the URL in in a weird way, they might trigger a POST count for that, but those will be very easy to filter out when you're looking at this here. So the next thing that we want to look at, will help us focus our hunt, is to pay close attention to URIs that don't require authentication. If I'm attacker putting a web shell on any kind of web server here, I don't want to be blocked by having to authenticate before I can load the page. I wanna be able to access it with anonymous so that it comes right up every time. And then finally, you wanna look at GET requests that fail, 404 errors. Now, there's bound to be a few here and there, typically very low on static applications like Exchange. But the reason I have this up here is, last year I was investigating another web shell, and the attackers had actually modified the logging configuration on the Exchange server so that it filtered access to the web shell out of the logs. Now they hid it in that same error2.aspx, which I would expect somebody would run into an error eventually, right? You'd have a legitimate entry in the log for the error page. So what stood out is there were no entries for the error page at all. So I flipped over and looked at the 404 errors, and now, because attackers are humans just like us and make mistakes, I saw the attacker making typos trying to access their web shell. That gave 'em away. So once I've gathered this information, the next thing I wanna look at is the User Agent String. Now, even in very large environments where you have hundreds of thousands of users accessing this all day and night, the attacker's User Agent String tends to stand out. This is because they often use it similar to the 64-byte key that Robert was talking out, it's a access method to the web shell. If I connect and my User Agent String doesn't match the one that I programmed into the shell, I don't see the shell. This prevents hunters, legitimate users, from accidentally stumbling upon the web shell. - [Robert] Yeah, and I can speak to the User Agent specifically to this threat group. They love really old versions of Firefox. Like, many many years old versions of Firefox. So they kinda stand out when you're doing this kind of hunting. - And remember, User Agent Strings are very easily faked. So, pay close attention to those. Once you've gathered your User Agent String, you can use that to start tracking some of the attacker's activity. So here I'm showing, we have the User Agent String that connected to the web shell, and then immediately after, we see the User Agent String accessing OWA itself and the user account that they used to log into OWA. So now I know my attacker is using the credentials that they stole to access this mailbox specifically. Now, there may be a chance that you could mix up some legitimate use, but to make yourself extra certain that you are tracking the attacker activity, you can add in looking at the ClientId field. This is in Exchange 2013 and later. In earlier versions, it was session ID. This ID is a unique GUID that is a server-side representation of the cookie that the client generates, so it's persistent and unique to the session that initiated it. So using that, we'll go in and dig through and find out, okay, what accounts did my attacker access? I know that I need to put those on my remediation list and do something about it. Might also help us track what type of information that they may have taken out of these mailboxes. One other method, I actually introduced this last year, it's a PowerShell script that's basically a wrapper around Jared Atkinson's PowerForensics, comparing the filetime in the master file table. So what this does is it uses the install time on the Active Directory attribute of the Exchange server as a baseline. This is the time that all the files in the install should match. If they don't match, throw me an alert, as you see here, saying that, hey, this file looks like it doesn't belong. It works in 2013 and later only because every update is a full install, so every file will have the same file time except for anything that doesn't belong. And with that, I have one last graph for you. So we use everything that we learned here today. Here our environment is already completely compromised. Our security guy asked for our DFIR help, our DFIR expert comes in and they start their investigation with the Exchange server. Sometimes you go in for a hunt and it might be proactive, you don't know where to start. Exchange is a high-value target, it's a really easy place to start, it's a really easy place to knock off your list. As you can see, you can just filter through some logs, run this script, easily find what you're looking for. So now our DFIR hero comes in, finds those unusual POST operations we were talking about, knows that this looks like a web shell. They go ahead and clean the web shell up and start tracing the activity of the attacker through the environment, cleaning up things and getting the attacker out as we go along. So with that, thank you all. - [Robert] Okay, are we all god? Alright, thanks everyone! (audience applauds) (mysterious piano music) (sibilant white noise crashes and fades out)
Info
Channel: SANS Digital Forensics and Incident Response
Views: 6,238
Rating: 4.9252338 out of 5
Keywords: digital forensics, incident response, threat hunting, cyber threat intelligence, dfir training, dfir, learn digital forensics, learn computer forensics, forensic data, forensics artifacts, free digital forensics, free computer forensics, Threat Hunting, Webshells, yt:cc=on
Id: GjquFKa4afU
Channel Id: undefined
Length: 34min 45sec (2085 seconds)
Published: Wed Feb 13 2019
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.