Hunting for Post-Exploitation Stage Attacks with Elastic Stack and the MITRE ATT&CK Framework

so today I am going to talk about post exploitation hunting using the mitre attack framework and elasticsearch first ii am i am john Hubbard I gave myself a little bit of an introduction earlier a sock late for GlaxoSmithKline I teach 555 and 511 per sans and I'm a co-author of the new 455 class which is sin design and implementation kind of inspired this talk a little that classes specifically focused on elasticsearch and using it as a sim so a little bit of that little bit of mitre I will talk about how I went through the process of using the miter attack framework and operationalized it in my sock to a great effect so first the modern defense mindset why are we focusing on post exploitation that is because we have to have a presumption of compromise nowadays right we can't delude ourselves to thinking that no one's gonna get into our perimeter it's going to happen people are going to get fished people are going to put in the USB stick they found in the parking lot right this stuff happens but it's not over when that happens there's a lot of stuff that has to churn when the attackers get that initial access before true damage is done so what we're talking about here is putting up a detection oriented defense for stuff that's already got past the perimeter but still has time before big big impacts are caused on our organization so we have to put up a detection oriented defense to try to catch this stuff before it actually affects us we have a hunt team to do that we saw a great presentation on how we can implement hunt teams we're gonna have them go through and looking for evidence of post exploitation under the assumption that people are indeed in the environment there was the question about how do we know if our hunt teams just not finding anything or the environments clean does anyone here know that their environments clean probably not right your hunt teams just not finding it that's the answer in my opinion there's always something right whether or not it's apt or whatever so we're gonna be focusing on post exploitation prevention is ideal detection is a must that's the slogan that we're going for here we have a lot of challenges involved in this how do we collect the logs how are we physically collecting them from the endpoints there's multiple options for that which logs do we need collect how do we parse and enrich them if we can't parse and enrich the logs they're not very useful to us so we have to talk about how we do that and how important that is and what do we actually look for in that mountain of data because we truly do have a mountain of data from numerous different sources so first how do we collect logs I'm interested in elasticsearch primarily because I'm seeing a trend of a lot of companies pick it up maybe not as their primary sim sometimes yes sometimes no it can be a secondary sim but it is a search engine if you're not familiar with it and if you think about how fast Google can search I don't know how many terabytes like petabytes whatever it is tons of information very fast that is what we want our sim to do so it's an interesting package to kind of supplement your sim or actually be your sin for those who aren't familiar with it the basic thing you need to know is takes your logs you put them in the log stash lock slash pre parses and enriches those logs in any way that you want and then it puts every field into an indexed elasticsearch storage mechanism right any single field you can search equally fast this is different than normal Sims so it can search things very very quickly of course with the caveat that you have to do the work on the front end first of parsing and enriching everything that you want now you don't have to have this as your primary set up most people don't some people are going in that direction but this is one thing that we do see a lot we have the compliance sim the coffee-break sim is Justin called it who here is sat for more than a minute waiting on their results for their sim right but what about more than an hour yeah what about more than a day yep still hands right that's insane speed is critical for incident response and elasticsearch excels at that so even if you're just supplementing what you have currently it can be an interesting ad because it is free to do and you can put log stash in front in your log pipeline and have it just kind of why via you know logs out split them between the two options you can have an ad enrichment and do things and have a tactical sim and the compliance sim coffee breaks and whatever you currently have it can be a free upgrade to your capabilities as a sim yes it's not a scimitar box but it can collect logs parse search and add a lot of value to your team I can do these things very very quickly it works at the speed of analysis so if you're thinking of a question it can answer most searches in seconds or less it is very very quick we just went through an RFP with a number of Sims and we did a bunch of tests and it would take minutes for certain queries to finish that literally took one second in elasticsearch so in that respect it is actually an enabler of something that you may not be able to do currently ok so which log should we collect this is where visibility is the most important for post exploitation if you need to find someone moving around inside your environment you have to obviously have a good view of your endpoints now we all know the log channels the security system and the application logs but how many here are pulling in PowerShell logs not as many hands right think about how many attacks are going through PowerShell nowadays if you don't have PowerShell logs you were missing a large portion of possible attacks and the big trend in attacking right now PowerShell Empire and a lot of in memory stuff is completely PowerShell and if you don't have these logs you're going to miss that stuff sis Mon we talked a lot about system on that one's super important you need to have a filtered set up for that we talked about Swift on Security's version of that great place to start auto run items that's an interesting one anyone have Auto runs in their sin how do you do that yeah so a couple people have it what if you were to run the auto runs program the sysinternals tool that enumerates your endpoints output what it finds in a CSV format have this done as a scheduled task dip it with the day before and then send in with your log agent pick up that file send it into your sim now you know anytime in autoruns changes on one of your endpoints and you can actually collect that and look for the weird things that stick out or just collect all of them in general some of that stuff is super important as an endpoint log to pick up a plotter of course any time there's a weird binary can help you find that stuff to pick it up we have agents beets is the agent that works best with elastic search as a particular sim specific solution but there's also other things NX log and snare every sim has their own thing we have a win collect and Splunk universal forwarders but there's other ways of doing it that don't involve an agent for the sim specifically there's windows event forwarding a lot of people cannot install agents so if you don't have the ability to install a specific agent know that that is available if you google it you'll find out how to do it built into Windows kind of the same with Linux we have our syslog it's just log ng that can send in those logs for you for cloud environments we have scripts and api's that needed to be called but this information is crucial to have from the endpoint because there's a lot of things that you can see from the endpoint that you can't see from the network like what type of process is actually making network traffic things before they're encrypted things like that to catch post exploitation where we're primarily looking for in our host base logs is authentication traffic who's authenticating from where to where what account are they're using that's the super important stuff because as someone gets into the network from the perimeter they're going to start at one point and then they're gonna have to pivot around so if you can't see that you're gonna be very blind to a lot of attacks sis Mon helps with a lot of the network traffic from the endpoints process creation logs with the command line you need to have the arguments and the command line every time a process is started because a lot of the stuff that's working in memory only is going to rely on you having that to detect it process creation auditing can be done with system on or via built-in windows options so there's a 46 88 is the event ID you can turn it on Advanced auditing in Windows is available Auto runs scripted I mentioned PowerShell logs there's three different ways to do that fireEye has a great article called visible greater visibility through PowerShell logging if you're interested in looking up how to do that but the short story there is you can actually get a transcript of every single thing that's typed in a PowerShell session so if an adversary is in there using PowerShell Empire you'll have an exact copy as if you're looking over their shoulder and there's two other versions of logs that you can get to that go in even more detail than that then there's the network service side so we need to have our host logs but we also need to see the network and we need to see the network not only at the perimeter but also on the inside because as they pivot around right maybe you catch them on the perimeter and that's great but a lot of times you won't and at that point now we need to see as they start to privilege escalate from their user to an admin to a domain admin to move around on your servers that stuff is absolutely crucial to see some of that stuff we highly recommend bro-bro makes it very easy to get standardized logs for every version of software that you're running so if you're running iis and apache nginx all of those can be logged in one standard format from a centralized location with bro it makes it very very easy to pick stuff up you've never looked into bro I highly recommend you do that SSL Certificates it can also track you do not need to be able to decrypt SSL to detect badness that is encrypted if you have the information outside of a certificate that it's not decrypted you can still see the domains that something is talking to and a lot of times the fields in a certificate are filled in in a way that is very randomized and obvious that it's bad so if you are not pulling out certificate information you're missing information bro can help you do that for post exploitation specific evidence what we're looking for here is command and control traffic the repetitive activity going to domains that maybe you don't recognize or you know odd amounts of posting or getting or whatever it may be but more importantly the unexpected internal to internal traffic so you have those subnets for maybe your HR your finance and your engineering department if you can't see someone moved from any given department to another one which would normally be not done but in an attack scenario is you're not going to be able to find that so you have the person who originally gets infected moving all around if those things aren't logged as they move around now you've handicapped yourself right you can't see what's going on until the last second when they're now exfiltrating data out so that internal visibility from inside to inside is going to be absolutely crucial for this password spraying guessing and brute-forcing all that's going to fall under network traffic that you can easily see if you have it logged in correctly internal firewall denies if you have outbound firewall deny that's another one that you can see when they try to exfiltrate any kind of data don't let the firewall it can be used as a detection logstash is an interesting log aggregator how many of us are used to a syn that just takes in the logs and pushes them into the storage right and you do the parsing later long stash has to do the parsing at that time you can use it to actually enrich logs in arbitrary ways so you can run code you can have it do lookups you can have it pull in when a domain was created you can have it pull in Whois information of any given particular type what the Alexa rank of that domain is and then that gets put into the syn so now in your sim from the start you can do searches like show me all the traffic that is to a domain that's not in the Alexa top 1 million or whatever you want show me the domains that were contacted that were created in the last 30 days there's a lot of options here the enrichment with this is very very powerful and it can go in front of anything it's not specific to elastic so think of log stash as an arbitrary log manipulation and enrichment engine of sorts it's very easy to do this stuff the most important point of that section it's all about the high value tactical logs right a lot of people have to collect logs of various sorts for compliance Sims it slows down the search so if we have the dual stack sim or we have some way to enrich the logs and search them very very quickly we can put a subset of the logs that we collect the most important for tactical detection into that sim and it makes it very fast and easy to search through it and collect that data for post exploitation detection ok so what do we look for this brings us to the miter attack framework it's been mentioned a number of times in this conference but what I want to talk about in this talk is operationalizing this this is something that we went through in my sock and we had great results management was very happy we were able to improve our ability to detect post exploitation activity and so it was a big success story for us so I've been trying to spread the word of how we did it I'd give people some ideas to start you know chewing on and maybe implement their own organization so what is the minor attack framework it is a host compromised behavior list it is a list of ways tactics and techniques that adversaries can use for post exploitation activity and therefore it can be used as a list of stuff that you need to be able to detect it comes in multiple parts they actually have added what they call pre attack now which is the pre exploitation stuff they're going to merge it back into one big matrix now but it's for Windows Mac Linux Mobile there's a lot of different options for what ever you're interested in so to be clear here we're talking about on the kill chain everything after exploit so controlling executing maintaining this is going to be how do they maintain persistence how are they doing credential dumping credential access command and control exfiltration all known ways that are commonly used by attackers to do all of these items it is an awesome list of this stuff and a bunch of information about it if you've never looked at this matrix here's what it looks like we have across the top the tactics so the high level stuff the what is going on right is it persistence is it privilege escalation and then under those the columns are the actual techniques and ways of achieving those things notice some of these span across multiple tactics just because it can be used for more than one thing so tactics across the top that's what we have there this is the what the techniques below it accomplished is it giving us persistence is it giving us defense evasion and then underneath it the techniques for doing that thing so each column all known ways of accomplishing that tactic this is constantly updated so anytime someone's debating online with later you know I think I found a new way of doing it they'll eventually be like yep that is a new way we'll add it as a new technique so this is an industry validated constantly up to date list of techniques that you know you should be able to detect now as I mentioned they can belong to more than one category at once so for example the what persistence the how a boot kit the what privilege escalation the how you AC bypass we're making this awesome list of how to do all this stuff why is it important someone mentioned the pyramid of pain right this is going straight for the top we know adversaries can very easily change hash values IP addresses but if you go to the top and say you can no longer run scripts with long command lines right or something like that that really really takes out a lot of the capability it slows them down it gives you time to catch them in your environment before serious damages cost and that's the mode important part here this is a blue team checklist that's what I want to push here if nothing else realize that you should be able to detect all of the things on this list right yourself some high-level analytics for these particular items and you will be in a very very strong position as a blue team to detect all this stuff that adversaries are doing for us now this is the most dangerous stuff to miss and I say that because the post exploitation part right you're that much closer to disaster so if you can't catch these things you're very likely to have that attack proceed all the way to the end where you're going to end up on the front page of Krebs or something like that right you have to find this stuff so if nothing else use this as a checklist look at the list we started we said okay we have this whole list of stuff what can we actually detect are we right about that we don't know we haven't tested it so let's you know mark where we think we are and then let's take what we think poses the most risk and say okay these are the ones we want to focus on because we also can't catch them and you know do these two things overlap of course we found ourselves like this right so we're like oh man there's a lot of stuff we need to cover here so we go through this exercise and through the course of a year we were able to bring ourselves from this situation to something like this right we felt a lot better about ourselves by the end of the year and we were able to prove it so how did we do that what we did is we quantified all of our detection levels for this and the next slide has how we did that but what we did is we wrote all these new analytics for these new techniques and we tracked our progress what we did is we use the ticketing system in our case specifically we use JIRA but anything that can make a parent child ticket relationship we made a parent tickets of all of the tactics and then we made child tickets for all the techniques and children of them these specific use cases that could collect hurt that could detect those specific conditions so now we have a way of running a report that says these rules can catch this technique which falls under this particular tactic and with that information now we know where our weak spots are and we can constantly keep track of that as well any time we improve that use case now we have a comment system so we can say oh I change this I change that we can add the custom fields that you need to be able to track this stuff and it makes it very easy to figure out how you're doing along the way so we made this a year-long project let's see how far we can get with all of these things and we were able to quantify it that was the big part for management they were always like you know well how do you know you're doing well well in this case now we have a number so what we did is we quantified our detection capabilities like this we had no detection locally logged centrally logged enriched and correlated stuff something that appeared in a report but wasn't actively alerted on and then we have like test alerts and high fidelity alerts so he just kind of came up with this arbitrary point system and then we tried to take it from where we were at the start to where we were at the end added up the points gave ourselves a goal and tried to achieve it by the end of the year management loves this because we were able to actually say with some objectivity how much did we improve and now we can say for this tactic we had 10% coverage now we have 90% coverage this was a really big help for us for expressing how we were actually adding value attack Navigator is a way to colorize this and visualize all the specific techniques one cool thing about this I wanted to call out Dave and Tim from Splunk or actually orchestrating this this is really cool where you can click on one of these things it will automatically run a test and then have the logs come back to spawn and tell you if it's actually detecting this or not so that's ultimately where I want to say we should take this is we need to automate these kind of tests the way they're doing it is the atomic red team I call that out here if you don't know how to run some of these things to check it read canary thanks guys for putting all this stuff together unit tests for these specific techniques you can execute the tests collect the evidence prove that you can find it or you can't find it so it's not well I think the use case will work but I'm not sure now you know you can visualize this with red team purple team exercises there's another program called vector the website is vector dot IO and we have used this with success for a purple team this is a free tool it's made by us are a security risk associates what it is meant to do is track purple team activities over time and say how successful you were and blocking detecting or if you did see a particular type of attack you sit there with the red team and blue team together red team launches an attack they say I just launched this attack you record the result and then you have multiple snapshots over time of how your blue team is actually doing it's been very very effective for us once you can get the unit test thing down then you can actually graduate up to full automated adversary emulation when you get up to that level caldera from mightor can do that and uber also released one of their tools that they use for this called meta so there's a lot of open source tools out there for doing this so I've said hey go write a bunch of use cases that's gonna take a lot of time right this is the situation we find ourselves in I was like hey guys there's like a hundred things here go write some rules and we have a bunch of new people right and so they're like well we don't really know how to write rules specifically because there's so many edge cases with the sim etc help us out here how can we we want to contribute but we don't have the ability this is where Sigma comes in how many people here have heard of Sigma good so I'm gonna keep talking about this until I will it into existence the blue team needs this right I'm gonna keep saying it because this is really really cool this is a project to make a generic format for writing sim rules so it's a gamble format it's easy to read and what it can do it's the way they describe it is Sigma is two logs what's noir is to network traffic and Yara is two files so if you come from one company you can write snort rules you can probably write some rules in another company but sim rules not necessarily write much more complicated you don't know what fields are named you don't know what type of maybe you have a totally different sim and you don't know how it works Sigma's goal is to get rid of that so you write a specific animal file and it can work anywhere it converts it to whatever sim and then it takes your specific implementation of that sim and also it still makes it work so after you set it up anyone with any level of talent in terms of knowledge with your sim and otherwise should be able to write a use case for it that is the dream here here's an example rule of it fairly easy to read on the right side here we have log source windows system on and detection selection event ID one so this is a process creation log the title of this rule is this macro starts command so what we're looking for here the parent image to be windward XE or Excel XE and then the image to be started by that to be cmd.exe so it's pretty easy to read you don't have to know anything about what your sim actually calls these fields you can implement this as a use case so it's really cool it takes someone to setup it doesn't support all sims right now it's supports Splunk and elastic I think one company has made their own version to making it work with QRadar and I'm not sure where it goes past that but people are constantly developing this so this is really cool it gives you a list of text-based use cases that you can use for github or whatever it comes with a lot of these out of the box in the package so really cool way of doing that okay examples so I set up an elastic search cluster all free tools just did the collection I've been talking about in this particular talk and so here's what it would look like in some of the logs we're starting off at post exploitation so someone has clicked on a macro right this happens all the time every company all day most just word doc runs what happens next so post exploitation adversaries want persistence because they don't want to have to trick someone into clicking a malicious link over and over again or opening a file and saying yes I want to run the macro it's not dependable to do that so they try to establish persistence that's a lot of ways to do that new services scheduled tasks and registry run Keys all of these are very common this is all stuff that would easily be caught by all the logs that I was talking about so the stuff in the system log the stuff in the security log anything from system on it would catch all of this if you have all of the information for startup items across your entire set of desktops if you do something like long tail analysis the odd stuff is going to stick out the more data you have the more easy it is to find the weird things and the people who are infected are going to have the weird things you will find this stuff so we do that kind of search in Cabana of NID 13 we look in current version run key there's only going to be one that has something that looks ugly like this random name VBS script we find the persistence we can go from there so what happens next persistence is there to run something so tactic here is execution and maybe you have AppLocker on that's great but what is still risky to you you look through the attack techniques and you find run DLL thirty-two register over 32 and PowerShell and scripting these are all whitelist bypasses so it helps you align what actually works against your organization to what can actually be something you can write a rule for so we start collecting process creation logs PowerShell logs this helps us catch what is left over even though we have white listing how are we still getting attacked this will highlight that and identify it so system on logs app Locker rules eight thousand to eight thousand three eight thousand four even if you don't have a pluck returned on putting it an audit mode can say this thing would have been blocked but you don't have it in enforced mode so just so you know weird thing ran still a great detection right so in this case they were using these scripting and PowerShell techniques we have a VBS script that was run here we would have evidence of that and PowerShell as well PowerShell with the command line that has a super long and coded command in it if you don't have command line arguments you will never find this if you do have command line arguments it's super super easy to find a long command line right so we can collect that for free but that in the sim that's easy run dll 32 here's another one this would be a whitelist bypassed potentially what we can see here is two different ways of catching that someone was using run DLL 32 to bypass whitelisting running Metasploit DOL eight thousand three caught with the audit mode app Locker as well as these system on logs up above there then we have powershell as a technique it was reaching out to the internet that's weird if you put on the correct filters for a system on it will tell you when something that isn't expected to contact the internet or use the network at all is doing that so we can use that very easily to catch this and identify it 17 connections on the network let's see what was going on there sure enough the event data command line parameter their power show web client download string and then a shortened URL what was it it was probably mini cats because if you see the semicolon invoke mimikatz dumb cred right so now someone is stealing a password we know it because we have the logs to prove it okay so they have a password what were they doing with that password we have our internal network visibility and we have our host lock so we can look at this where are all the failed logins occurring we see a spike in them the yellow spike shows okay they were probably spraying these around and trying to figure out where else this particular credential worked eventually they were able to find some stuff because we see some successful logins after that great easy dashboard to make up visually identifies when password spraying password guessing something like that is in effect easy for you to collect so what happened there we see that they actually were able to log in on something called admin PC we know this because command lines were being recorded here so every net use admin PCC dollar sign user administrator it was a shared administrator password so they logged in over the network and they were able to probably stage that Metasploit dll on the administrators PC they used PS exec to actually run it same exact thing whitelist by pass so they were able to compromise an admin PC now we have proof of that in a number of ways then what happens finally there on that admin PC they find a file called customer data we were able to prove that Metasploit dll was run we have the Apple locker logout up at the top there searching for that we see they ran the tree command to enumerate all the files on that administrators PC and then they conceivably found customer data text which was interesting to them they found 7-zip on the computer this is a very common technique they used that to create an archive called photos 7-zip customer data text was then put into that archive and encrypted with the password of seven eight nine zero UI you know it's to walk across the keyboard afterwards curl was used the administrator had grow because administrators like having tools like that limited rate upload 150 K to my data Dropbox top is so we can prove that this was done now I point this out specifically because if you only have network logs you would see that go across the network and you would say Oh someone's uploading photos dot 7z to my data drop box stop it if you catch it maybe you can carve that out but you don't know what's in that file because it's encrypted if you don't have that password in the command line log all you would know is something was stolen but you wouldn't know what was stolen so it's important that you have both of those pieces one to carve out what was sent and two to make sure that you can actually get into it and figure out what was actually there so it's two important pieces of the puzzle how did they do it if you have a graph like this very easy to make with elastic DNS traffic by type a sorted by domain name so that one sticks right out right txt cname records MX all to one domain covert C to up in the corner right there this would visually stick out whether or not it has a dumb name like covert c2 or not but very very easy one to catch bro would cut all this up you would instantly catch all this stuff so this is a very common attack right someone gets their computer compromised shared AB and password moves around the network this is the way that it went down all of the particular techniques that happened we have persistence VBS with auto start and the registry was caught the PowerShell scripting was caught for execution the download and meterpreter AppLocker there I'm a white listing bypass rather I was caught we have evidence of that they were able to move across the network they were able to download and run baby cats we see that that happened lateral movement where they scanned they got on to the admin PC staged the same DLL ran it we see all that the exfiltrated photos dot 7z we see the password for it so we can recover it and decrypt it and see what was stolen we have the exact typing of what they did like we're looking over their shoulder so we have that covered and the command and control channel of DNS tunneling obviously would very well stick out as well so we've got all of this stuff in every step using all the tactics from the mitre attack techniques this is very very common progress of a pen test or reveal attack so this is very very effective and shows that you can really get a lot of awesome coverage out of this and you can do it for free which commands are used most if you haven't read this paper I like to point this one out Japan cert wrote it's called commands abused by attackers attackers love living off the land so it is a list of the actual commands that they've seen run and the frequency that they saw them in actual incidents this is the stuff that you can put in to watch for in your actual command line logs and it helps you pick this stuff out so the review here we have high value host and network data that we need to collect we need both pieces of it otherwise we won't be able to fully solve the puzzle of what happened and we also won't see that internal to internal pivoting that goes on it'll only be once it's too late so we need to catch that stuff and that's how we do it we use the mitre attack techniques as an analytic guideline what we need to cover we talked about grading those things how well can we catch them now making a goal for ourselves with a number how we can quantify that to management and prove that we're actually improving over time and we caught all of the stages of a typical attack and multiple ways by doing this collected it all for free takeaways here if you're interested in doing this if we do have the SEK 455 class the new one for elasticsearch if you're interested in doing it but aside from the elasticsearch part i highly highly recommend you to do this sort of activity with the mitre attack framework i rank yourself set a goal quantify it and then you can parade that around as a success for your blue team it's a very effective way and it sends a strong message that you are doing what you should be doing and it's based on real industry validated knowledge doing this gives you an outstanding incredible detection superpower all right all right [Applause]

Channel: SANS Institute

Views: 18,264

Rating: 4.9502487 out of 5

Keywords: sans institute, information security, cyber security, cybersecurity, information security training, cybersecurity training, cyber security training, ATT&CK, ATT&CK framework, threat hunting, Security Operations Center, Security Operations Summit, SANS Summit, Cybersecurity Summit

Id: PdCQChYrxXg

Length: 32min 38sec (1958 seconds)

Published: Tue Sep 25 2018