107 The Need for Pro active Defense and Threat Hunting Within Organizations Andrew Case

Mr Andrew Case is the digital forensic researcher and trainer he has traveled many miles to come here so like to everyone give a big hand Tampa welcome to the man the myth the legend himself Mr Andrew [Applause] Case okay and one more thing there is a happy hour that we are sponsoring at Gator Dockside it is actually half a mile from mosy if you leave the museum and make a right go away from the University should be on your left hand side the happy hour will run from 7:00 to 8:00 so that you bring a friend or bring your thirst and we'll see you [Applause] there that is Cy of our friend ever live [Applause] CL good afternoon everyone I'm going to can you hear off this mic no okay fing it doesn't squeak in my ear how's that better okay so good good afternoon everyone uh hopefully you've had a good day there's been a lot of interesting talks I'm going to be speaking on the need for active threat hunting and some strategies that you can use to bring it into your organization or improve the process if you're already performing it how many people have done thread hunting before in their environment looking for adversaries looking okay how mature would you say your process is is that something you do on your own or is it something the organization does as an ongoing flow is it kind of ad hoc okay so I talked to a few people today that know do IR um it seemed kind of ad hoc um so we're going to talk about some ways uh the technical things you should look for but then also uh kind of how you can get into a real process within your company for those that have not met before my name is Andrew casee and I spend most of my time working on memory forensics incident response and malware analysis one of the core volatility developers I saw several talks today that mention volatility in different ways and I'm also the co-author of the art of memory forensics the book that was given out during the raffle if you haven't heard of the book before or seen it it's over 900 Pages covering memory forensics and malware analysis across Windows Mac and Linux it's essentially a brain dump of everything the volatility developers knew as of 6 months ago so we put a lot of time and effort into it before getting into instant response I spent several years working on the red team side of the house so I was doing reverse engineering for vulnerability Discovery source code audits penetration tests and the normal sort of set of things found this to be very helpful though understanding how attackers broke into systems the type of Maneuvers they would do how malware looked uh how vulnerabilities looked and then moving on to the defensive side was very helpful um I knew what the logs were going to look like I knew where they were going to be stored and I kind of knew how they would manifest in memory if all you've done is offense before I would definitely recommend learning defense and on the same hand if all you've done is defense before learn how to do pen testing learn how to break into systems if you understand what the attackers are doing and how they're going to move in you're definitely going to have a better chance of finding them and stopping them so why is threat hunting so important and why did I choose it as the topic for this presentation I did so as it's currently the most effective method for any organization to combat modern threats we're obviously losing this fight people are compromised all the time attackers are on the network for long periods of time and most companies don't have uh an effective way to deal with that and when you're dealing with adversaries that have large budgets when your nation states your real criminal Enterprises only if you're organized and prepared you have any chance of dealing with it you also have to be active one of the big parts of this talk is that your organization needs an active approach to defense not a reactive state where you're waiting for an alert to trigger or something visible to happen you need to go looking yourself and for those of you who have performed incident response you're quite aware that one of the most difficult aspects is sifting through all of the data you need to take the all of the logs that you have all of the information from the live system and figure out what's normal versus what might be an artifact of an attack and the reason this is so complicated is because of how complex modern systems are if you saw the previous talk if you saw the talk from Ryan earlier when you look at say the list of processes there's hundreds of them when you look at the network connections there's going to be a large amount of them without understanding what we call normal of the system what should be there you really have no way of finding the attackers and then eventually getting them out of your systems and through threat hunting that's how your organization can get there you can have you will have deep examinations of systems you'll know exactly what they're supposed to look like and after that you can have a fighting chance against the attackers they're still going to break in they're still going to be on your systems but you're going to find them in an effective way and then the other thing to realize is that no matter your budget you could have a billion dollars to spend on security tools on forensics tools but there's no combination of tools that can tell you everything you want to know about your network they can maybe alert you to something but if you've seen real attackers in action if you've read the reports they live on the network a long time before they're found and then even if you have some agent or network sensor that's going to tell you there's an attacker here or the attackers moved to the system or they were on a certain system those aren't going to do response for for you eventually a human's going to have to move into the process and actually root those attackers out of the network so if you take someone who doesn't know your environment doesn't know the systems you run the applications you run they're going to be at quite a disadvantage and they're probably going to lose um no one's going to raise their hand anyway if I ask but if you're fighting nation states on a daily basis these people are smart it's their day job to break into systems to stay on network steal as much data as possible you're not dealing with amateurs so if you bring in investigators who don't have that same amount of knowledge uh you're going to keep losing and nothing shows this more than when you look at the statistics about attacks attackers are on networks for months or years before they're noticed and if you look at how most of those organizations eventually figure out that the attackers were there it's by third party third party organizations the FBI notifies them uh a vendor that has access to different TAPS in the internet or other networks they notice that a network is attacked the network and the Network's owner don't figure it out on their own they have to wait for someone else to do it it's inexcusable that attackers on the other side of the world know your networks better than you do or that your IT staff does in some cases I can maybe understand it how these attacks persist for so long you do have very skilled attackers they have advanced ways of exfiltrating data and moving around the network but on a realistic side if they're moving gigabytes of data outside of your network if they've compromised every machine of the thousands on your network and you don't detect it that's not a state you want to be in that's not a good way of dealing with instant response and it's definitely not a good way of attacking the problem I've seen that uh many times firsthand though uh and it's not pretty it looks bad on the IT team it looks bad on really everyone involved in the organization uh we've had a few examples of this one incident I was working uh we were kind of trailing behind the attackers figuring out what machines they move to and eventually we found a machine that they were on and left some files around all the other ones that we um had found they were active they deleted the files so we could see maybe the file names or some remnants of the files but we never had a good set of data but then but then on this particular machine they didn't remove the data they didn't do any anti- forensics and what we essentially found was an export of every vulnerability scanner within that client's Network so the attackers knew every vulnerable machine these scans were very new so most of them were not patched and then that's where they chose to attack and they just wiped through the whole network hitting every vulnerability there was we brought this information to the client and at first they thought we basically did a pen test while trying to do forensics they had no idea how we got that much insight into their Network how we figured out where the attackers move so quickly but it's all we did was follow their steps and when we presented to the client we said this is what was compromised this was what was compromised all the different machines they didn't even know why they had no idea what was important on those machines what the assets were they had Shadow it are people familiar with that term just stuff in your network that you don't even know it is it's not tracked that help some of the company's most sensitive information so it wasn't until we found out what the attackers did we went on the machines and found the data that the company had any idea what was there or why the targeters attackers were targeting it that's definitely not the position we want to be in as defenders in another instance which uh another instance that highlights how bad the situation is the attackers had moved onto a network again this is a different client but the attackers had moved on to the network they then found all of the resource that they wanted to steal and then they disappeared for about 2 weeks again we figure out this time frame by doing IR after but what we saw was that they came back 2 weeks later after moving around the network finding all the resources they cared about and they came back with very targeted malware and they launched the malware it was different samples throughout the network it was launched from specific machines it targeted certain data on certain servers and it pushed it all out across the network so after these attackers came back and launched their custom malware it was about 3 days later and literally every penny of Ip that that company had now sat on CH uh servers in China and again we met we had reverse engineer the malware we had to figure out what machines it was targeting what data it was targeting and we brought this to the client and they had no idea what was going on they had never heard of most of the machines the the DNS names were not familiar to them the IP addresses the subnets were not familiar to them and it wasn't until we pulled dis images and started showing the data that they realized basically every penny that they've ever spent on R&D was now within another organization again that's inexcusable that's uh not how defense in it should be and as we'll talk about uh through threat hunting and a more proactive stance that can be a thing of a of the past and you won't have these problems anymore so with that said what exactly is threat hunting several blog posts cover this concept in detail and the best of which were written by either current or former GE people they do this very very well uh the best two examples which I have referenced here on the slides are from Shawn Mason and Jack crook so these are two of the guys who really built out ge's uh IR process globally and the main idea behind threat hunting is a remove is a move from the reactive state of incident response so waiting for a hips alert or an IDs alert or one of those FBI notifications to proactively looking in your environment understanding your environment knowing the resources that are there and actually going looking for attackers so I can pull memory from a machine I can pull disc from a machine and I don't have a specific indicator that I'm looking for I just want to know is this machine in an abnormal state is it in a state that shows that an attacker is there versus normal running of the system but again this is not how incident response teams traditionally worked it's in the name an instant Response Team means a some type of incident happens and then the team comes in and fixes it but again that reactive State just doesn't work it's failed for a decade now it's not getting any better with with the adversaries getting better uh and IR needs to move into a more proactive state it also needs to become more active within the overall organization if we're going to go through a few slides but the idea is if you look at how security is embedded in an organization versus how the IR the IR team is embedded uh they're quite different and the IR team is always going to be lagging behind and it's kind of by Design if you look at how they fit into the organization uh it's almost hopeless for them to keep up now let's talk about the benefits of threat hunting the first benefit is that again you have that move from a reactive state to a proactive State and what that's going to do is make the job of the attackers on your network much more complicated if you look at skilled adversaries and how they work they know that they can evade every security tool on the market and the only way they're going to get caught is if they make a mistake if they leave patterns across networks that say the FBI picks up one if they break a machine if they brick a machine if they force it to reboot they basically have to screw up if they want to be caught on the other hand hiding from an active team is much more difficult if you have someone that's constantly pulling memory constantly looking on disc for indicators and taking the current state of the system versus what is the known good State and we'll talk about how to get there it's much harder to hide from and even in those cases even no matter how skilled the attackers are how well their malware is written how well they're embedded into the system you're still going to be able to find them another major benefit is being able to find gaps in your system and application configurations a common example of this is audit logging being disabled when you have that enabled in the event logs you're going to have every time a user logs on every time a user logs off you can even set it to do it every time a process is created the full command line of the process but most people don't turn this on it's not enabled by default so you have to actively talk to your assist admins talk to the group policy creators and make sure that's that's there on the system another major issue is Privileges and I'm going to go through an example of this in a few slides but the if you're familiar with the concept of leas privileges running with the least privileges you need uh this is not done in organizations and it allows attackers to move much more freely through the network it allows them accesses to resources that if privileges were granted correctly they just simply couldn't do it and the last two benefits we're going to talk about are your Defender familiarity with your network so I'm going to go through a series of questions shortly and if you're a Defender if you're a manager of a team that is in charge of incident response or forensics and they can't answer these questions or you can't answer the question yourself then that's a problem it means you're not familiar enough with your environment to really be doing effective incident response so as I'm going to go through in the next few slides understanding normal is it's a hard task you're not going to walk out of this with one script that you can run on every machine and immediately understand normal modern networks and systems are made up of many moving Parts they're very complex and so it's very hard to say what is the normal state of a system imagine if you had a router what's the normal state of a router there's always connections going through there's files being pushed through look at your email server what is the normal State at midnight versus uh in the morning when everyone gets in and downloads their email so it's an ongoing thing um and it's not trivial but by the time you get normal figured out your IR is almost instant or it's almost scriptable because all you want to know is what's different on the machine now versus when it's in its normal State and you're looking for those differences and again to make this work you're going to need someone who deeply understands your systems um I can certainly tell you what's normal on say a default install of Windows 7 if you install Windows 7 with say Chrome or Firefox or normal applications they can pretty quickly figure out what's supposed to be there or not if you give me a random computer from your accountant's office or one of your lawyers or one of your HR people and they have 20 different applications that I've never heard of before those are going to very much change what is normal in the system they're going to load processes they're going to make network connections uh if you have security software they're going to load kernel drivers and again without seeing those before that's going to be very hard for me just to intuitiv intuitively figure out what's there I'm going have to sit on Google I'm have to research the products but again your internal team or depending on how closely you work with a third party they should know what's there you shouldn't be in the middle of an incident guessing why is this connect connection being made why is this process running and again as the the purpose of the talk threat hunting is what fixes this threat hunting removes guessing it removes not knowing what's normal and it really puts you in the only position that's going to give you hope of defeating attackers so what we're now going to do is look at a subset of the data that defines normal this is applicable applicable to both end users and servers and the purpose of this is to highlight just a wide range of data that an analyst needs to look at in order to determine if a system is compromised or not and again most people I talked to quite a few people today most people work in big Enterprises so imagine trying to do this on one system versus a thousand systems when you don't know what's compromised when you don't know where the attacker moved you need some way to very quickly weed out the data and figure out what to focus on and again without previous threat hunting and the Baseline that it provides uh it's going to be very hard to do and it's very likely that you're going to make errors another thing to realize is that while some of the terms I'm going to use are windows specific because I assume that's what most people are familiar with the same exact thing applies to Mac and Linux and again you're going to have to go through all this to understand normal uh Linux is obviously seen in a lot of data centers on the servers and many organizations are now allowing Mac inside so even if you've perfected this for Windows you need to start understanding those other platforms as well the first s the first set of data we're going to look at is running processes if if you pull up task manager on Windows if you run PS list on the command line You'll see that there's hundreds of processes running on the machine maare only needs one of those to completely take over the system to inject code into other processes to hook what's being typed on the keyboard to hook Network flow and because of this power that just one process can have Gathering the list of running processes is one of the first things you do in IR when you look at volatility if you saw Ryan's talk earlier one of the first things people do they get a memory sample on the machine and they run PS list if you saw the previous talk you can run CIS internals in prmod and it's going to give you the list of processes that are there but as I said on a default system you're going to have many processes running so if I was to hand you a list of processes say 250 of them for an average of a server can you actually look at all those and definitively tell me this one's malicious this one's malicious this one's normal could you do that without a baseline it's going to be very difficult to do that again when you throw in thirdparty applications from all the different random Departments of your company pretty much gets impossible to do that just off your own knowledge sitting on Google is obviously non ideal but I've definitely been in underprepared IR teams we from a thirdparty perspective where I'm asking them okay there's this process running what does this do and they're calling people in the IT department or they're sitting on Google on their phones trying to figure out what does this belong to is it legitimate is it malware also remember that process names are not really the best indicator a lot of maare is going to run as service host.exe it's going to run as like conhost.exe you need something more so you can look at what path it's running on disk you can look at where it started where it fit within the process chain but again if you don't know all of that something else has to tell it to you it's also very difficult to do uh at any type of scale and then another point to consider is the relationship between processes this can be very different uh depending on how it looks so you have those service host processes and again it's quite different if it's running from where it should be in Windows or if it's running from a user temporary directory on the other hand if you see command. exe running and it's a sis admin's machine or it's a developer machine maybe that's normal and if it's spawned from conhost.exe then that can be seen as expected the person opened up command. exe and were was working from the command line on the other hand you can use CIS internals on a live machine or PS3 from volatility and if you see command. exe being spawned under Internet Explorer or adobe flash then you probably have a problem that you want to look into but again without that Windows internals knowledge without that famili familiarity with what normal should look like that's going to be very difficult to do Patrick Olson's blog post on this is a very good example uh he spent quite a bit of time re uh researching what are normal processes on Windows the relationships between them and not only is it a really great resource from the forensics perspective it also shows how hard it is to do uh if you would print his blog post out it would be 15 pages just full of stuff you're never going to remember off the top of your head and that's only processes we're about to go through six more six or seven more artifacts that are just as hard to track and in some case harder and again one mistake in the malare goes undetected if I give you the list of 250 processes and you miss one or you say oh that's service host.exe or that's command. exe it must be normal well now Ma is on your system and you just missed the infection the next place we'll talk about where normal must be understood is privileges adhering to the principle of least privilege is definitely not easy and it's also very easy to make mistakes these mistakes can be very costly though imagine if you have someone running as local admin and then they get hit by a fishing attack and now someone's walking around the network with a pass the hash attack versus if privileges are correct and the attackers are stuck on one machine that's obviously quite a bit of a different situation for your organization an example of this I was working a case not too long ago again I was kind of in as a like a third party uh helping out with the IR and I was in the lab uh someone went to the machine that was infected they brought me back a memory sample and select files from the Windows system what I mean by select files are the files that have forensics relevance not just everything off the dis part of these files that came back were the prefetch directory assuming no anti- forensics was used on the machine the prefetch files can tell you essentially every file that was executed so in this case there was a piece of malware that was run from that the user downloaded uh they tried to download something off of source for but they got redirected to a malware site cuz they had um a malicious ad never quite figed out I'm pretty sure it was a malicious advertisement or some JavaScript got uh injected but in the prefetched file I could see where the malow is run that's going to tell me the exact second that it was first run on the machine and I noticed in the timeline that immediately after that the Tas scheduler was run so after seeing that um I asked for the schedule task directory of the user that wasn't part of the files that I originally got and sure enough the malware had created a scheduled task and what this scheduled task did was every 3 hours it would run it would make sure the malware was still active and persistent on the machine so it check for the registry keys and then it would also contact a command and control server so as I was doing this analysis I told the IR team that I was helping out look you have the schedule test that was uh performed by the malware or planted by the malware and not only is it on the system but it's running as the system account system is the most privileged user on Windows it can do whatever it wants so I told the IR team this and they flat out told me that my analysis was wrong and I got something wrong along the way I showed them the output of the script which I mean I'm not making this up the script tells you what it runs as and it said system so they came back and said that can't happen because the employee that was compromised every person in his department they don't get to run as local admin they run in a non-privileged state so at that point I pulled out volatility to get the final answer got the memory sample uh I used the plugin that shows you all all the processes that are running along with their Privileges and I showed that for every uh process related to that login session sure enough they were running as local admin just as I had told them before few days later I was on site for a while uh a few days later I came they came back to me and said oh yeah we had a misconfiguration in active directory everyone in that employees Department was running as local admin obviously if this was not just some lame piece of malware that I was looking at and this was a targeted attack well now the attackers can remote into that machine isal CL admin pass the hash dump passwords and so on and it's probably pretty easily get to domain admin and take over everything versus if that active uh directory permission issue would have been found before and the group would have been locked down the attackers would have been stuck to that one machine and this is something that threat hunting is going to find you're going to pull memory off of someone's system uh out of that department you're going to look at privileges you're going to look at processes you're going to see what power they have on the machine and say you know why does Joe and accounting need to be local admin you can get that fix before the attackers find it but only through that proactive stance to your own defense do you have any chance of doing that versus waiting for the attackers to exploit it and then after the fact you go back and fix it another area that's important to understand normal is expected network activity if you've never run net Stat or you've never used a tool that will show you the network connections of a production corporate system you're going to be horrified IFI there's going to be 10 20 30 applications that are making connections to wherever they want to uh connect there's going to be processes waiting for incoming connections and if you look at that without any reference or Baseline to know what should be there or what shouldn't be there it's going to be pretty much hopeless and again during an active incident is not when you want to figure this out uh you all you want to know which ones are legitimate or which ones are not some more horror stories where again I was brought in as like a third party for these um had an incident uh a machine was beaconing out to the network uh based on netf Flow data we also thought that it was accepting commands uh from other machines on the network so essentially the attackers were coming back in through that machine I sat there with a memory sample sat there with volatility and I started asking the IR team okay this process is listening on this port or this process is connecting out on this port to this IP address is this valid or not they had no idea they were sitting on Google trying to search it uh they were calling their friends in it who had maybe help them quicker the ticket system uh and in one extreme example they actually called tech support of the product begging them to tell them what port says this listen on and doesn't connect out on the network this organ organization the IR and was so disconnected from the it process that calling tech support and kind of begging them for answers was quicker than just getting a new machine imaged if we could have got a fresh machine image that hadn't touched the network definitely didn't have malware all we'd have to do is Boot It Up take a memory sample and compare what we were getting essentially we be doing threat hunting kind of during an incident but without previous threat hunting which they obviously weren't doing and all of the documentation that it creates um they were pretty much hopeless they had no internal knowledge base of what it should be doing they had no idea what normal looked like and so it was one of the most inefficient responses I've been a part of because we simply didn't have data to support what we were looking for to give us any answers and then another area where network activity really highlights how important threat hunting is uh is again finding those anomalous artifacts so how many people run software firewalls on all the machines in their Network even if it's just Windows Firewall being turned on okay so probably most people if you look at that those firewalls are going to have exceptions people need to be able to do web browsing they need to be able to contact file uh file shares um if you have any type of video conferencing many times those applications talk on really weird UDP ports so your firewall is going to have exceptions of applications that sh should be able to contact the network but again during an incident is not the time to figure out what those settings are and what they look like from a forensics perspective malware and active attackers will poke holes in the firewall that will grant exceptions to their own applications and if you again if you took a stock uh install of Windows 7 it's really obvious if malware.trace to sift through those again and figure out okay why is this there is a group policy uh is it did it come with the base install of the system does the application and uh poke its own hole in the Windows Firewall that is just wasting time over and over again and I'm going to have to do that for every system that's part of my investigation instead if you had a baseline of what normal looked like from that system and you saw normal in action through your threat hunting there is no guessing at that point uh in some cases it boils down to the diff command tell me what's different in the config now to what the current state looks like and then and I can just look at the differences another thing to consider is Kernel level root kits so these are going to operate with full power over the system to gain access to the kernel uh they're going to load a kernel driver so there's going to be an executable uh the operating system is going to provide some built-in way to load these into the kernel address space and at that point the rootkit can do whatever it wants and unfortunately it's not too easy if you're not familiar with the system to figure out what are legitimate kernel drivers versus the abnormal ones be also because of the power that kernel level Rook kits have it's one of the most important things you need to figure out you definitely need to know if malware is in the kernel or not again without threat hunting and all of that Baseline knowledge and documentation this is going to be really painful if you would take uh a default install of Windows 7 and you would get a list of the loaded kernel drivers you're already going to be looking at over 100 of them probably somewhere around 140 150 if you then look at your third party components your security tool tools any type of driver for a USB camera microphone these random Hardware devices on your machine they're going to load their own drivers as well so again do you want to give your analyst a list of 150 drivers or do you want to say this is the one driver that's on the machine now that wasn't there when we installed it as an example um hopefully you can see the bottom of the slide The Last Bow of the slide lists the names of four drivers as you would see them from a tool like volatility against the memory sample or from a tool that you ran on the L system uh MRX net and MRX CLS are stucks net while MRX SMB and MRX daav are legitimate Windows components without baselining your ability to detect stuck stand or not is basically your analyst knowing that off the top of their head and not screwing it up another thing to consider is the only reason I know the two stuck snet drivers is because people did analysis on it they said these are the malicious ones if someone had never heard of stuck net or some other malware they're going to say oh all these drivers have MRX as a starting point they probably have something to do with the network this machine's clean Meanwhile your Network's infected with stck net the next place where normal needs to be understood is common persistence mechanisms um these are a pain um they're very hard to deal with traditionally they were a bit easier so malware would focus on the Run Keys um on the system there's going to be a set of keys that specify which program will run every time the system boots and then and then what programs need to run when certain users log in this is obviously a very nice Target for malare and it's been targeted for the last 15 years or so unfortunately the Run keys are not the only source of persistence um Adam's blog series on this called Beyond The Run key is currently around 27 I think it's 27 entries he posted a new one yesterday on all the different ways that you can make malware persist on a Windows system without touching the Run keys so again most people probably I certainly don't know all of those off the top of my head or at least how to look for them and if you don't have a baseline do you really want for every disc image that you get to go look at 27 different places and look for not just look at those places but figure out which ones are legitimate components versus malware scheduled tasks are another place where this is very helpful uh schedule tasks um when you look at your nation state back groups this is one of the best this is one of the most common ways that they gain persistence on a system they drop a scheduled task um the triggers for scheduled task are pretty wide ranging so it can be when the system starts when the workstation locks are unlocked uh the reference I have to the msdn gives you the entire list uh of triggers that you can use but the idea is you can specify actions including programs running or drivers loading at very specific times so not just when the system boots or when the user logs in for a wide variety of actions so threat groups really like using scheduled tasks to um get data running on the system or get their programs running on the system and again if you look at a normal Windows install you're going to have 100 of these or 50 of these at the very least do you really want to look at those and try to memorize them or do you want some way to quickly difference the data and then the final persistence mechanism I'll talk about is Services these are extremely powerful they allow you to load kernel drivers so you start the service uh the kernel driver gets loaded into memory and then depending on how you do anti- forensics you can remove the service from the list but your kernel driver is still going to be loaded another thing you could do with a a service is load your dll in to a shared process so you might have 15 Services all running inside of the same process so now when you're doing memory forensics it's not just there's a service host running there's a service host with 20 Services running which one of these are malicious again without a baseline is going to be very difficult this isn't even just a time saer this is very easy to get wrong you get someone who's an expert with memory forensics volatility windows it's still very easy to screw this up and then the last area I'll talk about through kind of this uh this exercise of emphasizing why you need to know normal is your antivirus and your hips product I had several questions about this today um we use volatility but we also have security products on the systems and they inject code everywhere volatility picks up on the injected code but you trace it down and you're like it's just sem Manch it's just macafee it's just this other product again if you have some way to your normal system before any infection can happen and you know what those code injections look like from your security tool then when you see a code injection from malware it's going to standout is quite different but again without threat hunting and documentation of the results it's hopeless it's going to take forever and you're just going to keep going down rabbit holes that end up being legitimate components and essentially waste time so everything I talked about in the last several slides takes time so obviously looking through all this processes figuring out what should be there takes time and it also takes an experienced analyst don't take your Junior person who just joined the team and try to get them to Baseline systems it's likely they're going to screw up so if your organization is dedicating the time and the your senior level resources to do this it's essentially a complete waste of time if it's not being documented you want all you want for those things that I mentioned plus quite a few others you want to know what normal looks like you want in a way that's easily searchable uh easy to difference and then you can take that information and diff it during live engagements how many people have read the uh Phoenix project before the book by Jean Kim a few people okay so if you haven't read that uh consider it required reading kind of no matter what your role in it is but the idea here with this bullet that says do not allow Brent to be created is if you don't document what your organization knows and what's being learned during threat hunts as well as responding to actual incidents you're going to create silos and you're going to create Brent in the book Brent is the whiz kit in the company the book uh kind of follows a major it organization and they have Brent who knows everything he knows all the how the networks configured the architecture all the servers all the applications and the problem is everything in the organization serializes on Brent if something breaks Brent has to stop and fix it if two or three things break it once well Brent has to fix a then B then C and nothing else gets done unless Brent's doing it the same thing can very easily happen in your own organization if you have someone who knows everything the person doing your hunting the person who leads all your investigations and they don't document and share that knowledge all you've done is create a Brent and if that person leaves if that person's sick if that person's just on vacation around the world you have no effective response at that point and one thing that kind of drives this home is is the last bullet point if your entire IR team were to get up and leave if firey was to come in with a million dollars and say we're taking all of you right now how long would it be before you had effective response again all your institutional knowledge is going to walk out the door if it's not documented and everyone that knows your Network's going to leave versus if you had effective documentation and you had ongoing training and processes you could bring new hires in throw the documentation at them and let them sit there for a week and now they know your network and your resources just as well as the people that left and that's that's where you need to be U I'm sure if you deal with hiring if you're a manager turnover is very high if you have skills in this industry you can go to one of a thousand companies whenever you want uh so the between the turnover and between people leaving for better money better opportunities if you're leaving if you're creating Brent if you have one one or two or three people that know everything you're definitely setting yourself up for failure so if you implemented everything I just discussed to a t what does your IR team and your overall organization gain again the biggest thing is you're moving from a reactive to a proactive state of managing threats no more receiving you know blacked out letters from the FBI saying you've been compromised no more saying this much data left the network from one of your providers you know when the attackers are there if you're doing small scale hunts once a week once every 2 weeks well that's the Gap where the attackers can be on your network before you're going to find them the other thing is you're going to have effective response don't be like the places where I've walked in and we're sitting there Googling process names or we're you know Googling kernel drivers trying to figure out what's going on know your environment you certainly want to know it better than the attackers and after a while you're just immediately going to know what's there if you have developers in your organization or even people that can just script things out eventually you'll get to where you're running the diff command or you're taking a script that you wrote and you're diffing the known State the normal State versus what the system currently looks like and then the other thing is you need to follow through with a documentation that needs to be forced onto people make them actually document what they find and then roll that into internal training so with that said how do you get there uh you really need two things that you there's two goals you need to reach if you want effective threat hunting and if you want it to benefit your organization as well as it can the the first one is the executives need to understand the value of a properly prepared team uh if your team is constantly swapped with swamped with useless alerts so if every time someone logs in after lunch or every time someone's machine shuts down or some random update comes in if alerts are being generated and your IR team is constantly consumed with that you might as well give up they're not going to be able to prepare for a hunt they're not going to be able to set up the processes to do hunting effectively and then they won't even have time to do the Hunts anyway so you need to be able to break out the team priori ize your alerts uh Sean's blog that I referenced before has some really good advice on that but you need to be effective and looking at a thousand alerts today that really aren't that helpful is not a good way to use your team and then the other thing is which we're going to go through is the IR team needs to be integrated into the organization we're going to look at how security is integrated and we're going to look at how IR is not integrated and we'll see the issues that that creates so again if you look at this and basically every organization every mature organization security is everywhere so security is involved in the rollouts of applications the ongoing security they're in the planning meetings yelling at developers that they can't do certain things or they're going to give away all the customer data but how many times have you seen an IR team member there how many times have you seen a person from the forensics team in those meetings or in that planning so if you look at the predeployment phase here we're going to look at predeployment and post- deployment uh there's a few things kind of comparisons so before an operating system ever touches production before an application's Ever rolled out to production it's reviewed for security issues as well as security compliance in fact there's entire companies that provide these Services you have companies that will come in do your source code reviews they'll do the Baseline analysis there's no such equivalent to that in forensics there's no companies that do the same thing if you look at Microsoft's Baseline security analyzer it's a free tool that almost does this for you so Microsoft realized the importance of this they wrote a tool for it and it basically does configuration checks of your install if you look on the app development side there's an entire methodology around the secure sdlc this is how people smart people sat down and figured out how can we embed security within the development team we want to find vulnerabilities as soon as they're committed to source code we want to find them while they're still within uh the dev environment and we never want vulnerabilities in our code to reach production on the cloud development side uh Rich Mogul has done some really great work in the devops uh side of this so taking security as well as security policies and through the entire devops life life cycle you see security there nothing's ever done where there's not security considerations in the process and again in mature organizations this is just seen as normal uh it is or security is embedded in it and it can't do anything without kind of security signoff and then in the post- deployment cycle it's the same thing you have dedicated vulnerability scanners you have either if you have an internal red team they're constantly trying to break systems otherwise you're bringing in third party companies to do pent test and you're trying to figure out is everything up to date are the patches up to date what can be compromised and then if something's found which it often will be it's immediately dealt with a you know a severity one tickets created this application needs to be fixed this server needs to be updated and kind of everything Deadlocks until that is done on the other hand if you compare this to kind of how IRT IR is within the organ organization and the power they have and how they're embedded into the on ongoing flow there's really no comparison that you can make IR is nowhere to be seen except when an incident happens and then everyone's yelling at IR until the attackers and the malware is gone but this can be fixed if you change your processes so again if we look at the predeployment phase IR should have just as much input as everyone else does those misconfigurations where there's no audit logging on where logs are being truncated to small sizes that are kind of ridiculous all of that can be fixed before anything touches production also when you look at your application installs so from third party companies many of those have settings and they can log a lot of data but again if no one's looking at what's can be logged and the types of logs that they create that's never going to be written out and then when you go to do forensics you say man this app could have logged exactly what we wanted to know but because we weren't proactive about it that logging wasn't turned on so we can turn it on now and then if we get popped in the future maybe we'll find it but we lost our best evidence and then if you're developing internal applications you need to make sure that it's auditing anything that you can reasonably think a user a rogue Insider or an attacker is going to do should be logged and it should be logged in a way that your FX team can go back and find it it's pretty inexcusable that a rogue employee that wants to rip you off or an attacker that comes in through the network can use your own applications against you and you didn't build fics into them if they use the application to steal employee data personal information Healthcare data why isn't all of that log somewhere so when your forensics team comes back comes behind they say this is exactly what happened this is the time it happened this is how they got in through the system and now you actually have effective forensics in your organization and the end result of this is everything that's in production is ready for forensic exploitation the artifacts are there they're logged how you're logged in they're logged in a way that the team knows how to find them and how to parse them and then you're enabling really the best investigations you can get and then on the post- deployment phase it's the ongoing threat hunting you need to know what the applications look like when they're actually deployed what the systems look like as updates happen as applications are updated and all of that needs to be documented and if you don't do this you're going to quickly fall behind if you only do the predeployment phase if you only know what the Baseline looks like you're going to fall behind uh the applications again when they update we've seen major changes in Windows uh we've seen major changes in third party applications push in more d LS you'll see security tools that got burned by a piece of malware they now add 100 more hooks to the system to try to find it in the future and all of that's going to break your existing uh what your existing break Baseline looks like and what your existing normal looks like and then another place where IR needs a lot of support is incident preparedness too often an incident happens the IR team has to scramble to get access to data and you end up dealing with horrible resources I recently had a work a case where they said we don't have any extra Hardware to give you we can give you this machine to work on I figured out I wasn't even on Native Hardware I was in VMware it had 4 gigs of RAM and one virtual core and then they gave me 3 terabytes of dis images and 10 memory samples and at first I thought they were joking but that was the only thing they could spare to me to work on the case and I basically told them okay I'm going to kick the tools off and come back in a month and then we might have results versus if they had dedicated hardware and dedicated resources to work with that's not going to be a problem and you'll have actual effective response uh so I have a list here this is certainly not comprehensive but it's definitely a good start your IRT members should be able to monitor portions of the network um obviously most places don't want full peap or they can't afford it or the engineering is just too difficult but if you don't have that and there's an incident or if I'm threat hunting I should be able to monitor subsets of the network it's very hard to track where attackers are going or to see how data flows between machines if I can't look on the network another place is dedicated storage servers so if I want to store dis images memory samples select file pulls if I if my tools are generating output that I want to save I should have dedicated storage for that how many people have heard of crypto wall or the friends of crypto wall how many people have seen the IR team because of lack of preparedness have to authenticate to production file servers to store data if anyone will admit to that I see people laughing that's good as good as a hand what if crypto wall was on the machine that you're investigating and you you connect to that file server because you need the extra storage to store data well now your corporate file server has just been encrypted and now you're fighting ransomware and that's not an extreme example it's happened to more companies that will ever admit it if it wasn't for backups if it wasn't for tape they would have lost everything they had and they lost the data that was in the Delta between when they backed up last versus what people were doing work for storage is uh especially if you're talking a couple terabytes of storage uh depending on the size of your organization that's Dirt Cheap give the IR team their own file servers don't let other people use it and let them store the data as it's generated and as it's needed you also need to have your applications predeployment downloading other tools while machines are compromised and while that data that we wanted to capture was quickly leaving memory because time was burning versus if you had VMS already developed there's free VMS you can download that have tools you can build one out yourself or just to have the tools inside your company ready to be pushed out then you can quickly get the data you can get it in a state that you know is going to be reliable and this is where hunting really helps if you get someone new in the organization they've never pulled memory before they've never pulled disc they've never used PS exact they've never used F response let them do it during the hunting and if they screw up it's not that big of a deal there's no active response at that point but if they perfect it and they know how to do it during hunting and while they're getting used to the environment then when you have that real outbreak you're not blue screening the machine because the person pulled memory wrong you're getting a valid memory capture out and you have data that you can work with yeah and then the other thing to do is make sure that you're utilizing the documentation uh as you're doing hunting as you're finding these artifacts they need to go somewhere um I don't really have uh an off-the-shelf solution for this like some documentation product uh we kind of rolled our own and C ciz one uh but you need some way where every member of your team can say okay this other team member knows what our email servers look like and knows what they should look like in memory how can I find that even if it's just a Wiki to start this is what the processes I should see this is the kernel drivers and through that documentation and forcing people to do the documentation uh is how you get to that step another big benefit of threat hunting is adding to your internal training so if you look at some of the work Jack's done in the public space uh Jack crook from GE that I mentioned before he creates very realistic forensics challenges these are things where if you get someone who's never done forensics if they want to see memory forensics if they want to see what very real very realistic attacks look like give him one of Jack's challenges they're modeled after exactly what he sees hitting ge's Network and but as great as his challenges are and as much as you can learn from them they're still public so this is not specific to your environment but as you're doing threat hunting you're taking dis images you're taking memory captures why not roll those into your own internal training take a memory sample that has malare on it take a memory sample that has an active attacker on the system and store it somewhere put it on secure offline storage someone new comes into your company and you want to you want them to get familiar with what does volatility look like on the system what does fdk look like on the system what does end case look like on the system use your own images use the the data and the files from your own environment that way when they see it again they're that much more familiar with it but again if you're not doing threat hunting you're certainly not going to have that and if you don't have the resources the hardware resources and the the network resources that the team needs that's going to be hopeless and then your internal training is going to rely on other people's environments and other people's view of the world versus your own company's view of the world and then the other thing besides benefiting your internal employees is documentation and threat hunting really helps third parties I kind of explained some of the bad situations I've been in the very headache inducing situations and something to realize is that very few organizations outside of uh ge Lockheed Martin and those can handle all of their breaches internal if your entire network I compromised right now you'd probably be calling a third party to come in and help and when you think of what knowledge a consultant that's coming in for the first time has it's basically the same knowledge that your new hire has they know nothing of your environment nothing of what it should look like and you're paying them a ridiculous amount of money so if you if they have to sit there a week while you generate documentation and you install tools and try to explain to them what things should look like or if every 5 minutes they have to ask you questions hey is this normal is this normal is this normal that's not effective you're wasting your you're wasting your company's money in a sense uh you're kind of wasting their time because they can't just do analysis they don't know what anything's supposed to be uh and that's very ineffective ineffective response versus I walk into your environment the first time you say here's a 10-page PDF of what everything should look like or we need you to analyze this Windows 7 sample from accounting this is what it should look like there's no more guest then and I can work on my own and I don't have to bother your team while they're trying to respond as well again in the the example with the Brent you can't do anything at scale if only one or two people know everything and then this is a the only uh graphic I have in this the deck but this was a a tweet that I got back I asked on Twitter I believe it's when I was asked to do this talk or when I was talking about threat hunting uh at a different conference and basically someone replied the reason more companies don't do threat hunting and they don't take it so serious or they don't understand the value is there's no compliance function for it there's no checkbox that an auditor is going to say do you do threat hunting and when is the last time you had a threat hunt but if you look at the number of breaches uh in the last two years if you look at the severity of them Executives definitely know about this on the other hand and I'm definitely not a lawyer uh but if your company has cyber insurance which if your company is worth more than a million dollars say is a low estimate you probably have cyber insurance insurance and if you look at the due diligence Clause within those policies they're going to say you need to be actively preventing breaches on the network you need to be working to detect them and once they're detective you need effective response sitting on Google and searching process names and ports and applications is certainly not effective and then if you're insur I've been on both sides of this if the uh free with their call like the insurance auditor or the person trying to figure out did you meet your contractual obligations they're going to say okay well how did you you look on your network how do you know your network what type of response did you have and you say oh well Brent over there knew everything and we just did whatever he said versus hey here's our 50-page PDF of documentation this is how we actively work on our Network looking for threats this is how we respond when certain threats are found that's going to look a lot better and depending where you work also if you ever have Congress knocking on your door or any of the other Regulators giving them a blank piece of paper that was written the day before is going to look really bad versus having documentation of what you should have been doing all along and what your company was actually following through one other thing I briefly wanted to touch on was uh spending inside of organizations so due to some of the compliance issues I talked about and just kind of the deeply ingrained thoughts that most organizations have they shift a lot of money towards penetration testing and vulnerability scanning and application assessments and this leaves the IR team with very little resources again in most organizations IR is thought of something after the fact you sit in a corner by yourself uh something gets popped people get into the network you go fix it and then you go back to your corner and leave us alone otherwise in reality this is a that's a pretty silly approach no modern network is unbreachable in fact most organizations whether they're admitted or not is one fishing email away from total disaster think of some of the biggest attacks RSA several others were fishing emails once the attackers got in the network they could roam around where they want and take the data they want and when the other thing to consider is no matter how good your internal red team is or the third party company you bring in for a security audit and some of them are obviously very good at what they do they're not going to match the resources of a determined adversary if you look at some of the documented attacks against big companies um you're talking threat groups that spent months two months 3 months four months patiently probing around a network trying to break into the network before they finally got in if you hire one pen tester for one week how does that really compare to that attack versus do you have the budget to hire a 20 person penetration testing team 24/7 for 3 months and if you're not getting pen tests on that scale you're not you're not matching what someone like China Russia or an organized crime group is going to do when they decide they want your data the other thing to realize with these same groups is they don't play by the same rules you might write a scope of a scope of Engagement for your pentest that says fishing isn't isn't a valid way to get in or this application server is new so you can't break into that one the person in China doesn't care they're going to get in through whatever means they want and it's simply impossible to replicate that level of determination and kind of endless scope through a penetration test so if you accept that you can be breached and that if a determined group decides they want to get into your network that they're going to get into the network then you're going to shift where you focus your spending uh I have it on the slide here I'm sure everyone's heard of shell shock but if I was to say hey I just got an update on Twitter Shell Shock 2 is out or some equivalent vulnerability for Windows just came out would you feel better knowing that yesterday your vulnerability scanner said hey the whole Network's cleared and we're fully patched versus have an IR team sitting in the sock or sitting in the office and if machines get popped or when machines get popped we're going to have very effective response I know I'd rather have the response because even if you're fully patched even if your applications have been tested for vulnerabilities your network is only quote secure until another vulnerability comes out and vulnerabilities come out all of the time versus IR where once you you get the process rolling once you have effective response in your organization that lasts essentially forever as long as someone's keeping the processes up as long as the budget stays behind it that's not going anywhere so only have a few minutes left so I'm not going to read this but these are um steps you can do so if you wanted to take it to a manager and say we need thread hunting these are the steps you could follow so you want to make sure that again your IR team needs time to do it uh another thing to realize is you can start small make sure you get the process down right make sure the do your documenting what you want uh if there's gaps in the documentation then refine your process and then you can move to full scale hunting and really start understanding your environment this is uh again steps on embedding IR into it um when the slides get uploaded I can put some slide notes on here to kind of flush out some of the points uh but these are the things if you're going to get IR into the it processes this is how you're going to do it effective and not waste time so in conclusion uh hopefully see the value of threat hunting by this point or at least trying it in your environment trying to get that understanding of normal when it's done correctly it's not only the most effective way effective way to stay ahead of modern threats it's really the only way to do it reactive simply doesn't work anymore hasn't worked for a long time but in the last couple years it's really come to the Forefront if you're not actively looking for vulnerabilities if you're not actively looking to figure out what should be on my system versus what shouldn't be there you're going to lose and eventually you're going to have a FBI notification you're going to see your data being sold in a Black Market somewhere attackers are going to get in and they're going to take what they want and you're never going to know the difference versus through this active view of defense and actually figuring out your systems you have a fighting chance and in most cases you can very easily or maybe that was overstatement you could somewhat easily uh get the attackers out and at least find them on your systems so that was all of the slides uh can now take any questions or comments if there's still time all right uh one question what do you see it's role of threat intelligence the how Okay so so the question was uh for the video what do I see as the role of threat in Intel during the hunting process um good threat Intel so kind of targeted threat Intel not just like a billion indicators of whatever uh that can obviously be very helpful especially on the network side so if you see uh depending on where you place that if you you can look for lateral movement inside the network that can look for files being transer that can see where the outgoing connections are um and then you also have indicators on the host so assuming malware is not interfering with say the list of processes that's being generated or the network connections uh that good threat Intel that thread Intel that's decent can be very helpful and and that can really help with the scalable and the automation part of it as well did that answer your question so it's definitely useful as long as your thread Intel is not here's every you know ipv4 address with no context or something like that so yes name it's called The Phoenix project by Jean Kim the Phoenix project um and that's not me just uh saying the book's awesome it's a tech book that has over 800 reviews on Amazon or stars on Amazon uh you really should go ahead and read it Jack in the back is giving me two thumbs up for that any uh that one online resources for daily security topics was that the question um so one of the best ways I keep up with security which might sound silly is tweet deck uh so on Twitter even if you never post you can use tweet Tech to automatically search for terms so I think I have like 60 Search terms in mind so anytime someone talks about forensics vulnerabilities memory forensics this forensics I see what the Tweet is and that's not just people actually posting a lot of those are like RSS feeds that feeds off of blogs and other things um on the forensic side another thing you do is go to harling carvey's blog windows. blogspot.com not only is his blog good but he has the blog role on the right that lists pretty much every other blog that's relevant to forensics so I do that pretty often as well um and then there's some mailing lists so there's the win forensics mailing list the the paul.com mailing list daily Dave uh we have a volatility mailing list for that's open to anyone if you care about memory forensics and the malware stuff uh keeping up with those kind of is very good for keeping up with latest things so yes one of the things that you mentioned on the first slides that hunting a lot ofs get Sy us and I just want to clarify you're essentially saying that by understanding Network perspective perspective yes so the question the summarized version of the question was was hunting with IND with indicators so like the thread Intel versus just taking the Delta of the know the good State versus the current state uh hunting with indicators certainly works if someone said uh this big piece of mare came out of this big threat group likely compromise your network yeah I'm going to start with the indicators and work there but kind of um I guess the Holy Grail would be the best term I can think of of forensics is and that's how we deal with memory forensics I don't care what a specific piece of hour is if you look at volatility it doesn't have an indicator for stck net right or any of the other big pieces of app malware but if you run volatility on them it's going to find it 10 different ways so uh again the holy gra of forensics is not an indicator of this or a hash of this or a string of this I just want to know should this be on on the system or not or was this on the system the last time I looked and if you have that deep operating systems internals knowledge and you can look at that Delta then I I mean I could do forensics all day without indicators I really don't ever use them unless I need to know like the specific piece of malware or the specific threat groups I just want to look at that Delta so uh was there questions try I guess scale autp of what all areas would you suggest that you try to collect the Bine okay so on the disc side like what what what we call Select files would be the the registry hives the prefetch uh if we're looking at end user activity we'll put like the link files and the jump list uh we always pull the scheduled task folder just because of how important that is um I have a I have an actual list I could send you if I could look at it but off the top of my head and then um obviously memory as well so we can do memory forensics against it that way so um geography so so the question was how do you scale not only size but also uh or number of nodes but different configurations different applications obviously different uh countries even if you change the language pack that'll change stuff a lot um uh there's no like automatic solution to that you you really just need um baselines of what's there I mean one thing you can do is let's say um everyone in accounting across the world uses one application or something you can Baseline that application so even if you don't have an automated way to to diff it against that machine or the diff might get polluted you at least know that these 10 you know when this application is running or this background service is running I should see these 10 processes um so at some point though you have to do the baselining if you can script it out then it's going to work um we've done some stuff with VMware uh with VMware snapshots you can get a Delta of the dis that vmdk uh I forget the extension I think vmdk is like a Delta of the dis uh you can also uh take a dis image before and after like applications install uh for the registry there's a tool called redshot you can run on the live machine um and then in memory it's uh you know volatility and all the plugins that we care about before and after say an application was installed uh one interesting thing you can look at to build on it's um it's called Dam uh different it's called like differencing analysis and memory it's open source on GitHub under fibo forensics um but you uh it's meant to use in a VM so you can like run an application run a piece of malware um you take a me you take a snapshot before and a snapshot after and then it automatically does the diffing of all the the plugins for you uh so that's a good way to automate it but again um you can't get around the Bas lining so you need to scale out resources or like automate the process as much as possible so uh so unfortunately I don't have like a script that can give you to make that easy um right so cat M right exactly and imagine if your baselining took you from there's 250 processes to there's eight we need a question and then if you're many times like we scope out like uh everyone say on the same like VLAN or something depending on the network um if you figure out those like eight kind of extraneous processes and then you go to the next machine and you see the same eight process is you've already kind of done the research and then if you document what you did when you come back a month later you don't have to do it again um so it's definitely it definitely builds and if you have an internal team that keeps the documentation rolling you get to a point where there's not a whole lot of guessing left you just know what's going to be there or not so so yeah did that kind of answer your question a little bit okay any more questions yes ask you one uh there's a few ways you can do it so as in many cases if you've been compromised before and you can show that this is how long we took to respond to the incident this is the resources we had to dedicate this was the end result versus uh here's a hunt so here's like a mock scenario you can even like throw malware on a test system and say with this new process this is how long it took this is the documentation that resulted and if we would have a same or similar outbreak it would take even shorter because you're narrowing that gap of unknowns on the system and you know like what's anomalous or not so or even if you haven't compromised just throw M on a system say this is how we do it now give us some funding to do hunting and and documenting and this is how it'll look two months from now or something so any other questions okay thank you for your time hopefully you enjoyed the presentation

Channel: Adrian Crenshaw

Views: 6,600

Keywords: hacking, security, irongeek, infosec, BSides, Tampa

Id: 751bkSD2Nn8

Length: 68min 30sec (4110 seconds)

Published: Sun Feb 22 2015