How to Build Threat Hunting into Your Security Operations

during april 18th through the 25th this year any new orleans so we hope to see a lot of you in attendance as we deep dive into the conflict so we're going to be talking about today so today we have a joe moles from red canary joe is an IR and digital forensics specialist I use more than a decade of experience running security operations and we discovered I said erector of detection operations at red canary he leads a team of security analysts to help organizations defend their endpoints against threats prior to joining red canary Joe built and led security operations intimate response and discovery programs for fortune 500 companies like OfficeMax and Motorola also presenting today myself with malfoy security strategies for carbon black nineteen-year security veteran spent some time over in the DoD space but really specialize in building rapid security operations programs and going from a to zero maturity to a high level of maturity as fast as possible that's awesome so what are we going to talk about today so we're going to we're going to ask the question what thread hunting is we're going to talk about automating thread hunting and we're going to talk about how it's autumn a tional the hunt is not what most people think it is and then how to build this practice into your security operations next so why do we threat on well we threat hunt because adversaries have changed right we have nation state state actors that are out there their tools are now being made public and in the hands of everyday cyber criminals right we have the cyber criminal teams we have hacktivists insiders and then finally you know I think an often ignored group which are scriptcase right so people who pull tools of github publicly available things point of matcher systems and attempt to do harm so what is threat hunting thread hunting is the practice of actively looking for anomalous activity that has not been identified by your existing tool sets by searching through various sources of data the right hunting is the of actively surging with an environment for evidence of previously undetected attackers so I think the biggest point I would make here is that threat hunting is a proactive practice that involve humans and technology to seek out bad in your environment what should the goals of threat having be well the first goal should be to identify solid evidence indicating the presence or residual activities of attack attackers within a network or computing button you want to use threat hunting to assess your existing security and net and network security tools and identify gaps within process people's technology and education you want to improve your prevention and detection as a result threat hunting should as you're out there threatening it should identify gaps in your program that you're then able to quickly close so with that I'm going to turn it back over to Jill for the next couple of slides yeah so so quite Rick mentioned your threat hunting really a lot of it comes down to ax t it's a continuous ongoing process is continuous is to continuous improvement effort so you're looking for a bad thing you're finding a bad thing and you're figuring out how to find it faster and you keep doing that over and over and it's you're working on we're talking about using we ask that activity of actually delving into your information to to look for those bad things however you've heard about them have you thought about them you know finding those things in your environment taking action on them and then figuring out within your tools that within your security program how do we do that faster next time how do we continually improve based on the the things that we found this time how do we find it faster next time how do we get that in front of my analyst how do I get that from my security team how do I even automate that so that you know my people can find the more advanced things next time so it was how do we get there you know starting in ten of these the least effective way it's you know taking an ad-hoc process going as a one-off getting in looking for things that you found in a new a new court things right you say hey I wonder what this looks like or hey this some activity I just happen to notice that smells a little funny I've got a hunch you know just kind of poking at it when you got free time you know then working all the way up to full automation letting the tools do the work for you and not for not thinking for you but you know getting that information in front of you so you can take action on that and getting ahead not working towards the easy things the things that you know about you should let your tools find those for you so then you can continue to dig in find the things that your tools don't know about you know this is kind of a uniting technology and people I mean people have you know there's a power in people there's a power and tool and we want to bring those two those two ideas together augmenting that that human capacity to identify zhing to work through complex problems to do that analysis but allowing the Machine the technology whatever that is to augment that ability and really bring those the strengths of both of those together into one unified effort so I'm going to pass it back to to Rick here we're going to kind of just talk through some layered approach to or hunting yeah you know I can't black our philosophy is really this you build on top of the pieces that you have in your program right for us everything starts with visibility if you're not collecting the right data from the right assets there's simply no way to start enabling things like alert triage and thread on so we're going to break this these kind of layers down and talk about the impacts for threat hunting and how you get there right so the first one we're going to talk about is visibility all right so you've got to collect the right things to analyze the right data if you're not I would say step one is drive the visibility right so so if that's on the network drive the visibility on the network if it's on the endpoint often collect the right security data so you can start to analyze that and here's why like take a look at the picture can anybody tell me who caused that accident based on the picture I would say no right but you know if you're not collecting that data and understanding the traffic patterns you're simply not going to be able to stop things like this occurring however when you start to drive the visibility in your environment what what you see is some normalization but you see patterns of things that occur and I think it's much easier to detect say someone driving up across the track in a NASCAR model than it would be in the previous picture so for us you know again it does become about the visibility because it's the foundational layer that everything on top of it is built on I think we're going to talk about how that is associated with the attack kill chain and specific to things like IO sees patterns of detection and that's type of stuff and with that all I'm about to go so it's a good starting point so giving that disability in your environment looking through your data sources you know some of the first you know low-hanging fruit attacks key ideas that you can become kind of attack is you just basic indicators you have you know threat Intel feeds you have things that other tools have ought you know that URM animated that I've identified for you you know so looking at those things feeding those in your system you know raising those up and then also looking at the behaviors of those items that you've previously previously identified again we're talking leveraging technology that's already potentially in your stack and in getting that information eternity of your human and your team and letting them start identifying you know that you know what they got out of that information so again looking at basic indicators you know indicators they're they're usually pretty easy to run to your environment and then looking at you know those patterns of behavior you could you know things that you've already found or things everybody gets the reports and the blog post and everything and looking through that information not only taking those hash values as IOC s as IPS all that kind of information there has got value but also read through that report what does this look like within my tool set what does this look like as a behavior within my my environment one of the you know great ways I think I refer to this endeavor versus the other talk to see the miters attack matrix I think this is a great framework for understanding those behaviors of hey attacker got in my environment what are they going to do other other good frameworks everybody knows the kill chain model look through these you look at you start laying this next to you next to your your your tool set medic your team and say okay let's step through this with sepsis in the environment let's look for these kind of activities what does this look like in my environment what does this look like at my tool set and work through that systematically also you can use that as you're going through and look at it as a gap analysis it could identify within your within your environment and just kind of switch and give a little bit got a question that came in the question is you have some good resources for determining what kind of data from an endpoint is most relevant from hunting working you where can we get the most bang for the buck in terms of visibility so from a visibility and endpoint visibility aspect I'm a big fan at the endpoint that's the data that I work with imagine rich Neal you know you probably agree with me but you know as you know we both work for you know companies that focus on the endpoint within that endpoint data you know there's a lot of different places that you can that you can kind of jump off from what kind of direction I tell my team when we're doing development on trying to come up with you to the pagers looking at a new environment is you know let's take something that we already know about and pick the the smallest unique behavior that we can find them and look for that behavior within within an environment and then we start stacking up those different behaviors that we hey maybe it's a an instance of PowerShell running a certain command line flag okay let's see everywhere that we've got that going on and then when we find you know some of its going to be good some it's going to be bad but you know the ones that are bad let's see what other behaviors then followed on before that after that and add that to our ad that kind of to our stack or a list of behaviors that we're looking for or again I point back to a point back to the attack matrix is a-- if you go through their their webpage there's a lot of really good information of hey use some basic attack behaviors at these different stages of a compromised you look at those within that yet and point data and then use that as a pivot point yeah Josette that's a great point I was going to bring that up right and say yes you can use attacks frameworks to reverse all of the visibility that you would need right to actually detect those types of attacks yeah I'm a little jaded I work with car black and that's what we do right we're for a big invisibility so so yeah you want to look at cross processes right so when a process kicks off another function or process that's an interesting a security metadata that you want collect dll changes those are interesting from our security behavioral perspective registry changes what user ran what process of what had those are all things that you absolutely need to collect to get that definitive answer right because I know for me I'm part of the reason that I came over the card block if I had these unanswerable questions on my endpoint three years right and we you know I kicked off internal projects we wrote a bunch of code we bought a bunch of tools and still couldn't answer the fundamental question of like what actually occurred on that end point so could we drive to ninety percent accuracy yeah but ninety percent our world isn't good enough I need that last ten right so so I think looking at the fundamentals of what you actually need to record so you can do detections along the kill chain or the attack framework is is where you want to be yeah and there's just there's another slide that talks you know just kind of reiterate from a lot of these data points looking at me so you're looking at some high-level behaviors what does that look like and again about it within the tool tips that you have you know you have you know a carbon black or do you have another end point tool that can provide you that disability look at you know look for these behaviors within within those environments and start working down through those figuring out what that looks like what has fears to be normal and what what's not some of these are very you know is when you dig into endpoint data some of those types of behaviors looking at it from that telemetry standpoint get to be super noisy and then one of the questions that came up from the audiences how much time to you do you have to focus on indicator based detection compared to behavioral based it seems the industry is split on the efficiency at right Intel feeds so so for my my my team we use threat Intel feet most of our detection is focused on behavioral information but threat Intel feeds are there's a lot of values that are still for for that type of you know quit wind detection and quick rim you know validation fades if I can quickly check within you know within a tool I can say hey let's see has I have I seen this md5 anywhere in my environment know okay good but you know I can and then I can also easily feed that in you know it's a quick win from an automation standpoint you know I can say you know here losers with the md5 right here this list of domains I can quickly add that in either from a prevention standpoint or maybe it's a bunch of known bad IP push that out to my networking put that in my proxy put that in my firewalls shut that down right away you know from a tool like me and carbon black I can take those md5 I can do a quick you know create a quick watchlist I can do a quick query and identify those information and do some quick lists if I do find that information I can take quick action and you know one-stop that right away but then too I can look at that behavior around that and then learn from that and use that to drive more in-depth detection going forward yeah I think just secondarily to that a couple of couple of things I kind of thought of as Jones on a tree right so the first thing I'll say is but bring in your pen test games and pen test for your gaps right so if you specifically want to be able to detect say a PowerShell type based attack once once a attackers are in your environment scope that for your read to write or and if the red team can't do it go find one that can that should reveal your detection gap from the back end which should give you a short list of projects to go Lum tree whether that's buying a piece of technology like harm block does to cause a visibility gap or doing that through some other mechanism I think it's huge and then and then I think the last thing I'll say about il cheese versus behavioral it's a maturity thing right so I think it's an industry we weren't doing anything about in the indicators of compromised now we've enabled sort of these quick easy wins as Gio said right so I got a list of bad IP addresses let me go in and block those real quick but we all know that data is transitory and very easy to change for the attackers right and we're going to talk about this a little bit towards the end of the slide changing the cyber economics behind it right so actually making the attackers job harder by by improving the barriers to entering into your environment so that eventually they move on or the attacks that they're hitting you with I don't have an ROI right which means they're going to switch what they're doing so as an example if we prevented ransom are 100% across the board there would simply be no people participating in in the cyber economy of ransoming systems because there's no payoff form right so yes I think for us it really is right and IOC do we do patterns of attack prevention detection but really what all that stuff does is allow for faster triage like so being able to actually drive to the root cause and spouses policies and say what happens you know did the user click on a malicious link where they fished did it come through the a you know rapidly being able to collect that data analyze it and then say this is what occurred which should drive remediation efforts right that should allow me to look at where your prevention dollars you're being spent today and maybe in some cases it's being spent incorrectly so you have upstream filtering that should take care of this stuff some of the phishing attacks some of the spam attacks things like that but maybe they're not as effective as they could be when you when you have that correct endpoint disability and you actually see what all that filtering is either doing for you or not should be very easy to then justify either in increasing your prevention spend or to look at the prevention tools that you have in place and pivot those to different technologies that they have more epic and here's why it really comes down to this we can't know what's bad ahead of time right so uh next Jill if you want my awesome so we can't know what what's bad ahead of time right I mean we're we're always chasing the tail of the attackers things we know attackers are innovating wildly they're moving as fast as they possibly can and oh by the way now we have large or intelligence agencies that have produced lots of lots of really good code and lots of fun stuff that's not male t'market right so over 20 17 and 20 mm making my personal prediction is most attacks become a dance attacks you know we talked about advanced persistent threats years ago and sort of started ringing that alarm bell now what we're saying is that actually come to fruition where you see standard pieces of malware using a Bayesian techniques on a regular basis to avoid your other technology right so prevention is a bit of chasing the tail detection and response really becomes about rapidly seeing the unknown and that that bottom of the iceberg being able to put that picture together and then actually drive up your program and mature your control next so so yeah so I mean it becomes very key we talked about collecting the right data for visibility and why is that well look the pings out there struggling and I've met with problem over the last few years I've probably met with 500 plus organizations at no point in time as any one of those organizations said I have enough people and I have enough time to do my job right so being able to quickly respond into tech these things and have a short list of alerts when the analysts and the folks in the socks come in to do their job all save them time and and we'll talk about that time equation again when we get into automating this stuff but essentially you need to be able to quickly recalls it right so in this particular instance you know Windows Explorer launches chrome from them launches Java that's fairly standard you're going to see that you know hundreds of thousands of times on endpoints and it's it's meaningless from the security perspective right a user's willing jump no big deal but then Java does something funky right and it kicks off CMB WEP which then starts to execute some malware now you see the attacker you know doing data exfiltration and or lateral movement they gain persistence in your environment now they've got credentials they're probably doing some other stuff right so being able to put that whole entire chain of events together and rapidly triage and rapidly remediates it's really the key to success right now next you want to take this one yell yeah yeah so you're just kind of building off of that so say we've identified you know we have we have that visibility we've identified the threat and part of this continuing process is is then what do we do after we found it you know in first couple those any of those steps are is is retrospection and heartbase so looking back at what happened and why it happened and then looking forward at how do we prevent how do we reduce that you know that attack footprint in the future so you know looking in part of that retrospective section is okay again thinking about how do I find this faster you know do we have visibility where is that where else do I see this you know take that time you know you find one bad thing look for those days you quickly go back look for those behaviors make sure it wasn't within anywhere else in your environment and then also push forward and okay what do I need to change my environment to prevent that as a day is there a user rights issue is it a patching issue is it a you know hardening on a server issue where where was the gap in my in my security controls that I can use to to stop or at least reduce the risk around that specific pattern of attack that I've now found and I think they have been the point I would make on that is as you're walking down here environment right as you're hardening your environment it's going to give you less places to hunt right so if you look at technologies like light listing those are going to call a lot of noise in your environment which means that again that precious time that we talked about other the security defenders is better spent on actual alerts on actual things they need to go do to stop bad right yeah so all of those layers right as we build through the visibility as we look at adding i/o fees because those are quick wins right Joe I think that a good job talking about that as we're able to quickly triage and will mediate these issues hey we've got retrospection answering the questions that this ever happened in my environment how prevalent is it we move into hardening right because now we've got better visibility to determine where we should spend our preventative controls okay awesome so we so we've done a lot of manual effort we've created a bunch of process that didn't exist before because we want thread honey now I want to make it efficient right now I want to be able to do this at scale now I want to be able to do it and be independent of say my awesome threatened team that goes and works red canary right you know so you want to make your strategy and you really want to think about this when it comes to threatening right it's got to be a process that exists over the life cycle of your program right so building is a point in time dependent on individuals I would say it's bad you want processes and technology that are going to exist outside of the individuals that are running them because as we all know resources leave all the time especially good senior resources right so a couple of things I'll say about automation if you go to the next one jump first thing I'll say is that the current state of cybercrime shows an extremely profitable market according to a 2014 report by Rand the cyber black market was found to be more profitable than the illegal drug trade in 2011 the UN commissioned an illicit funds report which found that the largest income from transnational organized crime came from illicit drugs accounting for two point four two point one trillion dollars right so now what we're saying is that for 2014 it's actually more profitable than illicit drugs right so so using that to estimate the size of the black market if you took one illicit drug such as a cocaine that's estimated to be an eighty five billion dollar industry Google last year brought in about ninety billion in revenue right so cyber crime as a whole is exceeding the the profit and profitability of Google as an entity and oh by the way it's estimated 1.7 million people are actually employed and this which would make sever crime one of the largest corporations if it was helping some entity on the air so automations become super important right it becomes important so that we can change these economics I talked a little bit about that earlier but it also becomes important to your team because it's scale and as we know senior engineers and analysts hate to do manual activities right we're professionally lazy and I'm okay with that I don't think that's saying that as bad right so look for opportunities to automate manual processes the kind kind of my next cut yeah actually that's a good one so kind of mine my take what my team has always been message do the manual activities five times and the results come out exactly the same put it on the short list automate write some Python code there or hopefully your manufacturers that are out there or allowing you to do this type of automation next and what that's really going to do is you got to prove it out manually right because if you don't you're going to run into issues as you fail your program right so now we're going to talk a little bit about we talk about automation now we're going to talk about orchestration amongst your products right so a optimal space and alert is generated say it comes from a threat intelligence feed in this case you're going to send that over to some type of system to cross-check the content you know typically we call these sims but it could be customers and stuff that that you have in your environment it's going to pull things like the device history the user profile on behavior now now you have an enriched alert and oh by the way humans didn't have to touch any of it so when we talked about uniting man and machine the machine should really be making the humans job more efficient right so now that we've got a cross-check alert for context we've pulled all this information that would have normally been pulled manually right so you know maybe the engineer is calling someone on the on the desktop team to go actually pull that laptop so that they can get the data off right all super manual waste of time in my opinion those should be automated because really what it's about is is getting to remediation steps right so identifying the issue detecting the issue stopping the bleeding through isolation and whatever techniques that you can team uses now you're able to get into telling processes on the endpoint resetting those credentials blocking the IP addresses right that were involved with you know command and control channels those are all things that you should be able to do in as close to real time as possible so so as you're thinking through your strategies for threat hunting you definitely want to make orchestration automation part of it because again that's how you're going to get the scale and overall with empowering to change the cyber economics behind what the criminals are doing it a couple questions that have come in from you came in from the audience here you know first one here do we have any resources that we recommend to support automation or are there any resources that I can teach my that can teach me best practices and then I'll just throw out the other one we can maybe it answer these both in the same same response do you need them to support automation and orchestration and are there ways to set those actions without a thing I'll take the second one first yo you want to take personal yeah so so just keeping it recommended for supportive automation you know I you know kind of the first answer is yes mm is a great great tool for for automation the other thing is looking at you know looking at certain things that anything that you can script so one of the things I tell people you know why I you know why I like having human for the mix is if I can if I can consistently show that same behavior that same action then I can I can script it so some of the actions that we take you know is some that low hanging from things is this consistently shows this behavior as bad okay then I just block that behavior from happening in my environment that's an easy automation can I you know use a white listing tool or a black list a tool to say is it's the software publisher I don't want that in my environment you know some of those kind of controls those are the heat those are the quick and easy automation other things going to get get to no maturity thing you know can i integrate this into my SEM is there you know do I have orchestration tools or you know control tools or other solutions within my environment you know how is my I key controls built around can i leverage some of that type of things do I have a a good you know endpoint management tool outside of America textured capabilities that I can use to to set up some basic things start uninstalling removing and preventing those type of things and working with my IT organizations part of the that process and automate that process in maybe it's a other things that I've used you're building into my alert sources if there's again low low things or things that I know I can take an immediate response can I build that into my health double solution so whatever alerting source or a tool I'm using can I send a quick email over just I know this behavior is bad I know this is exactly what it is here's the steps I just want to send an email right to my help says hey you know this person's computers got this infection you know we need to just pull them offline and go clean them up right away and here's the steps you know there's a quick and easy wind to catch that you know some of the lower hanging things but you can get a lot faster response on that yeah those are good points yeah I would absolutely bring up things like we all will most of the industry uses ms SPS or in the case of red canary that you have a managed end point detection and threat organization that looks at logs and at some point they're going to a scaler to you right so even if you took a simple process like an alerts escalated I should know whether that needs follow-up or if it needs to be closed immediately and looking at your ticket of your iti skill based ticketing systems those should have those capabilities built in to answer the question about sim if I was looking at buying a sim today it would absolutely be a weighted criteria for me that there is some sort of orchestration that can occur in that and I think you'll see the major vendors moving towards that a number of orchestration automation vendors have appeared in the space right to sort of solve this problem of not enough Tier one resources can't get them in a socks fast enough can turn them fast enough actually do you work I won't plug any particular ones but I think you could do a little fruits and find those the other thing I'll say is you should have talent on your team for scripting right to learn that pipe bond whether that's - whether PowerShell I think as I was looking to hire new folks to come into my program and sure my fret hub those are absolute skills that I would require coming in the door so I hope that answers those questions ok so a couple other couple of questions here do you have any guidance on do you have any guidance on automated response actions when you gotta when you don't have a hundred percent certainty in your opinion should you limit automated response to threats that you're not 100% certain oh I said I said I'm going to say my answer is it largely depends that comes down to a risk discussion you know you know is the is the and it depends on what your your automated response is is it you know is it shutting down that you know shutting down the processes is isolating a host and what's the impact of the false false positive and in that effect now instead of balancing nearly that comes down to a discussion of risk which which is the greater risk in that situation do you have other response processes that you can quickly respond without doing that automated to get that you know resolved right away or is there some intermediary so hey maybe I can isolate that host it kicks the high-ticket over to my helpdesk have them engage with that user right away and it's F and assess that behavior and take take action or is this a production server that's hosting my EECOM environment and if I shut anything down I shut down business that case I might be a little bit more leery of you know taking an automated response but I'm also going to be a little bit more you know intent on getting on that box giving you know give digging in that information and validating that so again it's balancing that that you know the risk on either side of that yeah I think to add to that absolutely great of cultural I think when you're starting your program it would probably be a bad idea to rat you all alerts and just start like isolating hosts and turning off services but as you mature I you should be able to highlight some of these things and go we'll look yeah and I love jokes point right so I have a server farm and one of my you know one of my e-commerce server farms gets gets owned shouldn't I be able to take an automated action on that particular thing if I have reasonable assurance that occurred and not take down my website right so through some of this process you may actually highlight other areas that IT needs to improve right so having an actual fun having a load balancing that actually works so you can pull systems offline and maintain up done you know ability is still part of the the security CIA triad and so I always I always look at it that way as I was working with other leaders and IT and other leaders in the org there were certain cases that hey we're going to isolate this host maybe it's an endpoint made it to desktop and yes we may impact one user but here's our reasonable assurance and then to Joe's point I think you you step into production systems that have a big impact to your to your financial bottom line but in the end if both teams are working together and the architects of those solutions are sitting down actually looking at the overall impact you should be able to build that resiliency in there to be able to do those things and still maintain availability the kind of moving on from the McLintock automation side you know we get back into the actual you know intern actual hunting yeah so the actual you know we've gone through we've done some behaviors who triage we've gone through and then we get back to the top which I really think is just we're circling right back to where we started is you know engaging in this you know taking this in it you know as a real-time you know real-time activity you know in as you mature this becomes faster and claps and this becomes very much a real-time activity so you know maybe I've gotten alert automated looking for certain PowerShell activities all right let's jump right in but as soon as we see that we start we've already filtered out you know the legitimate activity we know hey somebody you know administrators using certain PowerShell commands part of our exchange exchange management system of our server management area thing like that we can then look you and filter that out of the view we already know what that looks like so we can quickly kind of shift gears and I do see what other activities is this doing digging into that looking in around that activity and finding that you know that root cause because now we've you know part of this hunting activity is is is not only visibility by you know tools but visibility by understanding your environment so that's the other really great takeaway from going through having your team go through this activity now I understand my environment so I can continue to refine the knowledge I have in my environment what's going on and we can work down through again working down through this whole chain of activity and take take action as we go along you know as we mature this process we can start and all those steps we've kind of talked about we can start taking action around those in real time working around around those around those behaviors and then yeah I was just going to say you know it really becomes about the taillamp right so awesome so which will we being able to quickly quickly triage we drove the visibility now we can see the malicious activity we can scope the attack but how quickly can you get that information back into your program and make changes right so you know agile secure operate you know security operations right so being able to say hey I don't have the logs I need to actually go hunt well that should be on a project list somewhere and it should be on a capital project list right okay step one get two logs right you know so and we'll talk about visit sort of at the close of the free zone but it should really be about you know you're not always going to find evil while you're out there right sometimes you're going to identify gaps in your program but that needs to be tracked and someone has to manage it like a program right so you need to be able to come up with a strategic set of things that need to be done right as a result of information uncovered during hunting and then and then you're probably going to have a whole list of tactical things like creating a process to actually go find evil on a Windows box and here we've got a related question here it's going to similar tangental but I think it's a good question while Phoebe is great in my view for cleaning up the environment with it or threat with it or threat hunting 100 exercise identify let's say malicious items and backups or email archives so in my view yes part of part of that hunt exercise is you're trying to find those behaviors you know your part you're trying to you're identifying something bad going on but then you're also it's going back and rich you know looking at that when did this happen how did this happen identifying that if I find one thing that hey you know this came in through email yeah it's probably one of my action items let's go back and you know look through you looking through those files or look for related emails look for related items and in the environment to identify and clean those up you know I've worked in plenty of our environments where everybody has a PSP we've got huge email archives and everything so we should have some way that we're going through and identifying through you know other maybe other logs maybe it's still going through you know working with the exchange admin working with whatever or the backup admin if we found this file on the server or something else making sure that part of our remediation process and part of our root cause analysis is is working with those other teams I think part of a great security program is a great threat hunting program and a great I our program is engagement outside of the security team do we have buy-in from the helpdesk you may have buy-in from the business do you have buy-in from other IT organizations to to do follow up to to assess to to triage to do all those steps along this process we have to like Rick said it's part of a program it should be a company-wide program security is the one driving it may be doing a lot of the boots on the ground effort but you know if we're identifying gaps or areas and things that we need to be able to do to do cleanup to do prevention to whatever that we need to make sure that all the teams within the organization has bought into that and can leverage that information that we've gained through that visibility through that hunting action yeah and I think you know what I would say to that is if you've seen any of my kind of in-person talks on cutting a culture of threat hunting like one one of the things I talked about is enlisting an army right so looking at those folks like email admins why not teach them how to go Huss in their own environment so that that that now lives outside of security it's a good part of system administration something they should be doing anyway but the chances are they're probably under educated and under trained chances are you know given how many people are hosts email they probably got some cycles to do it right again I wouldn't say that's going to happen day one but as part of your overall strategy that you should look for these areas where you're able to impart that education and knowledge and then give them tools to go do that or give them requirements for tools right as they're looking at increasing email security maybe that button doesn't live with you which which is often the case and lives over 90 cool you have a relationship where you can go to them with this weighted criteria and say from a security perspective there's things we'd like to see you guys do for activities and processes that should exist around this this particular system here I got one more question for us Rick and I think this leads into our next slide really well for someone just getting started with threat hunting what are some of the basic behaviors and things we should be looking for I think that leads into you you know we can I think we can tail that right into your next point here Rick yeah yes so I think it you know the first thing you want to do is sit down and come up with a plan right yeah that sounds a little cheesy but but but I do believe in that I think if you just run off and start doing stuff you're probably not going to have the impact that you want at the time you could probably save yourself some time for better plans right so I think the first thing is to make a conscious effort that this is an activity you need to do because detective controls are currently failing right we talked about the innovation going on in the cyber criminal and nation-states like they're evading all of this stuff right so they're evading your preventative controls so you should absolutely be doing this so you come up with a plan right 30 to 90 days you're going to start doing things like ad-hoc hunting now what do I mean by that you know did a good job earlier about talking about the beginning of the threat hunting chain night show I have a hunch I have a hunch that my see CTO and CEO just got back from China and I have a hunch that there's malware on their systems so I'm going to start with a very limited subset of data and start looking at processes to go find that right so I'm not going to I'm going to choose not to eat the entire pie I'm going to eat it one spoonful all the time right so I think we all know or should know we're high risk users are I think we should also know where our control gaps are right it's a good it's a good program manager you should know where your gaps are so those are the areas that you probably want to start hunting first if you've got good prevention in areas and over time you haven't seen things like compromises or breaches cool you can probably ignore those to start and go looking in these other areas there's some very helpful tools out there like hunting net thus and stern having an incident response summit as a wealth of Education it comes down around how to actually find evil on certain systems like hey behaviors analytics are going to look like this here's how I would go to tech up right you're going to create things like a rapid feedback loop you're going to look at the gaps like hey I don't have visibility and laws I should probably talk to my boss about that or get funding or start the discussion of creating this culture the rapid feedback loop we talked about becoming highly important so that you can actually drive change faster new programs so now you've been doing some hunting you've got guests identified hopefully you're starting to work on closing those gaps and then really you know three months in you can start to automate so I went and looked for evil on two laptops in my environment six times and the results came out the same and the process that I use was the same usually important that you're following the same exact process you start to automate cool now I'm going to start to automate some of that stuff through scripting so I can go through elsewhere I'm going to start identifying my tool sets it's important streamline that one hundred two hundred twenty days to a year again this is going to depend on your environment I've seen environments take three years to actually make a purchase so I hope folks in our audience don't have that large of a barrier to actually get into these tools then but I'm going to purchase an implement tooling I'm going to baseline on the training we feel like it's tougher staff now I get into year two and three and it's continual process improvement like so so being able to say the direct impact of me having a threat hunting process or throwing out a team and threat having tools has led to an increase in prevention automation orchestration more sophisticated hunting this is where the program leaders are going to start to track metrics like number of pieces of evil found right whatever you want to call that metric meantime can detect mean time to remediate are going to be key there and it really just becomes at that point much like any other part of your security program it's the constant feedback loop constant Karen cheating and then hopefully enabling your team by getting them from education dollars and sending them to things like The Sims 400 front summit which is just invaluable right there's tons of turn honey pots that go down at blackhat calm so yes you will have to make an investment in education for your people but that's not to say that there's not a lot of free education out there so so a couple of questions here Rick that's right here so one of the first ones what's the best practice for around ranking the stream of events that are made visible as typical one gets overwhelmed with the volume of signal-to-noise Wow glad you want to take that one for her now you can start it and then I'll I'll jump in for sure okay so I'd say one of the things that we do internally than with other campuses and I think Rick mention this too is is start small you're not going to you're not going to find everything in the environment you know it's the same thing you don't stand up a solution and say give me all the turn on all the things and hope to be able to consume that you figure out you know figure out in some fashion what what's important where do you have your like with mention where do you have known gap okay why don't start there let's give visibility around those known gaps and start working through it the other thing is part of maturity program if you have as you're automating these things as you're bringing tools figure out a way to track metrics on that metrics is huge you know measure measure yourself how long you know what's my fault positive rate from this you know signature from this behavior from this tool you know figure out where and use those numbers to drive one your confidence in a given tool or detection or behavior or whatever and then to where you need an opportunity where there's opportunity for tula tuning hey this one this this this over here is consistently hits almost 100 percent of the time I know when dis fires it's bad cool automate that now you have confidence in that then you know let's work down from there and okay hey this over here is cause amazed constantly noise noise noise noise figure out a way to figure out what is that noise is that something that you can you can tune in is that something that's known within your environment okay then you need to make adjustments for that and that should be part of you know part of the output of this exercise is you know pick up pick a piece work down you orth it down as hard as you can and then identify both the bad things but also the opportunities for bilious reduction yeah and I think secondarily of that too um hopefully you're doing outfit management right so hopefully you know where the case for the kingdoms are and you're able to identify those now now I completely understand because it's often been the case in my programs when I'm starting that other areas of the org maybe mature slower than the security game right so you come into sort of a program on page zero you've got a number of assets you've got a number of people you've got a number of controls maybe hopefully and one of them should just be good asset management right so in a lot of cases you should be able to integrate with your asset management system to understand which systems have more important that the others which should drive your learning and at least what your triaging now yes I completely understand that this will be manual on the part of the security team until IT matures their asset management but the other thing I would say to that is coupling it with the behaviors curve right so just just because I got an alert on well let's use the scenario of my CEOs laptop so my CEO get getting alerts from a traditional ad tool and it says hey this was partially quarantine like so oh something's on there it should really be about the behavior of what it's doing right because just because it's the CEOs laptop it may be a little more important but maybe he doesn't have access to all of the things that say your engineering and development team would like so the behaviors of what the attackers are doing becomes super critical in naturally being upper triage with stuff and say this alert is more important than miss alerts and here's why it's based on the behavior right so so mapping back to kind of the kill chain the earliest you can stop them in the kill chain awesome if you can stop them during recon cool it's going to make their job harder and probably MVA the recipe attack chain but I think those two factors for me kind of like if I had to start a day zero would be like what is the absent what's on it and what is the behavior that the attackers taken on that box and your tools like when you look at eg our tools they should do that for you right so so they should be able to based on these other factors actually triage your alerts based on that so alright a couple more questions here and I think these two I think closer real nice one can be done without a dedicated visibility tool either endpoint and network and then are there any open source tools that you would recommend oh so I will start by saying this this function typically doesn't exist in most security programs right so you have to start somewhere so while I will always stand on the best possible stance is to collect all of the right metadata at the endpoints you're probably not going to be in that state right I wish everybody would buy you know carbon block response throw it out there and and you know get the visibility but we all know budget but budgets are concerns maybe you spent some money on firewalls this year and you just don't have it so you gotta start so so so yeah I would say you have logs sources today you have DNS logs you have DHCP laws you have Active Directory logs you should be logging things from your your network like routers and switches those of perfect areas to start thread hunting without having the complete pool sets that I would argue you need to do it yeah I would agree with Rick there you know one of my favorites you know just from oh you make tactical standpoint one of my favorites is anybody running a Windows environment you know turn on powers analog there were some great logging options that aren't turned on by default in your Windows environment to provide a lot of values a lot of interesting information that you can start using to to get some understanding of what's going on in your environment you know you have to you may have to kind of tune and tweak and you know which places you're turning on certain things but you know maybe in your you know on your CEOs laptop and the keys of the kingdom server you've got PowerShell watch and then when you're seeing those PowerShell events happening within those windows log okay why is your CEO running PowerShell unless you're working for a tech company keep reading its own code you know those those are some you know highly suspect things you know and look again looking for you know getting engagement you know work work with your server team sit and and understand help them you know understand what's going on and maybe you know maybe not finding bad things maybe defining inefficiencies but you know again engage with those other team to use the tools that are already in the environment built into the operating system built in you know built into the network to look for those things DNS logs are wealthy acknowledge you get into you know passive DNS information gathering and everything this provides a lot of a lot of details and information as you can do kind of right out of the gate without a lot of financial investment yeah and you know we didn't list them all but I but I did want to point out thread having gotten that again because because I love when the community gets together and we just start publishing like here's the actual steps to go find this malicious process on this particular system which is cool so if there's any thread hunters out there I would encourage I would encourage you guys to participate because we all can't afford tools and I know especially after meeting with the number of companies that I have I have over the last few years that's just the reality that we're in so the term I like to use is being a baller on a budget and doing more with less so yes absolutely take advantage of scripts that are out there absolutely take advantage of whatever you can to drive the disability but I think what you'll find is there's there's a reason that carbon block made a product and it was simply to answer those questions that I talked about earlier that I couldn't quite get 100% assurance on my end point so I think we're almost at a time I did want to thank everybody for joining us today I also wanted to say hopefully we'll see you all in a couple of weeks if the thread hunting and I are summit in New Orleans that sans is hosting wealth of great information that you're looking at building a thread on each program if you're if you're looking for process and frankly if you're looking to interact with other folks doing the job of IATA and thread hunting a daily basis and get to share some other information I can't reiterate enough how great that event is so thanks everybody for your time and we'll see you soon thank you and just one other thing for any questions we are unable to answer before a will look out for a follow-up email with some of those questions that will okay and respond up offline

Channel: Red Canary

Views: 19,095

Rating: 4.9219513 out of 5

Keywords: endpoint detection and response, managed endpoint detection and response, cyber security, information security, threat hunting, Mitre ATT&CK, ATT&CK, Carbon Black, threat research

Id: 58lS_pEElt8

Length: 56min 42sec (3402 seconds)

Published: Mon Mar 27 2017