CSS2018LAS8: Incident Handling Process - SANS

[Music] welcome everyone hopefully you're all here for incident handling process my name is Brian Ventura and I'm a sans instructor I also work in the public sector I work at the city of Portland as an information security architect that's my day job so I have similar background as some of you and and deal with some of the same issues so hopefully this will be useful for you I've got my contact information there and also the link to my courses I teach a number of courses for sans and they're posted on that link at the bottom what I'm going to be talking about today is primarily based on or the sand security 504 class and that's the incident handling class and hacker tools it goes into a lot more depth than this this material is basically the first day of material I'm going to give it to you in an hour and it takes a whole day for the course so there's a lot more to this than what I'm going to cover and I believe that after this you get access to these slides the slides have lots of details in the comments so I highly recommend that you read through those after you get this maybe in a week or so when they send it out but lots of really good information and so let's get started on this so sometimes with Incident Response it feels like we're falling if we if we don't have a plan we have nothing to grasp on to it's it's a scary thing and we don't want it to be that way so instead of having this feeling every time someone says by the way I think that we have malware or we have a phishing event and they've ransomware to our whole environment those kinds of things instead of feeling that fear and and uncertainty if we have a plan then we can pull that out and that can be our safety net to help us walkthrough remember all the steps and do what's supposed to get done so real quick incident handling for us is basically having a plan so a plan to deal with adverse conditions something bad happens on our network we have a reference manual that will tell us go talk to these people bring bring in those teams look at these kinds of evidence our or logs and we'll get through this there's a lot more to it and that's what we're gonna start digging into anybody have have a need for me to explain why you should have Incident Response with how things are going today and the big news items I think it's pretty obvious but some key points that we have up here most of us have some form of compliance and we have to deal with most of those compliance regimes say you need to report when you lose data or when an adverse event happens the other thing is that Incident Response by itself is a form of due care and that's actually a legal term like the idea that if you were ever sued for losing information or something like that one of the questions would be what would a prudent person do in your situation and that's due care so a normal person that that knows a little bit about security what would they do when you're you're given whatever that bad thing is think about the big ones Equifax you have a struts vulnerability someone broke in through that that when they go to court that someone's going to say what would a prudent person do if they were working at Equifax right and we want to make sure that in our plan we have a reasonable amount of incident response that we can show that we've done the best best practice kind of thing that we need to do and then we probably have our own policies that we need to meet some definitions we want to understand the difference between an event and an incident an event is anything that's observable so someone logs into a computer a computer reboots those are events are they malicious we don't know yet normally when in art when a user logs in that's probably not malicious normally when someone reboots their computer not malicious but we but these are things that we track we probably have logs with this kind of information right in our organization's that's what an event is on the other side we have an incident that's where we take that same event and we realized that it's adverse it's bad for environment maybe this is an attacker trying to get into our system logging in instead of Brian Ventura logging into the system working at the city of Portland we have we've we've had a recent rash of fishing and no no one else is having that problem were the only ones but but seriously for us we had an incident where three weeks earlier someone was fished and we didn't catch it then that person sent out invoices to everybody in the room or in our organization a large percentage the unfortunate thing is this person does that for the city so they sent out invoices many people were like well I don't know why I'm getting an invoice right now I don't need one but I guess it's for me and they opened it so that so it hit us really hard the fun part about it is that person that that account people would respond to him saying I don't think I'm working with you on an invoice why'd you send me this and that person would respond oh no that's for you that is definitely your your invoice go ahead and click on it it's good it was the attacker logged into our system so logging into a single login event is an event but then in this case this was an adverse event because that wasn't my real user logging in so there's a kind of an example of of an incident a couple other examples system crashes boot logs Network spikes those are just events they happened on our network they may be something that we want to correct like the network spikes maybe we want to correct something but it's but it's not a problem that it's not a security problem then the incident is that unauthorized use of another's credentials well that's what happened to us during that phishing sniffing wireless traffic they're trying to see if there's a way to break into our wireless if we can see them doing that that would be an incident and then of course malware is going to be an incident all right we've gone over enough material that I think it's time to test you you ready didn't know that it was doing that did you all right seriously this is an example of some log output and don't worry about getting this right or wrong but let's look at this the top piece of information is some snort put anybody used snort or the snort is an intrusion detection system so it has noticed a malicious packet coming across our network it's identified it as a microsoft iis web server attack looking for VT i underscore INF which is an informational page that's standard in microsoft systems so that's what we see on our snore snort output it's an event we think it's malicious so we're gonna investigate for a little bit deeper so then we look at the lower information and we look at our web server logs to see what the web server responded to the web server says hey I'm Apache it happens to be in the log and I have a file does not exist message for you when someone went to the site and tried to do the Microsoft IAS exploit or attack that the response was that the Apache server said the file was not found so from this much information we can actually decide whether it's an incident or event so what do we think what's that event anyone else no one else okay so why event okay yeah so that that's correct it's an event because the attack hit the wrong kind of server was looking for a Microsoft server it found an Apache server not compatible doesn't have the same vulnerability so this isn't an incident that we have to dig into so let me change this scenario I didn't mention this but you see the IP addresses up there 63 209 is something on the internet not my network tend out whatever that's my network so you can see that it's coming from the outside into my network this is probably a public-facing website so it's okay that they're talking to it but that's what's going on there now let's change it and say it's instead of 63 209 now it's my onna I pee space so it's 10.0 dot something doing the attack doing this malicious connection does that make it an event or an incident does that switch it for us well for me I say that's an incident because my internal machine should not be attacking other machines even though they're not successful it it shifts it the internet I can't control all those people I can't stop them from a trying to attack me but I can block them with my firewalls and an intrusion detection systems I've got solutions to help protect myself but internally I have acceptable use policies and things like that that says all of you that are in my organization you're not allowed to hack why are you hacking so once that's an internal IP address that's initiating it I'm gonna call it an incident so that's kind of how we can see this so that's that's kind of the background in this now if now that we understand what an incident is how are we going to deal with it and sans has this six step process that we're gonna go through this aligns with most of the best practices out there so if you use NIST or use something else they're gonna have similar phases that you would go through in your incident response they may change the names a little bit and that's unfortunate that we have different names so if you already think about these kinds of phases and you use a different name just substitute the name my name is not better than your name I'm just these are these are what we're calling them here and I'm gonna go into detail about all of these but the concept is we're sitting in preparation and identification right now we're doing preparation right now because we're learning about how to build this process so we're doing our preparation we're figuring out what we need to do and we're also sitting in identification mode probably right now because back at the office you may have an intrusion detection system you may have a firewall that's logging all the connections you're doing you're looking for an attack right now in your organization now you specifically but the tools that you have and the staff that's that are still back at the office doing that grind work that we don't we get out of for today all right so the identification is happening but then once we identify something we're gonna try to contain it we're gonna try to eradicate it we're going to try to recover and then very important we want to do that's lessons learned so let's dig into those so in preparation again the goal here is to be prepared to know what we're gonna do to respond to a an incident and we there's some components that we need to think about so one of the big components is people and there's two pieces of this people that we want to think about there's our end users all our end users are some of the first people to notice an attack and they may they may be part of our response to attacks and so we want to give them education and of course I'm saying so I'm gonna mention securing the human right that's the that's the training that sans provides to end users you can buy modules little videos things like that there's other vendors so if you use a different vendor that's great but you have end user awareness or end user education where you're telling them how to respond to phishing emails and things like that turns out that most of the attacks today are these phishing type attacks where we're asking our users to please participate in the attack and allow people into the organization and the reason for that is that server side attack we've got really good controls over we've got firewalls we've got intrusion prevention systems all kinds of things stopping those and so they've switched to that send a message inside and hope that person clicks on something hope so hope they run my malware hopefully give me the credentials so I can log in so we definitely want to try and train those users the other people that are important is our Incident Response Team so who is on an incident response team alright a couple hands are out there in when you think about your incident response team it may be cyber security professionals like us that are doing that it also on my team I've got my legal department because they've got to tell us what we need to do to meet legal compliance and where we're gonna run afoul if we don't do it I also have at my city communications expert this person talks to the public all the time they tell people about the latest thing that happened with maybe what we're doing with our roads or what we're doing with housing or whatever they also talk about cyber security they are the expert at talking to people they know nothing about a cybersecurity but they know how to talk to the public I'm not supposed to be the person to not talk to the public now I'm kind of doing it here but that's because of a different role I would be afraid to talk to the news because I love giving information to him like oh yeah it was like this and they came in that way and the phishing email and the my communications like no stop it stop it stop it so I'm not gonna talk to the public about these kinds of incidents I'm gonna have that communications person so we need to think about the people that are in our team that's responding to these things and make sure that they're aware of this so we tell them hey you're a part of our team and then they can ask the questions of well what do I need to know so that I can do my communications so I can do my legal the other thing we need are these policies and one of the big ones is a warning banner now everybody when you log into your computer if it's Windows you do a control-alt-delete and you get that pop-up and it's a big long paragraph right and you have to press ok anybody read that no one reads that well they're supposed to write and that is kind of a just-in-time policy to let people know that we are going to do things on their computer we own the data we're gonna monitor you all these different statements that are in here are useful to have in that policy and that banner that you give to people because it's so they're clicking ok yes I read this which means that they know that we are monitoring their systems we're looking through their email if if you do that you you that they're not authorized to do bad things in your network they're not allowed to be that internal hacker my team actually does have a letter from the CTO our lead of the technology department and that says that Brian is authorized to do hacking inside the network because that's part of my role but not everybody has that letter and even though I have that letter I can't do anything I want I need to document what I'm about to do tell people go through change control actually a sometimes for some of the attacks that I'll do I have to go through change control so they know it's coming but this acceptable use banner is is definitely a part of my Incident Response it gives me the ability to do some things so then we got we're gonna have our what we're going to do during the incident so we need to make sure that these are the team members and they have the resources that they need these are the tools that we're going to use and they're in the right hands and things like that so we need to have a really good understanding of the different parts of an incident and document all of these things a big thing that I wanted to call out is this concept of going back back to paper and pen in an incident it's highly recommended that we take notes on paper and there's a couple different reasons for this I know we're electron we're in this electronic age and we want to do the computer thing and it's a great idea it makes us much faster we actually want to slow down in an incident incident we do not want to overlook things we want to look in every crevice and cranny and corner to make sure that we've identified where the attackers are and what we need to do to get them out of there and slowing down and writing on paper can help with that we don't rape make rash decisions it's also very useful that the paper copy isn't online and that therefore our attackers can't see what we've already figured out and what our plan is I've actually heard a story of an organization that had been attacked there there under currently under attack the attackers are in the systems they've identified what they think to be the entire scope they say these 50 servers need to be rebuilt these 30 workstations to be whatever the plan is they've identified it they're sitting in their war room the doors closed and locked there they're ready to do their incident they're talking about okay you're gonna do this you're gonna do this you're gonna do this they make their plan they walk out of the room and they find all of their their systems light up the attackers have shifted to other computers why'd that happen the the yeah so somehow the attackers found out something how they found it out they had they owned the voiceover IP system and they turned the microphone on in the camera in the sorry in the conference room phone in the war room conference room which you want a phone in there that's how they did it so kind of scary they didn't know that they were in the voice over IP system and therefore their plan was out the door then they had to go meet at Starbucks you know to do their war room because at least the attackers aren't over at Starbucks but but those are those are serious things that could happen with small attacks that might not be the case but how do we know whether it's smaller until we've done that investigation the last thing is that bringing paper into courtrooms judges and juries understand that they can't act they can feel it and touch it and and they understand physical evidence much better than electronic evidence so it's useful to have it in paper okay so what about the identification phase this is where we're we're running our tools we're looking for things we're looking through logs there's lots of different places we can be looking at things with our firewalls on our network perimeter our host-based firewalls or host-based logging system level logs our application level logs all are different places where we could find things and you may have a number of devices we have passive devices that capture all network traffic and do analytics on it and we have logs going to a central storage system sim is the common term for it but it's an event management system that sends out alerts when certain things happen all of those things is what we're talking about in our identification phase and this is an example of network detection we're running we're looking at the network packets and we noticed a bunch of what's called a syn packet so all these s is up and up and down that's the that's the first packet of a communication and a TCP communication and it's weird to see the first packet without more packets going back and forth so the fact that we see only a single packet and the port keeps changing 21 is telnet 5900 is VNC etc each of the we see that they're changing the port each time they connect to us but it's coming from the same place each time then we're thinking that this is probably scanning right so someone is scanning our network if this was an internal machine scanning that's a big deal because who should be scanning besides the security team with our scanning tools we don't want general user scanning but if it's an X if it's an external IP address then that's them hitting our firewall on the outside so there's one example you may have seen things like this where the Windows Firewall says I'm not going to allow something to happen this is another area where we can get intelligence do our identification our malware solutions would give us different detections for for malware or this one actually is a potentially unwanted applications so it's not officially malware but it's a fishy weird software that the system is talking about then we have application logs this one here is talking about all these super users logged in and did things and we're looking at the application layer this is the biggest challenge for most organizations because the IT teams manage usually the operating system level so they and the network but they don't manage the applications necessarily so this can be challenging in different organizations and think about cloud services do you get logs from that we're having at fishing event that I was talking about we're office 365 that's where that was happening and we had to get into the office 365 logs do you have access to that you know maybe maybe you don't currently have access to that kind of data and having that access is very important also our end users are ultimately sensors for us and that's why we do some of that awareness sessions with them is because they can report that is how we're catching our fishing right now the good news is we're averaging approximately five to nine fishing events per day at the city of Portland so this is a big deal for us right now we're we're we have a proof of concept of the solution to resolve that for us but we're getting inundated by fishing and our end users are finding it for us we have a lot of tools that help us contain it really quickly but we have we don't have the tools to find those emails today we have a spam filter this isn't spam phishing is different than spam our spam filter doesn't catch enough of it well and the fact that we're getting five to nine different campaigns per day shows how poorly our spam filter is doing so so we're bringing in a phishing solution our helpdesk is dying they're they're hating us because they're getting all these these calls for this and and that's why we're trying to help them with some kind of solution sans has a ton of resources if you have how many people know what sans is been to sans training but a few of you okay sans has a ton of information for free and we have our courses of course and the courses cost money highly encourage you to go to him of course but we also have reap free resources so that link at the bottom of the screen is where you can find these little cheat sheets to help you go through this but if you search for cheat sheet and sans and you'll find cheat sheets for all kinds of courses that we provide and there could be a cheat sheet for instance the tcp/ip stack not really incident response but maybe that's what you need and then these ones that are specific to incident response so I highly recommend you look out look for those because there are a lot of resources that we have there's also all of our reading room papers all the white papers that people write our sorry gold papers that people write anybody who gets a certification can make it a gold certification by writing a paper about that certification and that gets published back to the community so sans holds that information for you but it's out for everybody so you you write your paper and then anybody can read that and they can get the information that you have done in your organization so they're it's a great resource highly recommend you look through that some of that stuff okay we identified an incident sometimes it feels really bad to actually call it an incident oh boy what's gonna happen we're gonna get in trouble we're gonna have to pay fines it's it's horrible that's not really true so declaring an incident incident is very important because that kicks off the investigation and the determination of how bad it is just because you say it's an incident doesn't mean that you're gonna be in the news like Equifax or like Target or whatever the big companies are that are getting hit declaring an incident might just start the team working so that you can identify these things and clean it up before that breach happens so I highly recommend people think about it and have very quick response into the incident when it makes sense so you do a little bit of investigation but then you determine it's an incident so that you can get the bigger team together if you ever learn later that it's that it's not a big incident you can turn off you can shut down the incident just say we resolve this we learned that Brian actually was in Nigeria logging into his system and he did send those emails okay that's fine and we can close the incident as resolved it doesn't have to be a big event so that's a big one for us once you've identified a and and declared an incident you're gonna start doing your investigations getting your information getting all the pieces together and you're gonna want to move into the containment and containment is a short-term solution for making sure that the attackers don't finish their mission maybe stealing your data or we're getting there ransomware to run on all your systems you're trying to contain it you're not getting them out of your network you may not even want to alert the attacker that you're that you're on to them so you may duplicate your network and put them in a test network where all the devices that they are currently resident on are in that network but your sensitive information is not on that network so that they the attackers are still doing their thing they're trying to dig in deeper but they are walled off and can't do the bad thing one of the obvious ways to do this is that you may tell desktop or somebody if it's a workstation to go out and pull the network plug so that the machine is no longer on the network that does alert the attacker if they're actively working with that machine that the machine went away they don't know why it went away but they they're gonna guess that you're on to them but it stops that machine from communicating back out sending data out whatever that machine might have been doing the reason why I say unplug it from the network and not power it off is that powering it off remove some of the volatile information from memory specifically memory images so we want to make our take our forensics images in this phase and unplugging from the network allows us to get that memory forensics but still blocks the attacker from doing more damage so that's the kind of things that we're talking about here there are a few sub phases or things to think about while in the containment phase there's short-term containment which is kind of what I was talking about with pulling the plug there's system backup which is that getting the forensics images and I'll talk a little bit more about getting forensics images in a minute and then there there's the possibility of long-term containment in an incident incident this is hard for me to conceptualize as a technology person but now that I've been doing Incident Response for a while I understand it better what do you do when you find out that malware is actively running on a machine and that the attackers are communicating with it and it's attacking other parts in your network the default might be quick let's run in there and unplug the machine turn it off whatever contain it and that's okay as a response but we're missing a keeper person or team in the decision-making process I haven't talked about yet and that's the business owner the business owner owns the data owns the process and they're ultimately responsible for the data and whatever damage that might occur within this incident and therefore they get a lot of say on what we do at the incident and they may say leave the system running don't play with the attacker let them attack and I don't like that right I'm a technology person I don't like that but there's legitimate business reasons for instance you're a hospital there's loss of life possibilities if you turn off those machines we run a 911 Center turning off 911 not acceptable for most people think that they should be able to call that anytime right I mean I've heard that that's what happens but seriously you know those systems are mission-critical we can't shut them down so we negotiate with those business owners and say well maybe we can split your network and take the infected systems offline giving you a few systems that you can continue to do that 911 processing or whatever you're doing I don't know but but we have to talk to that business you unit owner to find out what's what we're gonna do with that and that's where we might fall into this long-term containment if the business owner says it's mission-critical we have to keep these systems running we'll design a long-term containment to say okay we're gonna duplicate networks and we're gonna build out an infrastructure just for the attackers so that we don't have problems so things to think about there's a lot more to this but that gives you a flavor for containment I talked about the teams there's some incident forms in the middle there sans does have some sub generic forms they're simple but they give you an idea of the the initial information you need to capture for the city of Portland we've written our own incident response forms they're a little bit longer than the ones that sans has but gives us a place to put in all the pertinent information who owns this asset who's affected who's on our team and what the timeline of events are we're keeping that information in our Incident Response form we're still doing our notes in in a notebook another key component to your incident response team is a communicator so upper management is going to want to know what to do what's going on what how where should they make it make a decision how bad is it they're gonna continually ask you the person doing the incident response the hands on the keyboard the person on the ground they don't have the time to keep breaking out and talk to the management because they're trying to contain they're trying to eradicate they're trying to do all these phases so you want to be have a clear way to get information backup and for the city of Portland we're set up for that because my team is a GRC team we do governance risk and compliance we don't do operations for the most part we run our vulnerability management system but that's about it and so we have the ability in an incident to be an incident commander which means that we're telling people go get some forensics images did you network team what have you found whatever you know keeping the project running or keeping the incident response running and we also have that job of being the communicator so management will come to me and say where are we with this and I'll say this is where we are so far I have a check-in with the technical team in an hour and I will get more information they're feeding me information as they go and then they can send me information in a really unformatted way they can just say found something or did this and then I can translate it into something that management wants to see here so that they can make a really good decision and so I'm that middle person going back you know back and forth between all those different groups I definitely bring my legal team in immediately the city attorneys to be able to so that they understand what they what their part is so that that's a very important piece as we're looking at the system at the incident trying to figure out what's going on I hinted at this in the beginning we want to keep a low profile we want to make sure that those attackers don't realize we're on to them so we don't want to jump on the machine that they're on and start running these machine these commands that make noise on the network they may be monitoring for this waiting for this so that they know now we need to pivot somewhere else so we want to use some of our passive tools look at our logs for the city of Portland we have something that that captures a lot of our packets on our network so we go to the network capture solution and we look at it and say oh I see that the attack came in at this time and then that machine talked to 20 other servers we had a lot of in ransomware events a couple years ago and the this packet capture solution was awesome for us because we would identify that Bryon's computer has ransomware but we don't know what Brian Brian's computer connected to after that so the network team would find all the packets that came from my computer and they would say oh I see the Brian went to file server 1 and file server 2 well those two file servers might have encrypted files that we have to clean up so this we'd give the server team that information they would go look on those servers see if they find any encrypted files so it was very useful for you for us if you don't have a full capture packet capture solution maybe you can capture some NetFlow information that's the same kind of things without the full cap packets it gives you the flow of Brian's computer talk to file server one you just don't see the stream of data that I sent which is enough information usually to be able to work on the incident okay I'm gonna have to move a little faster because I'm a little behind here short term I've talked about this that some of the examples disconnecting the network cable pulling the power if we have to maybe using the network switches to turn off ports if we have network management tools to turn off the ports it's a lot faster than sending a desktop person out there to actually sit at the keyboard and pull the plug applying filters changing DNS names things like that that might be able to fix our solutions one of the things we've done at the city is we've blocked URLs but for those phishing emails as soon as we find out what the URL is we put in a block so that no one can go to the command and control channel after we figured it out I talked about talking meeting with our business unit and I'm talking to the business owner we definitely want to talk to them about it and if there's a disagreement they win it's their data and that one's hard for me to swallow and may be hard for you to swallow but that is true because they own the data our job is to inform them so we tell them here are the risks here are the dangers you may become Equifax you know we can use those kinds of examples we can tell them how big the problem is it's bigger than a bad bread basket so therefore we should do something about this and then if they say nope still it's more important to save lives then we say ok and then we work with them to build a plan maybe for that longer-term containment think about your external agencies that you were do you need to work with thinking about denial of service if you've heard of distributed denial of service I think there was one a couple weeks ago that was exciting it used what what's it called memcache was the new attack and most distributed denial of services we are in the the megabyte range or the or gigabyte range now this one's in the terabyte per second of data that it's sending to these there's no way we can fix that on our end so we need to talk to our ISP and say can you block this before it gets to us can we go upstream further and block it further away from us for that kind of an attack so talk to talk to your ISP and in our plan we should have that come that connect the sorry that contact information in our plan for for me I also have the FBI contact and I have regular conversations with them because I want them as a resource on my incident response team if I ever need them they're not in most of my Incident Response projects but they but I do have them in my list for you you should have those same ones we have an external company that will do our heavy duty Incident Response if we ever have to do that I have that contact we're part of the MSI sac multi-state I sac the MSI sack information is in there they do forensics for us so we have those things in our plan so that when we need them we can call them and something to think about is you want this plan in paper on a shelf as well as electronically because if they've shut it down your network was there it wasn't there a city in Ohio or something like that recently in the last three months that they shut down their whole network I sent that to my boss because I've told him that we might be next but they shut down the whole city Network because they had ransomware dand and encrypted enough of it so we want to have a paper copy of this so that we can get it offline I have a USB with me now that has our incident response plan and all of our information because if it's a catastrophic failure in Portland I'm the only one left they can they can do Incident Response and and I'm in an area where I can get to the cloud and I can contact these people and and do this incident I talked about for our forensics we want to dig in a little bit deeper we use ftk imager is the free software just for the imaging to grab memory forensics and we also have a solution with a write blocker that so that we can grab the disk information and guarantee that we didn't alter it while we were copying it these are all important when taking forensics images you always want to do the most volatile data to the least volatile data so you need to think about that this is what right blockers would look like there they physically stop the right channel so that it's impossible to write data to the infected hard drive while you're copying it so it's useful tool and then we want to think about whether we can continue operations etc and make recommendations for those long-term and I talked a little bit about that okay some examples of long-term term actions we may patch systems patches specifically patch systems that weren't affected by this but patch systems that are inside of this so that we can limit the the spread of this attack we can do we can put in intrusion prevention solutions change passwords things like that we want to continue to talk to people find out in my phishing example where someone's account was was hacked when we found out about that that person is not in trouble because someone attacked them they're the victim here we don't want to do victim blaming especially while we're running our incident but even afterwards I argue that we don't want to do victim blaming because this is hard just two weeks ago I clicked on a phishing email I was talking to the person that's sitting in front of me my screens here I looked down at an email thought it was from my boss clicked on it opened the thing looked at the person huh yeah go ahead oh I just clicked on a phishing email so I had to report myself and I had to do I did full forensics on my machine because that's what I do but but but I had to report myself report the phishing and actually do many incident response I quickly figured out that I'd just downloaded a file that I didn't run so I didn't get infected but I did full in forensics just to make sure that it wasn't a problem but we don't want to blame people it's the big thing that I wanted to get across we then move once we've contained it once we understand how big this problem is we really want to understand all the systems that are infected before we go into eradication many people want you to go into eradication as soon as possible and I caution you to make sure you understand all the machines that are affected otherwise you're gonna do what's called whack-a-mole if you've heard that the game for you where you knock them down and you have to keep hitting around that's what you'll do with your eradication you'll say oh they're on Brian's computer clean Brian's computer reinstall it do all kinds of good stuff now they're on Sally's computer okay let's go clean that one up there back on Brian's computer you know that you're just gonna they're going to move around so we need to find all of them before we start doing something to ultimately restore from our systems we want to wipe the systems completely and restore from backup that's going to be our best way to do this because that way if they have some software on there that we don't know about it still gets cleaned off and I highly recommend using gold images for your for your deployments of both servers and workstations so that it's quick and easy and you have a known good state that you're coming back to in the case of the city with a lot of our ransomware that we had a few years ago we found that we would re image the workstations that were infected and then the file servers we'd restore from our backup solution and we'd be back to to running and we wouldn't have to pay any ransomware or anything like that so we got lucky that way if you can't rebuild the system for whatever reason you can remove the malicious software and make sure you do some of these other things one thing I cautioned though we had a ransomware event and at the city two years ago and we decided to send forensics off to MSI sack just for the fun of it so I did low level forensics real quick look through it and found the malware and said I did it you know and we're good to go but we're practicing so we send the forensics off and they get it back they get a report back to us about a month later and we've already recovered everything in this we're good to go I get the report I'm reading it oh yeah it was that malware that's what I found seven other pieces of malware huh I didn't find the seven other pieces of malware that I wasn't looking for them I wasn't paying attention because I didn't care as much did it add a dropper oh that's how they got there's a the first application they give you is typically just something that downloads more software it's innocuous you don't get most datum um antivirus doesn't find it so I'm like okay that one the next one a keylogger they had all our passwords of anybody who logged into that system that freaked us out we got lucky in this incident that the person that was infected said here's my password so you can fix my system I'm gonna go to lunch because they told us their password it's our policy we changed their password utin someone else knows your password it's me I know your password we got to change your password so we changed that person's password but when we found out about the key logger we realized all of our system level accounts could have been attacked the ones that install software and install patches and things like that so we went through a process to change those passwords and that was a major endeavor because that's all the systems have those passwords right also in our eradication we we're gonna want to put in barriers to stop them from coming back in so we're gonna do some of these things apply firewall rules things like that change DNS names anything that we can do to make sure that our defenses are stronger and the attackers are not as likely to get in and then last in eradication you want to do some vulnerability scanning that there's there's a number of tools for that and highly recommend that you scan your systems before you complete your eradication stage so that you ensure that there are no vulnerabilities left for the attacker to get back in all right so we're nearing the homestretch here we feel like we're ready to recover from this and the goal of our recovery is to bring things back to business normal business process so when we're doing this you want to you want to think about the fact that you may not have cleaned them all the way off your eradication may not have been successful this is us testing it so as we bring them into production bring it back to normal business we want to monitor heavily so you might choose a time when it's going to be easier to monitor and so you can get kind of that baseline of what's normal after the event so think about those things the system owners again we need to consult with them or the business owners we want to consult with them make sure that they understand when we're coming back into production that they're comfortable with it etc they make the call and we provide advice that's our jobs definitely want to monitor definitely want to watch our logs make sure that the attackers are not coming back into the system this is probably the most important phase for me lessons learned after the event everybody feels like oh load is pressures off we're done we can walk away we did it we were successful and that's great but probably through the process there was some heartache there were some arguments there were some missed decisions there was things that we had to change on the fly let's document those things that the process might not have been perfect and every time we run an incident we've changed the process we enhance it in some way something we didn't think of so we definitely want to do this lessons learn we want to write a report that has all the information about the process not the incident now we're talking about the process hey Brian started the process and it took three days before his boss realized that it was a problem and started actual incident response you know that we're not playing blame we're just saying it took three days to initiate that that's too long maybe our desktop team didn't use the right tool and the forensics images got messed up or they didn't know to use the tool or they didn't know where to find the tool etc so we're just talking about what happened and we're building a report of what's going on we definitely want to meet as soon as possible afterwards because it's fresh in our brains even though we want to walk away and have our vacation and not think about incidents anymore we want to get that information as soon as possible while it's fresh in people's minds and then of course we want to fix things if there's anything wrong out there that in the process we want to fix it update our plan maybe communicate to different teams that we hadn't brought into into it beforehand and make sure that they're ready for the next incident and that's our phases so any questions a ride seven phases of did it and how many phases did we have actually I showed I'm missing a whole phase uh-oh so let me see I don't know if I can go back that far that fast let me break out interesting I apologize for that um yeah there you go yeah we've got six phases in there and and we put seven on that yeah definitely I mean I'm English is not my major that's why I'm a computer geek okay so I apologize for that but these are the lessons are the phases that we identify and again NIST and others have different phases so that could be a problem anyone else have a question yes yes a lot of data it's a lot of data so we don't have a lot of time I can talk to you offline a little bit more when it's not on the recording but it's a solution that we bought and it's very versatile and powerful it's actually a diagnostic tool for the network team it's not an it's not a security tool originally and they used it to diagnose weird problems we have a pretty elaborate network at the city and so that that's why they had it and then we recognized that wow that's powerful for us we're going to use it in our incident and so it became part of your instrument response plan yeah has this worked for you is this useful is this something that that you can apply again I highly recommend there's more information in the notes and and when we get to that because you know we had an hour and and we're about the end of that so any other questions Wow that either means that this was boring or that I did it perfectly and it was it was so obvious and now you know it all all right still digesting that's fair let me put you on our last slide if I can just so you can get the my information in case you want to contact I'm again I'm in the public sector so all of you that are in the public sector I do a similar job as you so I'm definitely interested in communicating talking about these things at a deeper level I'm doing another talk right after this so I will have to run right after this to the next room and that one's on the cybersecurity framework so but I'm interested in talking about that as well thank you and that's all I have all right [Applause]

Channel: Public Sector Partners, Inc

Views: 25,528

Keywords: Public Sector Partners, Russ Hicks, Russell Hicks, PSP Forum, Executive Sponsor, Cyber Security Symposium, 2018 Cyber Security Symposium, Cyber Security, Cybersecurity, CSS2018, Orange County, Jacob Margolis, George Khalil, Southern California, SoCal, SANS, Brian Ventura

Id: wP08bImLU6Q

Length: 49min 54sec (2994 seconds)

Published: Wed Apr 11 2018