Investigating a Security Incident with Riverbed NetProfiler

uh my name is john murphy i am a technical director with riverbed i have been specializing in network security i have i started off actually in robotics and then moved into network security using machine learning techniques uh specifically with netflow to detect malicious insiders to detect botnets and other problems and for the last decade or so i've been helping riverbeds customers um and lots of folks use flows to find security problems like uh crypto miners whoops all right so what i'm going to be using here is a tool called net profiler uh net profiler is a flow analysis product where we are taking uh flow uh netflow is a often described as a phone bill for your network um i think at this point more people know what netflow is than phone bills i feel a little old sometimes making that analogy but it is a really useful one so while we do have products that are looking at individual packets very often all we really need to know is who who is communicating with whom at what time and just how much data is passing back and forth we don't actually need to go down to the individual packets or make a very expensive data request for every single uh thing that we're looking for and we can make use of deep packet inspection at the flow generation level to pass up information about application that we can use in our investigations so we can get this data from a very wide variety of devices such as routers switches firewalls we can also get flow like data from aws and azure and in terms of flow logs and we bring it all into a central monitoring infrastructure that profiler which allows us to get that global view in a single pane of glass so that we can actually slice and dice this in a variety of different ways and because the netflow format is very compact we can store everything at full fidelity meaning that when i am going in i can zoom way out and i can see you know the the bytes per second for my whole network either right now or going back over a month or so however long i've got disk for it or i can drill down to very very specific uh filters and pull out a single dns request out of all of that going back again months and so this is useful for a very wide variety of techniques but for me this is a great data source for security forensics because uh when john comes to me and says hey i see evidence of a crypto miner i am able to go back and look at the traffic that's actually being passed regardless of whatever the hacker has done to the host itself they may have turned off logging they may be faking logs or other telemetry but they can't fake the network data and i've got all of it so i'm able to go in which i'll do in just a minute and actually pull up what has been happening in addition you know that's very manual we do also have a variety of alerting capabilities a lot of it is geared toward network performance but there's also a fair bit geared towards security and that'll come into play here in a little bit too so i think it's about time that i dive into the actual demo john i i just want to make sure i understand so this is an on-premises netflow collector that runs on a on a custom virtual appliance or or what does that look like so this is going to be it's going to be deployed on prem typically it's going to be an appliance you can also put in gateway appliances to collect flows anywhere your network happens to be or virtual network happens to be so that you're able to collect that compress it and securely send it to that central location for analysis okay thank you so what we're looking at here is my profiler dashboard now everybody is going to have their own sets of dashboards everybody has their own quirks what they like to look at first thing in the morning me i like to look at alerts first thing in the morning you can see we've got a handful of fun things you can also see down there that we've got our threat feed the advanced security module threat feed is a little bit like a blog we are going out regularly and pulling in information from news sources because as you all know it there's a lot going on and very often uh you know i'll go to ars technica which uh will have information about today's new hack unfortunately by the time something gets into the news these things have generally been going on for weeks or months they will identify for example ip addresses or ports things that are very nicely visible in flow data but by the time it gets to me again it's in the past this allows me to you know go in and search all of my data for all of those indicators going back however long i want to so that when i see a report hey back in march this thing was a thing i can go back in march today however i have a a little bit of an advantage which is that john slacked me the internal ip for that jfk host so i'm just going to type that in and i'm going to pull up just a quick report here now we've got a lot of information going back so what this is going to do just to start off i want to see what's going on with that host that may or may not have been reported into other tools so now you can kind of see here we're not looking at a lot of traffic on this host you know what is that 80 kilobits per second um you know the the the traffic that it's serving up is not all that much uh let's see so what is it connecting out with so i'm you know i'm mostly seeing web traffic google microsoft oh okay so um what we've got here is monero which uh is indeed a uh cryptocurrency not bitcoin but uh related yeah uh so john you you had it right uh we've we've got uh we've got a little bit of a security problem here so at this point in the game i kind of need to figure out how big that is now i believe he said that he's already seen three users uh mentioned i'm gonna well first let me see how uh let me just isolate this traffic you can see here by the way i have a lot of options for pivoting once i've identified a port and a host i can break this down by dozens of different ways to get very slightly different views of the traffic and investigate according to what is specifically interesting now i'm in this for security so i'm not really going to be looking into dscp or qos or bgp but host pairs by ports this is a really useful one for me now again we're not looking at a lot of traffic if you look at that you know we've got a max of about 400 bits per second this is the sort of thing that flies under the radar very easily which is what our hacker friends are relying on now let's zoom down here and see okay so now having pivoted to the port that lets me break it down more easily by both hosts involved remember my phone bill analogy you know it's telling me who is connecting to whom and in this case the whom is xmrpool.eu off in france so i'm going to pull up the host report for that so you can see that the process of this is i am going in i'm digging down drilling down and pivoting in a number of different ways and so you can see um let's see yeah we're still talking 600 bits per second coming in we are not looking at a lot of traffic here these guys justifiably think that they're flying under our radar and what we see here are okay so that first one is jfk um those other ones okay so we've got four hosts involved here that's uh okay so i i can take notes here and see you know i'm as i'm going for each stage i'm getting more information so before i go on though like right now i'm just looking at a short period of time i want to find out how long this has been going on so i'm going to just zoom out i'm leaving all the filters as they are and let's so because this is full fidelity data this can take a little while to go through but can you imagine how long it would take to go through a week's worth of packets for a filter i don't really need the information that's in the packets all i really need to know is which of those needles in my haystack are marked xmrpool.edu so let's give this a little more oh here we go so it's been let's see it's been going for yeah it's been going for at least a week let me let me go a little earlier very easy to pan back and forth um so this is let's see okay so it looks like this started around september 7th that's very interesting uh not least because watching john's presentation it looked to me like the performance problems didn't happen for another couple of days after that okay so yeah so i think i'm gonna at this point i'm gonna pull in some additional information so these are all the hosts that have been implicated there's more of them than i thought so now i've got some options here i think now what i want to do is i'm going to dig into those security events so throughout this whole time all of the traffic that's been coming in has been matched against a number of different rules and machine learning techniques that are looking for in my case a number of different security related events so i'm just going to pivot and see what started now i clicked on i think i forgot which one i think it was the jfk host so that is going to be what i'm searching on here that didn't come up with anything but i'm going to i'm going to add to that filter because there's a couple things involved um i don't actually remember the ip addresses uh yeah okay so let's just go in sorry i can't walk into them at the same time so eight and yeah so i don't i don't need to filter on all of them i just want to get a sense of what this is obviously i can fill in more than one or i can click the filter for there so let's see what comes back when i do that and let's see so this is going back for oh okay so yeah this one is a little more interesting so you can see here that we've got some flags from a number of different uh security alerts we have exfiltration alerts and blacklist alerts we've got okay so some of these have been flagged as malware download locations some of these have been flagged as command and control so for the exfiltration you'll notice some of these are very short some of these are like three and a half hours one's eight hours the exfiltration alert is a little different than the way other folks do it um because we are able to look at the individual sessions as they're going on instead of looking at the traffic rates we're able to track that network session over a much longer period of time and we're able to tell when a total amount of data has been passed outside the network regardless of how long it takes so if somebody is again trying to fly under our radar by going low and slow then we can still pick it up now i have this set up for a fairly low threshold uh the idea being that i'm not going to start my investigation with every single exfiltration unless it goes over like a couple dozen gigabytes or a couple even a terabyte i can set it for this is primarily useful the way i'm doing it right now where i can see in the context of an investigation what else might be happening you can see off on the right it's showing me the destination and that's not one of the hosts that has been on my radar so far so i'm adding to my little list of all the things that i need to go into now i'm going to have a look at yeah the commander control so this is just telling me i've gone i've set up these blacklists and these are you know known c2 servers that have been identified by by one of a number of different inventors and it has identified this particular host now this host you know i i recognize that that is in um azure i believe now i have a bunch of different options here uh obviously so far i have been pivoting inside our product primarily but i don't have to you can see for example i can go down and i can pull packets for this host i move over to app response and see exactly what that communication with that blacklisted host looked like i can also go and i've got some external links set up here i can do a lookup you know very easily for that ip go to sans internet storm center or mcafee trusted source i thought i had virustotal in here but that's also a great one i have a lot of options now for going in and just from this screen finding out more information now i obviously had a lot of work ahead of me right now i think i'm going to end it here the reason i'm going to end it here is that i've learned a couple of important things first i've learned that this has been going on for more than a week i have gathered a list of ip addresses inside my network that i believe to be compromised and i have a variety of ip addresses outside my network that i need to investigate either looking up in in a third party service looking up to see if there are known malware families using them and just simply doing a search engine for that is good and of course i can look it up in other tools however this has been going on for over a week and the performance problems have not on top of that those list of ip addresses do not match the list of users that have been having problems which means that while i have dug into this and i have made use of my full fidelity data to determine what the extent of the breach is i'm pretty sure that this is while it is a problem and it is potentially a big problem it is not the problem i think we're going to have to dig a little deeper and i'm going to pass it on to my colleague brandon to try a different tack

Channel: Tech Field Day

Views: 132

Keywords: Tech Field Day, Gestalt IT

Id: c0HXOcXsLWU

Length: 17min 15sec (1035 seconds)

Published: Fri Sep 17 2021