How To Use SysMon to Really See What’s Happening on Endpoints Webcast 2017

good to everybody Randy Franklin Smith here talking about a really awesome tool from Microsoft from sysinternals and the very great and smart mark russinovich and his team who wrote sis Mon on is free you can kind of consider it almost part of Windows in fact it it installs in Windows and it shows up under windows and this in the event logs and and so on we're going to show you why system on is important what it does how it works and how you can leverage these events in your sim to catch the bad guys today's real training for free is made possible by logarithm and today I have with me a really smart guy from logarithm they always seem to be smart and that's Jake Reynolds Jake thanks for being with us and helping us put together today's real training for free absolutely think it's been my pleasure let's see here so there's so much you can do with sis monde but first of all why do we need it let's talk about that then we'll get into how it works how to configure it understanding the events by the way I've documented now all of the system on events at the windows security log encyclopedia so I'll show you that too we'll talk about filtering that's one of the real strengths of sis Mon and then putting these things to work especially we're going to zero in on one example and that's caching new exe s and DLLs the first time they run anywhere on your network and guess what that would have caught want to cry that's all it takes want to cry just it used some new exploits of Windows but once it got embedded having used those exploits it dropped exe s so if we're just watching Jake for new executables the first time they run then we're going to catch so many of these attacks and even better yet if we can do that for dll's and we're not talking about just the filename either are we about the actual hash that's right we can get cash information we can see what's injected a process and and parent-child relationships through this month so actually that's a great bit of finding new stuff yeah and folks please use the question window we want to hear everything you got to say especially those of you already using sis Mon tips that you've got and so on so keep that feedback coming okay so why do we need this month I mean we have the windows security log right problem is some events are just missing from the security log altogether for instance when we when a bad guy injects a thread starts up a thread in another process I mean that's a common way that for instance many cats works is it's going to inject a DLL into the else ash process and then spin up a thread running on that dll and so now it's you know we basically injected and made that a zombie process that does our bidding and so now we can access everything and do everything that else asks does including accessing password hashes so there's no events in the security log for that so there's one example of my first bullet point some events are missing altogether other events are missing important information as an example when you go to the Windows security log and look at event ID 46 88 I'm going to bring that up here so then ID 46 88 and we look okay great a new process ran we see that notepad ran nowadays we do get lineage showing us that Explorer ran notepad and who did it it was administrated great the only problem is how do we know that that was really notepad a bad guy might have replaced notepad with his own program or he may have modified and in added bad guy code patched notepad in other words so there's no hash that's the point a hash of the exe or the DLL is very effective at identifying a unique executable and now Jake we can just dismiss the name of the program and you know if I bad guys more sophisticated and he needs to drop in Exe on a system if he's smart he would vary the name in the location of that Exe every time but if we're logging a hash the hash is always going to be the same or if we're trying to detect new systems as soon as we see a hash we've never seen before that's you know that's another way to detect this stuff so go ahead Jake yeah well whitelisting is absolutely one of the best ways to catch something on your network unfortunately that can be kind of hard to to do properly and that's we're kind of bringing system on in and looking at that data and baselining it is key exactly so there's more though that we log so butBut that's an example of other events that are just missing important data so you don't get the hash in the security lock as just one example plus there's no way to filter what gets logged you know all we can do in the security log is play with the you know like like this stuff here the the auto policy is what I'm getting at so we can come down here and we can go to our advanced auto policy configuration but like for process creation it's on or off it's going to log every process or none at all and what if we want to log just unknown processes maybe we want to suppress the logging of processes that we are familiar with or or that have been signed okay there's no way to do that with the security log either it's an all-or-nothing proposition so here's what sis Mon does it logs sixteen different events basically in these areas there's a number of events that help us really answer well what programs are running on the end point and then what weird things or processes doing like do you have any processes that are poking in and trying to access other processes in a way that normally only a debugger would would really do like if I'm running you know Visual Studio and I'm using that to debug a program well then that makes sense but generally everyday users are not doing stuff like Visual Studio where I would start up a program and step through it you know one line of code at a time and really that's the only time you would legitimately see this next bullet point suspicious access to other processes also system on will log network connections there's also suspicious file system activity by the way the suspicious network connections it's a lot of the same data that windows filtering platform events and the security log log but they are the big strength of system on in this case is that it allows you to get very granular with what's logged and what's not because these events can be tremendously chatty you know I mean it's when you think about all the network connections that any Windows system accepts or creates on a regular basis it's kind of prohibitive to log that stuff but this mod allows us to get much more granular and specify which programs we want to log network connections for or which programs we want to exclude network connections and we can you know filter on any of the other fields of these events and that's really the cool thing here Jake is the SIS Mon configurations with the filtering that determines what gets logged and what does it we get to filter on every field in every event I love it that's to me that's how they should have done the security log in the first place absolutely the security log most of the windows event logs it can be a bear to manage and bring them all into one place being look filter them before bring them all into one places is great yeah Nathan we're going to talk about app Locker and how it compares to system on events here in a second okay we also contract suspicious file system activity so there's a number of things that bad guy programs frequently do which normal programs don't do very often let's put it that way so we'll talk about that as we go along so there's specific events for that system on also allows us to audit registry changes and again we have that capability with the security log it's just the filtering is a little bit easier with with system on you don't have to mess around with the audit policy of individual registry keys you just specify it in the config.xml file you used for sis Mon there's also named pipes events which I haven't gotten into yet and then finally sis Mon does log changes to itself which is important for integrity purposes let's see here so just checking I'm going to hold on to some of your questions until I get a little bit further along okay so this one it's a small and easy to install service in fact you can install it from a network location here is an example right here as you know so this command could be executed as part of group policy like as a startup script or as a scheduled task in group policy or if you've got system Center whatever it's very easy you would just specify the fully qualified path to your UNC path to sis Mon somewhere on the network and add a couple parameters accept the EULA - I is install as a service and then finally here is the path to the configuration file that you want it to load now it does it's going to copy all the files it needs locally so this is these UNC paths only need to be available at the time of install after that sis Mon runs locally and then what if we want to periodically have one second what if we want to have all of our computers periodically refresh the configuration what if I make a change to the system on file I can just run system on again - C configuration and then give it a UNC fully qualified path to that XML file and it'll read that XML file it's not going to look at it in the future it's going to put it all locally go ahead Jake I'm sorry I walked over you no you're good scheduled test fit is pretty key I wanted add that in a pinch I've actually done this during incident response of responding to a warm outbreak and push this out with the PS exec or power shell remoting where it's available because it's encapsulated and kind of handled all that file transfer for you it can install very quickly very easily and help you get an idea what's going on immediately right good very good points so a couple people are asking how do I get this data and the right that you know the events I'll show you where they go in a second but the short answer is if if you don't already have a syn that you want to collect it with with an agent then use Windows event forwarding and I've done a number of webinars recently on Windows event forwarding and tools that are available to help you get that implemented and what we show you today with logarithm logarithm loves to ingest forwarded events in fact that's all I'll be showing you that as well but that's what you'd want to use but we'll show you that we'll get into that more as we go along so so yeah so stay tuned on all that okay in fact here we go this is where sis Mon records its events it puts them where they belong in event viewer down here under this lock application and service logs Microsoft Windows system on operational so here's where all of your system on events are going to show up and from there then probably the smartest thing to do is to use Windows Event Collection to have these events automatically forwarded to one central system where then you can ingest them into logarithm or whatever other method you have for for managing logs and I'll show you some of these events here in a second but but here's one this is the one of them that we're going to really get into detail on and that's event ID one process create and it's just showing you this is actually an event showing the system on itself was executed and what I did is I it gives me the whole command line what I did is I changed my hash algorithm to sha-256 and there's the hash of the system on program right there so whenever you run a program it's going to hash that deal that exe and and record it to you and by the way a couple people are asking what's the system load on this who asked that yeah Bernard says what is the performance impact on the hosts okay so we need to be careful what we turn on but even if you go nuts in fact I've installed sis Mon with a configuration that law every network connection and every DLL and it creates a lot of events but it has not slowed my system down now I've got a really fast CPU but there is no reason that you can't run sis Mon and users never feel it never feel it there because you got to think about who wrote this this is mark russinovich this guy knows Windows better than the people who wrote Windows and he's implemented all of this using asynchronous coding and and for instance when you load a DLL or an exe it doesn't hash it every time it doesn't want and then saves that in memory right and it's done all this by writing Mindi drivers that actually plug in to the file system and elsewhere in the kernel so you know it is running as close to the metal and as lean lee as possible and also know there's no known I haven't seen anybody complain about any stability issues caused by these which is something you might wonder about when you hear about device drivers and kernel level so as far as how this scales out with a log be overloaded yadda yadda okay again that all comes down to your filtering and how refined you make your system on configuration so that's not a system on question that's a configuration question and I'm going to tell you there's show you there are some sample system on configurations out there even on github and they represent some some good collaborative consensus work one of them is by this guy who goes by Swift on security so I'll show you that as we go you don't have to start from scratch is the point okay okay so let's a lot of you are real interested in the log management and the management capability so let's just hold on to those questions okay we're going to let's come back to them but let's get a little more down into the weeds of of what's this Mon really does and that let's start with one of the first questions I think this is where everybody should start if they're using sis Mont and that is using it to track what programs are running on well this computer or how about all computers on my network so the security log gives you 46 88 but it doesn't log dll's only recent versions of Windows does this event give you the command line or the parent process and only the program name is logged there's no hash now AppLocker remember you can also turn on app Locker in audit mode only so app Locker gives you a hash it doesn't give you parent process information or command line information so sis Mon gives you all that more whoops here is event ID one look at everything we get first of all we got our UTC time when this event happened we get not just the process ID the problem with correlating events by process ID is that they get reused Windows later on will use six two two eight again process grid is something that sis Mon generates for that one process on time it's absolutely globally unique throughout the universe and throughout time and so any other event logged by system on that refers to that process will bear the same grid which is you know very useful for correlation purposes here is the name of the exe that was run obviously we ran Chrome here's the whole command line that went into Chrome here is the current directory where it ran from here's who ran it logon IDs so that's my that's an idea of my logon session windows again reuse those sometimes over time if your system stays booted up so again system on generates a glid for each logon session and uses it a throughout here's the hash of the exe that ran so this is the sha-256 hash of chrome the beauty here is if someone replaces chrome with a bad guy program different hash if somebody patches chrome different hash now you folks might be thinking oh my goodness every time a program you know is updated or every time we run a Microsoft hotfix is that a new hash yeah it definitely is but here's the thing we're not what we want to do is set up an analysis system that constantly learns so we would run we're going to be creating a list of known hashes so when we're first getting started we might run that for a few days and just collect all the hashes that appear and if there's bad guy programs in that and those hashes go in to the list then we switch out of strict learning mode and now what we want to do is alert the first time a new hash shows up and give you the name of the program the user ran it and the computer that had executed on and with that same event add that hash to the known a hashes list so that as as that program gets executed again again on different systems we don't alert you about it each time because obviously you can't we want to know about the first time and then analyze it if it's a turn if it turns out to be bad then we'll go back Jake and ask our sim all right well you know where else says this program executed but the key thing is we know about the first time it ran we can vet it if it doesn't vet then we go back forensically and say okay wow this is bad where else has it executed thoughts on that one no that's absolutely correct and that's kind of the key point with with this you know once once you've baseline to that those hashes you can you can then well when the anomalies trickle up respond to just that issue so and that's kind of where this month ideal because this the the process good and and the hashes are going to are going to stay the same system to system right yeah and unless that payload changes drastically you see it once and then know it everywhere yeah Rob says what hash algorithm would you recommend and why you know it's sha-256 is going to give you the greatest uniqueness work this is not we're not using the hash algorithm for for encryption purposes here or for authentication purposes so you know like the vulnerabilities in sha-1 and what I really don't matter md5 really doesn't matter because what are we using this for we're using it to digest a you know multi megabyte program down to a manageable string of numbers that to judge the uniqueness of the exe and and sha-1 still does you know it's always done a good job of that it still would but if you want the maximum uniqueness you'd go with sha-256 that's that's really what it's all about because what you're doing is you're you're taking a very large a string of data and you're generating a hash from it so there are countless theoretical patterns of data strings of data that could generate the same hash but it's not predictable thankfully so anyway I don't want to dive down too deep into one-way hashes and and all of that but that's what it comes down to it's all about uniqueness that's actually a really good question it's probably important notes if Mon is capable of doing multiple hash types even simultaneously sha-256 should be more than then enough if you've got extremely old hardware you're you're in one of those environments that still has Windows XP running then system on can be configured to do md5 which will have a lower CPU load but but not quite as unique and number but but you would want to make sure that you use that across your organization so that while you're analyzing these things you don't start you know double dipping um your your analysis time on on binaries because you've got two different hash tags running around though right mark says PCI DSS would advise against sha-1 okay but it's what you have to understand is all algorithms can be used for different purposes I mean I'm not going to argue about it because just you shot 256 but it's we're not using the hash algorithm for security purposes that PCI would be pertaining to and I think it is important to understand that that we're using this to judge the uniqueness of the file we're not using it to hash a password for instance okay here by the way folks you is just where I've updated the Windows security log encyclopedia and now you can select system on events and we documented all of them there along with you know commentary explaining the fields and giving you some examples we also give you examples of all of the exome the XML version of each event as well okay now besides process creation what else do you get you also get processed terminated most the time we're not super interested in that unless I would say maybe you're doing an investigation on a specific system I think you can buy some some some disk space and you know capacity by not turning on process terminated logging but I would definitely turn on number six drivers so this will tell us each driver loaded and the cool thing is we can add in filtering that says if the driver was signed by Microsoft if it's part of Windows don't log an event and so that's going to wipe out 95 probably 99 percent of your of the events and only tell you when some other driver loads and then you can whitelist those what about dll's we you know it's not just the exe s that run the DLL that are loaded are super important too and especially since bad guys use this as a way to inject bad guy code into other running processes for one thing okay or it could be that we've got Microsoft Word opening up an infected document with a macro that then starts using PowerShell dll's inside of word I shouldn't be happening if we can catch that it's going to be awesome so you can do that with event ID seven image loaded it says image instead of DLL it's just the way it is but this event tells us let's go here to event ID seven image loaded and what is this one telling us so we were running notepad and we loaded up DLL la 32 dot DLL and here's the hash of that DLL was assigned yes who was it signed by Microsoft Windows so this is this is one of the really cool things about system on Jake if we try to log every DLL loaded we're going to get into trouble because you know a lot of programs load a hundred or more dll's and so we are you know if you run if you have if you think you have a lot of process events you're going to have logarithmically exponentially I should say more DLL load events unless you do some filtering right and so the cool thing is we don't need to make a whitelist of all our known hashes of known DLLs and then of course that becomes a maintenance nightmare every time a DLL gets patched hatched because the hash changes now forget about a left we can say if it was signed and if the signature of you know the signing party includes Microsoft and the signature is valid then don't log it and it's checking the authentic code signature and now we're only going to get dll's that are not you know that are not signed by Microsoft er and if we've got Acrobat or Adobe and others and we want to filter those out we can ya sure whenever you are whenever you filter something out there's some amount of risk but here's the thing folks are you logging are you looking at these events at all do you have any visibility into the DLLs running on workstations if the answer is no then you're much further ahead by turning on auditing of unknown DLL and filtering up you know signed dll's then if you're not doing any at all so you can always find a weakness in a control but don't become a security cynic and then not implement any controls because oh well there's a way you could get around that so I do it you know time San Jake's absolutely you're not going to catch something if you're if you're not looking at the forensic evidence at all yeah and you're never going to be able to look at everything you're never going to be to do things perfect but you make your best effort so how do we configure this then here is what the XML file looks like it's super simple it because it begins with the tag system on unit you call it your schema version which is currently 3.33 oh call out what hash algorithm you want uh here we had a question here Ronnie says what is imp ash providing on top of sha won't chop 256 and would it detect side loading so the hash algorithm is not going to detect side loading the impasse iymp hash is stands for imports and what that does is it's something that mandiant uses and other folks Mandy didn't come up with it but they did a good job of writing it up it's something you might want to research but the impasse the import hash looks at the dll and looks at it it pulls out it extracts every function call that that dll imports from other dll's or the operating system itself or whatever they're called imports and then it hashes that so basically it builds some metadata about the dll that speaks to what does this dll do and then it hashes that and that's what the impasse is and it's a method of detecting programs that are binary different but essentially do the same thing I think that's the best I can do explaining it right now but hopefully that helps you Ronnie let's see here okay so then we call out event filtering and then everything within event filtering by the way notice I don't close my event filtering tag there I left that in for anybody who's watching who's good at XML but then what we do is we call out a tag for the name of the event the tag of the event so event ID one is process create and I'll show you there's a table that that course that tells you what is the tag name in the configuration file that corresponds to the event ID but then ID 1 corresponds to process create and what we're saying is hey listen we're going to put criteria down here inside this tag and if we match it then exclude the event don't log the event now we could also say on match include so that's taking the opposite approach where we're saying okay only log events where the criteria does match this and that would be like if you have extremely limited narrow focused desires for saying I only want to know when when powershell and RDP and cmd.exe are executed nothing else very unusual that you're going to use include you're almost always going to say exclude which basically is saying log all my process create events except for those that I that I know I'm comfortable with and and they're they're huge volume so it's worth the possible risk of not logging them let's see here and now let me show you some examples of that criteria well yeah yeah here's here's some examples so notice process create on match exclude that's just what I showed you now here are some examples and these were actually pulled for the recommended template system on configuration that Swift on security put together and I'll show that link with you shortly but you see what he's doing here is we can take any field out of the event so we're using command line image integrity level parent image parent command line do you realize what these correspond to if we go back to event ID one we're taking any one of these fields out of the event like image it's in here somewhere right there and we're filtering based upon that or parent image parent command line and then we've got a number of conditions we can use begin with or condition is or it ends with all of they're actually more than that even we can say is is not contains excludes begins with ends with and so on it's really it's really nice and so put all those together and basically what we're saying not even basically what we're saying is if the event matches one or more of these conditions then exclude the event don't log it and this isn't just for process create you can do this on all of the events right so if you want to start logging some of these other spot events then it's just a matter of bear with me here [Music] it's just a matter of picking the event then getting the tag name of the event and it generally tells you that right here in the rule yeah so there's there's the tag name that you would use in the system on configuration file and then you just pick which fields that you want to filter on in which condition is it is is not begins with etc and you're good to go so we can log dll's being loaded and look at this look at what swift on security did here this is a venn ID seven that we're controlling with the tag driver load again on match exclude and we're saying look if the condition contains Microsoft Windows or Intel and of course the signature is valid then don't log the driver beautiful oh and this is by the way this is driver okay but we could do the same thing for DLLs being loaded and I can also answer your question from Doron how will we know PowerShell was loaded by word if we are filtering DLL signed by Microsoft well that is one of the cool things that we can do we can use a combination of exclude and include criteria to make sure that in general we get rid of stuff that's signed except for these you know except for these particular names so that that's one way to approach that door on so you will have to play around with include and exclude criteria all right let's see and Kevin good question about what's the difference between is an image why use one over the other I think you're referring to like right here so so image is the field name condition is is that reminds me of somebody in in the past a certain politician but we're saying is the image field exactly this that's that's all we're saying there we could say image is not for instance and that would make sure that that particular image is logged okay but beyond process tracking I want you to be aware before I conclude the other things that we can track and system on and I want to do more a lot more webinars on this for instance suspicious file system activity one of the things that bad guy programs will do and you would have to search for some descriptions of different exploits and bad guy behavior but what one thing that we oftentimes need to do is to help fly underneath the radar we want to drop files but we don't want them to show up as having been recently created we the bad guy so we'll drop a file and immediately change its file creation time to the past so that it doesn't show up in file integrity scanners and stuff like that any anything that's doing scans based upon file creation time there's also ways that we can get around a file access auditing and also even maybe get around permissions with something called raw access reads another thing of course is bad guy programs drop files now we can't necessarily hope to log every file created on the system but with file create we can say I only want to log file creations in the following folders and there are common places where bad guy programs drop files also if you are building anything that is some signature specific so normally i shy away from that but let's say you know there's an outbreak of one a cry coming along and in progress and so hey some intelligence is is published about it that it drops a file name this and this folder hey go ahead create a rule that looks for that and you could do that with file create there's also something for detecting when files are dropped by a browser because the browser is always going to add a alternate file stream called mark of the web showing the file came from the web and so we can that's a convenient way to track when files from internet browsers especially at least ie and edge drop files on windows do you remember I talked about stuff that like Nene cats does with creating remote threads in another process or going into another process and grabbing memory out of it or loading dll's into another process these are it's possible to catch those things by watching these events now 8 and 10 don't necessarily always mean malware especially on developers workstation that is using a debugger in fact that's how I created these events let me show you a couple of them I did it with visual studio system pardon so create remote thread yeah see this is visual Studios remote debugger creating a remote thread in my supercharger service that runs in the background and perfect example but you might also see this in fact I think you've got I bet you this is one of the events you're planning to show with with that hacker tool famously known as me me cats Jake I don't think I have event ID in my path and I am showing off you know where we're picking that off oh there's no problem probably will still dig through that in the logs in a bit but it may be in there with with my my test from last week here's also a process access event oh and here we go this is me me cats and look it accessed the wind logon dot exe process to steal us some credentials there you go beautiful Amy cats win log on dot exe so that's the real thing right there folks but of course remember there's many many many many implementations of me me cats so don't just look for that one event that's what's called signature specific you can also audit registry changes and the events are very close to what you get with the security log but the big thing is we can filter on all of these a fields and so it becomes a little bit easier so if we know that there's programs making legitimate programs making changes to the registry we could filter those out for instance and know only when some program we don't trust changes run ass and so that becomes you know much much more powerful in controlling the noise than what you get with the windows security log finally sis Mon tells you if sis Mon has changed so if the system if the system on service is stopped you're going to get event ID for when it's restarted you're going to get event ID for if somebody changes the configuration of sis Mon it's going to log that and the important thing is it will log the hash of the configuration file so you're not going to change the configuration file that often it's easy for you to get a hash of a legitimate version of your configuration file what you want to do is look for any of these event ID 16s where there's either not a hash meaning they could change the configuration from the command line settings and did not specify an XML file or the hash of the config file is different than a blessed version of your config file that's the idea there so how do you get your configuration right it's iterative I start with one category of activity get the noise down to a reasonable level and use group policy or system center to install system on automatically periodically apply your current configuration from a shared folder I gave you the commands to do that earlier on now how do we leverage sis Mon process events in your sim like what I would want to do is start out looking at sis Mon starts image loads and driver load events and then create a white create a list where those of hashes go after we run it in a learning mode for a while and build up a list of known hashes then we start alerting when we see new hashes and immediately add that hash to the list so that we only get that alert one means we got to follow up on that alert and and and find out if it is a bad guy program then we've got to say okay well where are all the places it's already executed right but we can also use these events to analyze process lineage like we already gave the example of word starting PowerShell analyzing user behavior I would love this why why is end user Bob running remote desktop or why is end user Bob running PowerShell so that's user behavior analysis and that's just a little more detail on exactly how I'd set up that whitelist and what not if I have time I'll show it in logarithm but I think I really want to get over here and show folks how you've already built in some really awesome capability into logarithm for dealing with system on events and analyzing them so Jake I'm going to make you the presenter and folklore we're going to get all these questions answered all right let me get my screen set up make sure to pick ripe one all right so let me know this is in the case we should have kind of my sample dashboard running right now yeah blah good walk through that in a moment um so what I put together here is just kind of a quick example of kind of analyzing what's going on with the endpoints from a perspective of the process telemetry we're bringing in right we can we can test for normalizing all this data we can very quickly take a look at things like host with an excessive number of processes and kind of just compare them side by side along with the same kind of telemetry data for users in this particular case for these two on emerging system on data with some process telemetry from our process monitor on Allen Xbox so you actually get systemd in here alongside our windows telemetry that this month's bringing it so it's pretty powerful by looking at that common event and not narrowing down too much to our sources we can kind of marry these things into into a useful useful dashboard I put together a start-stop as a running Delta in my mind that that's a potential useful thing if they've got a worm coming through one of crises perfect examples it's running through your SMB shares we should see a lot of process he's turning up and kind of run away from the number of stops that we see right this will always have a few more starts and stops based on our user environment and you know how many applications they run but but when when things go south you know you'll pick up this anomaly pretty quickly you know then I kind of put together some process to limit trees so we have kind of the most common processes running within an environment and then this allows you to very quickly take a look at at changes in there right if you're a Windows heavy environment service list is always going to be at the top but if something else are strictly enough again that's an almost activity something that we can put it down and look into and then I put together a similar one with our starting processes right they Amir pretty close but but that gives you an idea of long-running things service is a prime example being an OS level process right we shouldn't see that one coming up very often but it will be running almost all the time and then of course we can look for anomalies in most and least observe hashes of those of those executables that dismounts reporting for us so if we go and drill down into a couple of these you know we can we can quickly take a look at you know where our data is coming from and then this is a this is of those cases where we can we can come in take a look at you know our pit numbers tip it into a specific host and and investigate something this is a similar view only within a different time window and then I've got one here that that brings in host data so that you can kind of marry user activity with what they're doing potentially on multiple hosts and so that give it a different classification and we can kind of order and drill down a different way and most of these are you know we can we can kind of add this kind of widget information to our analyze area just like we do with our dashboard right so we can we can quickly take this and and add that into a drill down search to add some context and then I'll show you since uh since you mentioned Mindy Gaston I actually went through and ran this process last week and and took a different tact so in this case we actually used a social engineer toolkit to put together a boost website and and then use ie to launch the HTML application wrapper to then go ahead and open up our shell process right here so typically the first thing that you'll see is you'll either have an alert on something like MSHDA or powershell looking for an odd parent process right powershell maybe normal for powder shell to launch from from explorer but if if powershell or CMD s coming up from IE similar to here right that's something you'll want to look into we have save an alarm for that in here and so then you can come in into a pivot search and look at the events a couple minutes before and after and what we went into in here we use the powershell or the HTML application to launch CMD here and then pull up powershell with an encoded string value right so this is where we've got many cats loading through powershell here is an included command and so this is something that you could easily alert on leave for CMD or or powershell with the next event right here with this - ec flag and then that it's very quickly something that you might want to go and look into because typically if you're running powershell you're either going to give some direct commands which should be relatively legible or you're launching you know TS one file may be assigned but but these are going to be launched through regular parent type processes and and not typically with an encoding so i think that's about everything that i've got prepared here right you know we very quickly come up with a timeline of this and and graphically look at the same kind of data you know what was going on on the i did a rather broad search here so showing ten minutes before and after this attack so of course we get google updater and they're talking but but that gives you a good idea of where he's waiting very quickly like tie in to look at very granular data about what the machines doing great should I show them like some of the white listing capabilities to kind of illustrate what I was talking about with notifying you the first piece in exe run you think absolutely I think if you've got that loaded up in your lab at the moment I think that'd be a good demo okay I'm going to do that then let's see you should give see my screen again is that right I do okay so here's what I set up folks there's a lot we can do with these events but the first thing that I want to do is I just I just want to know when a new exe shows up for the first on my network if you can if you can just tell me that Jake you know that hey this new yuxi showed up here's its hash we've never seen these bits executed anywhere on the network before and here it executed on Randy's laptop at you know 5:00 a.m. here's the name of the exe here's the the hash of it and we may or may not have seen that processed name that yet see name before but we sure haven't seen these bits execute before and by the way here's not just that Randy ran it and it ran on Randy's laptop but the program that started it was this program if I could just get that you know BAM I would know as soon as one a cry happened on my network the very first computer now the way I'm setting this up folks it's we're not going to tell you if one a cry runs again on some on that computer or some other computers so you get one notification and then you want to take that information that hash or that program name that exe name and search your logs to see has this thing run more and the other cool thing is we can take that hash and we can upload it we can drop it into virustotal and see is this a you know a program that's been seen before and that can help you quickly that the legitimacy or maliciousness of exe s as well right so that's the idea Jacob I just this is my one chance this first time I see this program it's up to me to do something about that we're not going to you know the the safest what am I trying to say you know it's striking the balance between overloading the security analyst and and not knowing anything that's going on in your network so I don't claim this is the AB you know the end-all silver bullet for detecting our one of the things that is going to happen too by the way is whenever windows or office or Acrobat gets updated you know the bits of some of those exe s are going to change and the first time that exe runs on the first computer that gets hashed we're going to get an event saying hey I've never seen this hash before and you look at the name of the program it's Microsoft Word you would need what would you need to do you would need to say okay is this Patch Tuesday did Microsoft Word version 2016 get updated yeah it sure did okay that's good enough or maybe you want to verify that hash it all depends on how deep that you want to go but Jake anything to add to that hopefully that's making sense to people um no I think that's a really good use of whitelist I think in um you know one other point to make is in some of like a lower secure letter security environment you can also do trend trend baselining where you pick a period of time that is kind of a sliding window and look at process activity either across specific hosts or group of hosts and that that can help you find not just like the new you know the new crazy malware that's working itself around but but potentially end users who have maybe change their mind and and not so happy with the with the company anymore and maybe changing their behaviors well and then that will couple well into some of the stuff we're going to look at with this month later some of the network activity behavior and some of those analytics are built right into the into our platform great and by the way you are an organizer Jake you should be to see the questions you can make that window bigger if you want to but look towards the bottom of the list there's about five or six questions about longer than perhaps you could answer those by text while I'm showing this so first of all somebody asked hey how do I get system on events into logarithm well I'll show them that first what I had to do was yeah right here I created a new log source and you need to specify as your log message source type this right here ms windows event logging - sis Mon and then you've got to go to the flat file settings and specify this right here now I'm using remote log collection on this particular lab but what I would recommend for most of you folks is to use Windows event forwarding and get all your workstations to send your events to one Windows Event collector and then install system monitor on that event collector and if you did that you would change this to local host so now we only have to set up system monitor one time one log source and we get all of our events we get all of our system on events from that one system monitor locally from the SIS Mon log file okay so BAM that's all it takes I'm now getting my event sent now here's one little caveat caveat logarithm only wants to see the sha-256 hash and so what's going to happen if you don't have sis Mon configured correctly your events are going to show up here I need Chrome I need Chrome okay your events going to show up right dick if you are logging a different hash then shot 256 or if you're only if you're logging more than one hash logarithm is going to it's not going to recognize the event so let me show you and it's so what it's going to do is just log it as a common event general logging information and it won't be classified right and that the data won't be there and I'll show you why it was actually chris martin to figure this out for me so I can't take I can take credit for discovering the issue but Chris Martin's the one that solved the problem where is the oh I want to show folks the raw log data here it is the log message I was going to say if you on the right and the details and actions the middle tab says log message that's probably the easiest way to show up there we go okay awesome Jake so you see right here folks where are the hashes so it's showing me that I ran a PowerPoint and yeah you see folks I had an md5 hash and a shot 256 hash I think that's the default system on configuration so I had to change that to just shot 256 and I did that over here with this one so you see now these events are showing up correct as classification sort of shutdown common event process started and if we look at the log message that's because we're just logging sha-256 and if you've got the latest knowledge base it drops the hash right into the column you would expect it to be in which is the hash column it also for backwards compatibility puts it in the session column as well and that's what I've based my rules on that I'm going to show you okay so that's the answer to one of the questions how do you get sis Mon data into logarithm but then here's how I would build out that rule I would create an AI engine rule called new program detected and what I'm doing is telling it hey listen find events find this particular common event process or service started and we're session is nothing or you know this would make more sense if I did it this way folks hash filter out null is nothing so what we're getting is all of our process start events where hash is not blank so we you know for the purposes of this rule we only want to process start events that do have a hash now the cool thing I should bring out here Jake is that this rule is suddenly going to work for sis Mon events but what else AppLocker events if you're capturing AppLocker events carbon black events because you parse olivey and classify and normalize all these events to this one common event scheme right and so the rule that I'm building is going to work regardless whether I'm using sis Mon AppLocker carbon black or a combination of those absolutely and the other nice thing especially in this case with something like a common event if you have as I showed if you had a mixed environment where we've got some Linux box belong sets and Windows boxes our dashboards can utilize both and so can our analytics but also if you're in the process of migrating from one tools from another they you're currently using sis Mon and and you're looking for different data from the core windows general logs or changing those filters around synthase working with a normalized data everything's just going to continue to work you won't have to go back and retrofit anything and you've got large you know there's large organizations out there that might be using carbon black in one division but they've got lots of other divisions that okay we're going to have to use system on there so that's the beauty of bring it all we bring it all in analyze it the same way yeah that's absolutely a great point that I that I neglected to mention you know you're going to have critical server somewhere that's going that are going to run one suite of tools and then you know semi critical devices other elsewhere and then your end user environment something completely different so here we've got here's the fields that I'm focusing in on that that I want to capture because what I'm really doing here is I want to create a second event from this a meta event if you will and here on group-by I'm specifying the fields that I want I want the I want to know the program and its parent program and the way these happen to get parsed in is object and object name that's going to tell me the program's the program that was executed and the parent program and then to find out who did it I want user origin and domain impacted and then I want my in the past it was session now I would say my hash okay now once I've done that then that information some of it anyway goes into my into my whitelist and in my whitelist the only thing I'm tracking is my can't change it now is my hash I'm using session because that fits the version of the knowledge base is using in the past that's where the hash was go and so the white list then is just going to learn all of the unique hashes that I've ever seen before and it's going to learn that for let's see whatever period of time that I specify so you know ideally I would tell folks run this for a week and that way you get a an overall workweek of typical activity from your workstations and now all of those hashes are going to get grandfathered in to your whitelist without generating events now on the other hand you could make this an absurdly short learning period so that you would what would happen when you as soon as it stops learning after say one minute like I could do that right Jake I could say just run this for one minute into the future uh you know I have to change my actual dates but it's not really going to put much into the whitelist at all and now I get a chance to review everything going into the whitelist it's probably going to be voluminous but it would allow me to catch things that you would give me at least a chance to prevent things from being grandfathered or to detect bad guys stuff getting grandfathered in that's already present on your system that's that's you know the the price of a learning interval right whenever we're having to do our baseline I should say baseline that's the price of a baseline is garbage in garbage out if there's already bad guy stuff it's going to go into your baseline okay so here that's my rule then and what's going to happen this builds my whitelist and in addition we are going to create a new event whenever a new hash is detected and I get to name it so I named this new program detected and so this is what I would call a a synthetic event Jake or a meta of it it's an event about other events you follow me maybe you have a better way of describing it um yeah we usually call that those compound events a really good one is you know it since we narrowed down all of the messaging into behaviors you would then create a compound event to bring all those behaviors together so if you see something like communication with the sea to accept the file usage brand-new processes like you're about like you're working on here and then clearing of like the VSS cache that's pretty good indication ransomware starting to fire up right and then what I'm doing also is the normal whitelist rule of logarithm does not continue to learn and so if if after the learning period the baselining period ends if you want to add new items to the list and we want to do that in this case that we only hear about the first time a new program shows up then we add that as a smart response action and so here we are I'm using a plug-in that that you guys provided that adds I need to change that to a session that adds that session to my whitelist and it really should be like this I'm using something we've had before so this goes to my baseline file hashes whatever and we're saying the alarm field would be session that's where my hash is so now that gets added to my whitelist so that the next time we see this event or see this particular program we don't get bugged about it again and so then once we've done all this it's just a matter of going over to our dashboard and adding a visualization or these new events that I've collected such as common event is a ie new program detected and I don't have any results right now but I think you see then how this works we've got some really nice capability let's see here going back to questions and I'm going to be sure you're looking at the question window because I've answered a bunch of questions and so has a has Jake Don is asking about any hooks into virustotal or something else I know you guys have some threat intelligence feeds that you can leverage do you have the capable include being able to look up hashes of executables or absolutely so we have both within our threat intelligence our threat lookup API and through smart response you can send hashes to a number of threat Intel providers virustotal is definitely one of them I believe that that Smart Response may be available on the community site but I'll I'll confirm that but but you know that's that's one of those tools that we can leverage from both the mega grid and and the alarms now is is a couple of different threat servers hook ups and specifically with hashing great Steve asks our virtual log sources required if you collect from your whet collector the answer's no they've done a really good job they automatically pick out the computer name you know the host impacted or or whatever and and they get that out of the log so you don't have to do that in fact I just did a webinar on integrating logarithm specifically with Windows Event collection and the recording isn't out yet but Steve if you check my website you'll see that that webinar and anybody else that's interested you'll see that webinar pop up in the library soon if you missed that webinar but it's specifically on logarithm and Windows Event Collection let's see here I don't look forward at logarithms site it's only going to be shown up on my site at least for the time being let's see here Kevin says can we displace this pond data that logarithm has ingested can we show that in power bi so they're asking I guess if you have integration with power bi yeah I Kevin's got a couple in there related to power bi one of them are responding to eat we have the ability to to forward data out of the platform and so I am not familiar with power bi and it doesn't it's not something that I've seen on our supported devices list but but that doesn't mean an integration is impossible yeah so power bi is a like an executive dashboard thing and it can collect data from sequel server so I mean if you dropped you know if you sent data into a sequel server they could pick it up from there as just one example yeah there's a couple different ways we can work with that - if it can work with CSV files it can automatically going to have our scheduled reports pump out a CSV instead of like a PDF and and so we've had some some luck with external dashboards using that as a source several folks are wondering if they can get a copy of your your dashboards and the stuff that you put together if somebody's asking for a PB IX file not sure what that is but the Kevin and other folks so maybe we can include some if anything like that is available we could include information about that in the follow-up email that goes out Jake yeah that's good over with with Graham to get that together for you so gary asks you said have all events from hosts forwarded to an event forwarding host then set up system on on no no no no you got to set up system on on all of your workstations we're saying setup windows event collection on one server and have all those workstations forward those events in so there's two things being used here there's sis Mon on each workstation each endpoint that's what generates the events then we take a Windows server and we make it a Windows Event collector that's a built-in function of Windows and it it gets all of your workstations to automatically forward their system on events to that computer then you install a system monitor from logarithm on that one computer to get those events unless you unless you already have sis Mun on all the mean sorry system monitor Wow never even realized how similar the names are so since sis Mon is the Microsoft tool system monitor is the logarithm agent so if you already have system monitor on all the systems that you're interested in then just pick up the local system on lock otherwise have all those computers forward their events to a collector and then pick them up from there let's see here mahalia asks can system on get the application version I don't think it logs that no Kelly consistent be used as a fem solution not just create or is file create the only capability yes sis Mons only logging file creation if you want to log when files are changed or modified or deleted Kelly and you either need to use Windows file system auditing or the system the logarithm system monitor age and also has been built into it right Jake yeah that's great yeah okay stay with me Timothy asks could using sis Mon process event learning and then alerting on unknown process this as be as effective as application whitelisting well okay so application whitelisting Timothy I would say is preventive is a preventive control using sis Mon is a detective control so that's the difference there that being said I find detective controls are oftentimes much more practical to install and actually get running and certainly easier to maintain because the beauty here is if we update a program or we need a user suddenly needs a new program application whitelisting is potentially going to get in the way of that and and impact their productivity users don't feel don't know and are not impacted by detective controls but of course that means you've got to be responsive to that detective control and it means that the program does get to execute until you block it with with the approach we're taking today whitelisting application whitelisting you know stays upstream of all that but o as Timothy is saying in addition you could use logarithm smart responses to stop those unknown processes so that's a point too right Jake absolutely Greg says all this looks great I don't think I'd ever have the time to do all the configuring this would require to get even the base benefits of it well Gregg stay tuned well I'm working very hard on making this easy for you so stay tuned we got some good things coming down the line Bernard I think we answered your question right where's the data written if sis mon is installed you saw that it's going to the local log local system on log brian says for process termination does it log crashes I would think so but I haven't tested it so I can't say for sure Brian good question what is the network bandwidth impact to having thousands of endpoints forward windows event logs to a collector um you don't feel it on the the network it's more a matter of sizing your Windows Event collector and optimizing certain settings in the TCP stack on the Windows Event collector and assigning enough memory to the Windows Event collection buffers on the collector I have not found anybody ever talking about network bandwidth or the impact on the local local source computers with Windows Event Collection the only places we ever run into concerns or issues is on the collector itself Rob says would Detective be to whitelisting application whitelisting as a would that be in August to reactive versus proactive you know reactive is like a criticism is a negative whereas detective versus preventive I think is a more appropriate way to describe it because detective controls are not are not negative they're not weak they're not um you know the the controls are really really great and a lot of times you just can't do the preventive so that's how I would look at it rob but I do understand what you're talking about and and strictly speaking I'd say technically you're correct Jay asks if you install sis mod on all the workstation endpoints can you just sis Mon can you just collect system on and not the other windows events yeah absolutely J in fact windows event forwarding allows you to do filtering on top of whatever filtering system on is so let's say you just want a few events from the windows security blog you could have Windows Event Collection just send you those events let's see here Edward says how will we manage cost of sending system on output to logarithm before we tune its prolific output so there there yeah there's filtering you can do though you know so I won't go ahead just in the process of answering that so logarithm is not going to drop any of your logs or throttle anything if if you bump up against your your license you know capacity one of the things that you can do is a phase approach right at it add a couple of devices to tune the messages that you're getting and then and then add a few more and then you know that that highly filtered highly capable filtering system that that that this Mon hands is kind of going to be key to that process though [Music] Haley asks what mechanism does cis' Mon have to keep the integrity of its logs of its logs in order in other words have a system on prevent its logs from being modified or removed so once the events go to the log now it's up to Windows to protect it and the the point you know thankfully Windows does not have API for going in and deleting specific logs you can clear logs but that will get logged as well the really important thing is to get logs off the system where they're generated and into your log management solution or into your sim like logarithm and that means either having you know logarithms system monitor pulling those events or pushing them if you install it directly or using Windows Event Collection to just get those events off the system you know usually within a second of when they're created temp says if you have snare on endpoints can that be configured to send system onto a Windows Event collector we'll see you don't even need snare with Windows Event coach and that's the beauty of it right Jake is it just windows just sends the events and there's there's no need for something like snare that's the whole point of it right the built-in Windows event system we can pull directly from yeah yeah and logarithm not not all sims do a great job especially historically of consuming Windows events I mean pretty much we're able to get it to work with any sim nowadays but logarithm in particular has done it right and for instance you know it automatically recognizes which computer the forwarded event comes from rather than you having to do something special and map that so you guys have done a great job on that Jake you think that though on the Rd guy yeah okay I think that's it for today folks we're going to really try to put together a good follow-up email that refers to some of the things that we're talking about in fact one of them is this I forgot to include it in my slides actually but it is a example system on config file and I'm just going to bring that up for you real quick but I'll include the link in the follow-up but here it is it's right here at it's on github and that way you can keep up with latest updates but this is they try to put this together as a giving you a starting point for sis Mon that deals with the noise and is manageable so you know you'll want to take it and go from there but it's a great place to start instead of just starting from scratch and here is a link to that I'm going to put that in the chat window but I'm also going to go see let's make sure I add it to the slides and let's also add a link to the follow-up email okay I think that's it for today this has been great great questions from everybody and nice job on showing us how you put this together oh actually I've got a few more questions answer folks if you still want your question answered you're still here I'm going to answer that I tell you what actually I think I might just do this in a follow-up email because a couple of these I need to I need to research so stay tuned watch for the follow-up and we'll go from there thanks everybody have a great day and bye bye thank you

Channel: LogRhythm

Views: 23,232

Keywords: Windows sysinternals, sysmon, sysmon v6.01, Ultimate Windows Security, UWS, Randy Franklin Smith, sysmon tutorial, sysmon sysinternals, sysmon logging

Id: M3ptscFkD1w

Length: 90min 26sec (5426 seconds)

Published: Wed May 24 2017