Understanding Sysmon & Threat Hunting with A Cybersecurity Specialist & Incident Detection Engineer

Tom here from Lauren systems and I'm going to be joined by Amanda Berlin lead instant detection engineer a Lumera me and her are going to discuss how she utilizes cison for threat hunting testing detections by looking at real world examples and detecting malicious behavior in the wild she's a longtime cyber security professional and she's going to share with us how she looks at the world essentially Through The Eyes of a threat detection engineer so we're going to talk about anomaly detection and we'll get into some of those more fine details and the tools that we're going to be talking about you'll find links to down below and for those of you wondering if you can do this yourself absolutely I have a video showing how you can use the cismon modular and even export all the logs out of your Windows system or view them within Windows itself you'll find that video linked down below if you want to try these yourself and these are the same rules and threat detection that she uses so they're open source they're available this is one of the things that Amanda participates a lot in the community of sharing all this knowledge and threat intelligence knowledge cuz that's how we get better and that's why I'm doing this video so you can kind of look through the eyes of someone actually doing this well daily looking at millions of Vlogs and understanding threat detection so let's get [Music] started so how you doing today Amanda I'm doing great it's a you know nice blustery cold weather in Northern Ohio same here in Detroit we're pretty close together so uh might get it here and then pretty soon you should probably get a little snow as well oh yeah which sounds like a great day to talk about how to pull out logs from Windows using Symon you did this talk and I was like wow this was solid and of course I already I kind of know you from talks you've done before and then you landed over there at blumera some time ago which is a company we use definitely a fun fun tool there and pulling Windows logs is your specialty yeah it's one of my favorite things windows by default just doesn't give you good tooling for this uh sysmon is the glue to bring this together so we can actually get some useful information out of Windows and uh that's what Amanda is going to be presenting today so I'm ready when you are to get started awesome all right so let's just dive in uh this is one of my favorite ways to start out a presentation and I'm sure a lot of people watching uh have either stories to share of this or something that you've faced firsthand um and the stories around that so uh let's call this company Pat's grocery store uh my team is involved after a PS exec command runs over their VPN this uh triggers an alert to Fire and they have servers start to be ransomed which is definitely way later in the process than you would want to know about an attack right and pretty much too late in this specific case so there's no cismon installed and you'll find out why that's a big deal throughout this presentation and for some reason their domain controllers stop sending Windows alt together they aren't sending any of their endpoint agent logs but that wouldn't would have mattered because the domain controllers didn't have the endpoint agents installed anyways the account that was used to send the PS exit command ended up being a shared admin user with their pulse secure VPN admin account so same user same password that account had zero MFA setup and full domain admin access across the environment I think the PE people don't realize that the first thing you notice is ransomware but that's by far not the first thing that happened yeah that is that is the end like that's the end of that's the end of you having a good day right right yeah if if if something getting ransomed is the first indication that you have it's not good you definitely have had them on there for a while before that so other than the lucky fact that one of their endpoints ended up receiving that PS exact command was sending us Windows logs we would have been completely blind to all of that activity happening in that environment and it's always so nice if anybody's had to deal with this before threat actors a lot of times offer to give discounts for Speedy turnaround I love that which is they actually have from what I've heard amazing customer service yeah they build a reputation on it it's kind of they do blows your mind because they're doing terrible things right um I do believe Pat's grocery store had had to end up paying the ransom because their backups were also not working for a significant amount of time and that could probably be a total talk on its own yep and they needed that data on the servers otherwise they were just going to have to shut down all of their stores right and build from scratch exactly so they needed that that server data that was ransomed and sadly it's just one of those cases that you know you can't a lot of times we tell people you know you should do this thing this would be a great idea security wise and sometimes that just doesn't it doesn't work until something like this happens and a lot of of times this is also where you end up getting security budget sadly so as Defenders above and beyond all of the other roles that we play you know there's strategic thinking process creation research writing all of that kind of stuff one of the main goals we have no matter what vertical we're in is defending against threats that's you know a lot of times it's in our title so there's entire conferences companies Frameworks podcasts whatever you name it built around the three words and so prior to 2014 or so inpoint AV products were just like detections of md5 hashes so just plain virus virus signatures for the most part were doing all of the work as the amount of all of the things on the internet just started to Boom Out of Control right and that paled in comparison to what we see today so now we rely on what AI machine learning artificial intelligent boxes or maybe just waiting for that time that you see a splash page about the server being ransomed and as Enterprise networks grow and hopefully mature uh the majority of what we see is largely misconfigured or undercon networks and endpoint security products that just cost upwards of hundreds of thousands of dollars a year right like if you look at the budgets that security companies or security teams have to spend on these things it it just gets more and more every year and there's this repeated behavior of just trying to throw money at a problem and hope a new Blinky box is going to fix something right I was looking for some magic solution right right like I don't know our Auditors say that if we get this W everything's going to be fine right so you you have these Blinky boxes that are compliance check boxes that without the time and care and effort aren't going to do a whole lot anyways yeah yeah I see a lot of the Blinky checkbox compliance stuff and that's not where the real security comes from we have to know what's going on on these boxes right right invisibility is one of the uh anytime I get interviewed or or WR or or whatever um one of the uh you know common questions is you know what do you suggest people do or learn about or whatever uh to prevent you know next year's cyber attacks visibility visibility Asset Management top two hands down all the time know what you have and know what it's doing yeah exactly exactly so that being said how can we make sure you're getting the biggest bang for your buck especially when that buck is free so Sison is free here to save the day uh this is the Microsoft definition up there just in case anyone wants a ref wants a refresher um this was also released in 2014 as a part of the CIS internals Suite of products and ever since then I'm sure security admins server admins practitioners have been perplexed as to why it's not just installed on every endpoint by default yeah Microsoft acquired CIS internals back then they did yeah yeah and it's just one of those this is something you need to load on your Windows systems it was great those tools were so popular for years it was the missing component of Microsoft for a long time yeah for all admins right like I remember all the time needing to use sysmon and um uh process Monitor and all of those things for uh uh troubleshooting right like it was just not there and you had to install it as an extra thing um so we'll cover some of the use cases around why why we want this and then we'll dive into some threat hunting stuff too yes so here's some fancy stats um so Enterprise orgs have a lot on their plate on the cyber security front there's fast adoption and move to Cloud which I'm actually just writing about now like the the move from exchange John Prem to 365 exchange and then the last two years you know just the largest amount of move from a corporate world being from home from work to home right so you have this huge boom in uh work from home now where prior to co a lot of us were in offices um and that attack surface has significantly expanded right now all over the place you don't have everybody in the office from 9 to 5 behind a firewall this has created some really interesting challenges because now they're scattered all my users are you can't even hire someone if you don't put hybrid in the job title somewhere EX at least part of their time is not going to be behind your firewall but behind maybe some consumer router uh with access to things so monitoring what's going on in those end points is as critical as it's ever been oh yeah and um you know as as a former CIS admin that always just like scared us to death right because I worked at a hospital we didn't have anybody that worked from home but we did have like Physicians that would take laptops home and stuff like that and they were always always dir like just full of spy wear and just all of this crap right so going from that to like now everybody works from home is a is a huge shift um so going based on our average implementation at Blum so we would we were seeing like a single digit percentage of orgs that come with us come come to us with cismon already installed so we had so few that now it's just included in our onboarding process right like you want to get blumera license like not our free one because that's just Cloud stuff but if you want to do endpoint stuff yay you get Sison yeah because it's so helpful for detections and for incident response and it being free you know why not yeah absolutely just load it oh yeah and I'm a huge proponent of using the tools that are at your disposal so whether they're open source free whatever you kind of I've always felt that the the responsibility to whatever organization that I work with to look up those kind of viable options prior to just asking for like capital expenditure right so why not try like if if it doesn't do what you want it to do then you're not really a whole lot of money no right just a little bit of time is setting it up right so if you can reduce like time to detection time to remediation with a price tag of zero why not so I'm not one to read slides so I'm not going to read this one I'll let everybody watching since you can pause it if you really want to um but uh you know I I've always wanted the tattoo uh live laugh log live laugh log I love that um I've not made that jump yet but uh if I were to get another one I think this might be a little bit too wordy but uh you know this could be a possible possibility too um so at any point when you have an incent or if you had one or whatever and you have sysmon installed configured and logging you'll be able to find your breach scope right I can guarantee you that time to detection time to response time all those buzzwords are going to be key security metrics that people end up using in coming years and I know a lot of people are already using it already and it's one of the best things that you can pay attention to for program maturity um and then I'm sure a lot of people have also heard like the assume you've been breached yeah just because you know in the beginning getting breached was a huge thing right when it happened to Target their stocks plummeted when it happened to Equifax stocks plet like they did terrible for a couple years and then you know it all recovered but now it's just another one of the news right it the stocks don't even dip anymore uh you know people are less likely to get fired anymore when there it's just so prevalent to get breached that people are like I just assume it and then if you have been what should you be doing to find evidence of that right well and that's as you know is me and you both work in the infoset community with instant response teams and they're always complaining they should be about we couldn't figure out what happened because they didn't have any logging tools we know we know the results we don't know the how we don't know the backstory we don't know what led up to it uh there's an absolute lack of information we just see the Boom the results the the disaster that we're dealing with today but that's why getting these schools installed at the beginning is so important for that forensics information because if you don't know how they got there you don't know exactly how to prevent it exactly so that being said we can go on to some Sison use cases um first we're going to cover some of the specific benefits you can get from cspan so if you're beginning a hunt more than likely you are inserting yourself into an event related to a Windows host that's in the process of happening or maybe like we do a lot of times it's a retroactive step to find thread activity that's been missed in either those sit situations Sison logs give you by default an amazing amount of logs compared to any plain windows can provide you even when fully configured and honestly way better than a handful of endpoint Solutions so this is true yeah so here you see a difference on the leth hand side in the difference of Windows I IDs and then on the right compared to sysmon so there's 10 total types of data that fail to appear if you don't have Sison and then there's others that are possible that are extremely difficult to configure to even get a shred of information out of it uh if anybody has ever tried to log DNS from a domain controller without using cismon uh you have to like turn on debug logging and half the time it doesn't even tell you what device requested you know made the DNS request or it's just horrible and cismon just does it for you it says hey this host reached out to this website it's fantastic so for threat hunting what can we do with this kind of information um one thing that we do uh every now and then is using and and and I think a lot of us take for granted that we do this it's kind of just part of how our brain works sometimes and it's using standard deviation to weed out Baseline activity so this is the normal like curve of standard deviation and in this example graph like 90% of the results live in one and a half standard deviation and the rest you can call outliers that can be really helpful in threat detection because looking at those outliers regularly gives you a good look into some of the less common activities that happen in the environment granted if you take your standard deviation and you already are uh already are having an incident and it's been around for a while yeah it's just going to fall into normal activity right so like you have to have this for a while and collect a lot of data to be able to see what is normal in your environment um here I'm I just took I know this is like a crazy graph but I take an example of some sysmon logs across all of our customers and so the YX on on the left is total destinations so this is just like IP addresses that things have reached out to starting at 50 going up to the upwards of 500 5 million range and the x-axis shows standard deviation score so this we're taking all process names on devices across all of our customer data set which is why those numbers are so large and looking for Destination IP addresses that processes are accessing right so some of them make complete sense um and then we you know calculate the standard deviations so we're going to zoom into this which you'll get that joke in a second ah that makes sense right yes processes Zoom it's reaching out to a ton of IP addresses teams Java Chrome like this makes sense these are applications that reach out to stuff all the time Edge go to meeting that's Cisco WebEx ring central everyone's phone system and then we see notepad which should note notepad ever reach out to iPad or on internet but good news is that was just our lab um because if we start to see things in that standard deviation uh or outside of the the majority of that standard deviation that are like this like that's something that you're going to want to want to look into and this becomes different with every customer right like this is uh interesting data set because it's across everybody yeah but uh it's nice to see that it actually captured our notepad going out to two million IP addresses um another main detection creation strategy we use is uh adversary emulation so there's a bunch of tools out there like Red Canary has stuff um there's Atomic red team there's a whole bunch of different tools out there that you can use um this one is also free this is called Vector it's V C.O and you can import different Frameworks into this so I'll give you some examples of a couple of things that we started to do and this is like a heat map of miter so you can import all the things in miter and you can test against it whether your the endpoint agent you use blocks or detects it or your sim or your IPS IDs like you there's so much that you can do to track all of the different levels of miter right because there's different stages You Know It uh we used to have the tech um the um cyber intrusion kill chain right and now we have miter which I think is a little bit more extensive but there's all of these different ttps that people use and it's good to test these now can you test all of these all the time probably not without having several staff dedicated to that all the time uh it's really hard to test all of those but just think about you know um like spot testing this stuff so the amount of things that can go wrong in any given day with any kind of technology is you know overwhelming but you can pick all right today I want to check to make sure I can detect on uh if somebody installed a new remote access tool right just things that you I I was you know back in the beginning when I started doing this it's something that I've kind of uh always always Ed as an example think of the things that keep you up at night like oh my gosh I woke up in a cold sweat because I just realize that these users don't have tofa enabled I don't know if anybody else does that but there's things out there right well I think it's important too one of the things that we don't use team viewer within our organization right so when we see a team viewer install it absolutely Flags it opens a ticket it creates there we we don't even have to test it because well clients test it for us yes and you're start scratching head wait a minute what how' they get they supposed to have admin privileged so there's the first problem how they get team viewer installed exactly but yeah going through your organization and validating not just saying okay we we monitor for installs of new files we monitor for installs of remote tools being able to actually test and validate that finding is really important it's a heavy lift to to ask an IT team but it's kind of an important one to make sure that those flags are still being raised yeah and and you can pick pick like 10 a quarter right like you could probably do 10 tests easy in a day um especially with some of these tools so like an example is uh this is importing this is uh if if inside Vector this is an example of some stuff that you can track right so this is the red team section of this one miter technique all right so this is an example of T1 1482 and it's people doing uh domain trust uh Discovery so you can run this and it's it shows right in there when you import it this is actually importing the atomic red team stuff that the phase is Discovery it tells you what it's about and that trickbot malware uses it but it also tells you like these are the commands to run so you can just go to a Windows endpoint and if you can run this without being detected that's probably bad yeah and it's the important part because if you have let's say someone in your accounting office suddenly that accounting computer is running this you're like that seems unusual that is not what they run in accounting yes yeah it's that back to that whole anomaly detection this is right how you have to look at things holistically to say all right here's the whole picture here's this anomaly happening QuickBooks yeah reaches out to too many IPS and etc etc but all of a sudden that same computer is now running this and they're looking at what what is in the trust list so this is the instantly starts an investigation flag and getting that data out yeah exactly and and that Discovery phase is way before the ransomware yep right this is that's those are all of the things that attacker like we just saw one not too long ago where the attacker was using who am I uh and like uh doing user commands via the command line just doing NS lookup via the command line there's a lot of like Discovery things that most attackers either do or just have scripted to give them more information um and Sison helps with that too yep uh and this is the blue team version uh of that same thing in vector and it this is like where we'll store yeah we detected it um and here's the logic that we used in this specific detection right um and you could track those you know however you want and this is just you know our our tagging system internally how we how we track all the all the miter things right all right so now that we covered kind of how you can organize that stuff we can dive into some specific use cases um this is use use case one there's a ton of different ways that process memory can be extracted from a Windows npoint uh you can run things like MIM cats locally you can gain a uh access to hashes a multitude of different discreet ways um I can think of like five off the top of my head and here you can see they use com Services uh which is a comp plus service dll that was introduced in Windows XP yay uh that you can use to extract local hashes there's a handful of detections and um a couple of these will will cover a little bit that are called finishing moves meaning the attacker has all your keys to the kingdom the security team's lives have gotten much harder than they ever wanted to be uh or you're you know calling mandiant or some other you know IR IR firm to come help you this isn't necessarily one of those Game ending moves but it does mean that someone has local access to a machine and the files in their possession that they can crack to get credentials to your devices right especially if you're sharing usernames and passwords across devices uh you know it could potentially be the entire environment so we consider this a priority one threat meaning that you have to immediately act on it or you should immediately act on it um I do have stuff blurred out in this presentation because they're actually from customer Customer Events uh when it's not blurred out it's me doing it in the lab understand yeah so here we see uh the exact commands and the time stamp on the device in question so this is com services this is the actual command that was run right com services is doing a mini dump which um uh it lists the process ID and then where it wants to store that memory dump and then the keyword full um you see here uh let's see here there's other related findings um and that kind of just goes along with you know there's a lot of stuff that happens along with this kind of stuff yeah I I'm going to mention too on that page uh the log was from December 7th of 201 21 and these are all local tools that are found on there we we I finally was happy to hear in 2023 Lin come part of the common vernacular with cyber security but like yeah I mean cool that we got a name for it I like L bin as a name but it's not that new folks but thread actors have been using the tools that are available to them for a long time it's just been slipping under a lot of Radars of how' they do that well these tools are all on there I mean they're they're using built-in facilities of Windows to attack us essentially this is why it doesn't flag in maware Scanners this is why monitoring what's going on from a behavioral standpoint is so critical to cyber security and and difficult too I remember when Powershell by default started coming on Windows os's and they're like oh well we have to uninstall this like we can't have Powershell on every device like how are we how are we ever going to manage security if everyone has access to Powershell and then it didn't matter if you un installed it or not like you could just yeah reinstall it super easy and there was all all of that stuff so yeah like uh Lin and what is it Lin and L boss whatever they changed it or the additional ones they have now uh living off the land right which means there's so much R in Windows that can be used all right so this is this is and I have a couple examples of that yeah yeah it's definitely I I left because I mean I think in 2019 uh we had last time we had a Detroit spe sides my friend cavier presented showing how all the live off the land techniques how I'm gonna it was almost F bid I'm G to pone your system with everything you have right right which is easy right there's there's entire attack Frameworks built around Powershell yep uh so I ADV I apologize in advance for all of the screenshots of Event Viewer uh there's a lot of them but there won't be too much of a quiz um so here's when we see the incredible differences in native Windows logging on the left compared to cison install so even more dat oh yeah even if you can't make out all the specific ones you can see see just like a sheer volume of information uh event one in sysmon is the same same kind of as 4688 in just plain windows and it just pales in comparison right and the reason that 4688 even shows up is because we have to configure it in group policy to Output those results right you have to do command line logging and process creation to get everything that you need where you just get it in event ID one here's an example of um uh the specific um configurations that you can do for Sison so uh I'll cover it a little bit later but one of the things that you can do is uh there's like the Swift on security Sison um configuration and then there's also Sison modular which is the one that we usually use and the syst modular actually gives you miter techniques at the top of the rule names that you have in your configuration and then we can con uh correlate that to a couple different things the second one down is like we're just talking about before the living off the land techniques and that is a screenshot from um uh the actual system mon config and then the next one down is from uh uh the atomic red team tests yeah yeah yeah so it kind of breaks them all down and it's kind of cool that you can see all of that all of the techniques kind of match each other throughout those tools so in summary um this is what kind of the detection looks like um you can you know put it in most things that you have for a Sim so we're looking for Windows event ID one and then we're just looking for command or the parent command line to be that com services with a mini dump granted maybe you had an admin doing that for some reason but you're probably going to want to know about it anyways all right yeah so nice and short and simple uh then we can move on to use case two so this is an example I talked about a little bit before the finishing move something you want to know definitely as soon as possible uh another thing that just comes with Windows is ntds util uh if anybody's ever managed 8 databases that's the utility that's built into Microsoft for doing that be the command line and it's been used for years right you can use ntds.dit and that's the uh database of all of your active directory user information and you can just dump all of that information also and uh as a threat actor you can begin exfiltrating that and cracking all the passwords and looking and seeing what the directory for us looks like um I can tell you uh I've not had to do this but um it was really interesting the first time I gave this talk uh the person if anybody's ever heard of the M there's a Microsoft guide on network eviction process um the first time I gave this talk I'm like ah by a show of hands who has you know had to follow this or had to start a forest over from scratch because uh I don't know if it is still but that at one time are the two options you have if somebody breaks into your active directory you have to start over from scratch and manually do everything again or you have to follow this network eviction process and the person that wrote it was actually in the audience like oh all right well that's great and uh he actually told me instead of Doing Network eviction uh anymore he's like I just tell people to move to the cloud like all right it's time to just throw away your un Prem yeah no more DCS that's it I'm like oh that's real interesting um it's not an easy place to unwind when when people bury things and hide them in there man yeah there's there's a lot I mean there's some tooling around it that exists I believe blood Hound's one of those tools you can do some auditing with but it's not easy even even with the tooling it's it can be a uh it's a project it's its own project to itself after the incident yeah I would not want to do that it's just I'm glad I I kind of miss admin stuff sometimes but not always there's day when you're happy you're not doing that exactly um so here in our matched evidence we see the full Command right so ACI ntds sets that ntds as the active instance ifm is means install from uh media for like non readon stuff and then the backups created and the two cues are just quitting the previous two commands and it gives you a nice download of active directory information to go crack yeah and with humans tendency to reuse all their passwords and the absolute amazing availability of rainbow tables they actually don't spend as much time cracking they're doing more of a matching scenario right right it's it's much faster than people think yeah sadly uh and then here are the differences if you if you pay attention to you know the windows logging versus Sison logging uh on the left hand side it's just a huge list of$ 47.99 which is just enumerating security groups in active directory which that's how actor directory works you do enumerate security groups all the time um a 4 688 was generated and we saw that in the first use case and then uh here on the right hand side you can see huge wall of tiny text uh with the amount of information you can gather when that command is run using sysmon so the first one is the that event id1 again um it's comparable to the Elsas dump we went over in use case one you can see ntds utils the original file name the command that was run from the parent image of Powershell it even provides extended information like um that a newly created process uh when the command was run and then the full command line and we also see um is it in this one yeah so it gives a terminal session ID so it shows that I was connected via terminal Services when I ran this as well as I Ed P shell ice instead of just plain Powershell and then here's the first place where we see event ID 10 which is a process being accessed it's one of those 10 events that isn't included with Windows no matter what you configure and then you see this um that the source image of Powershell is using ntds util and then event ID 13 is a registry value being set and that populates when that process actually execute successfully and you can essentially there without the cismon there's not the entire chain of the event right yep yeah you get all three chain together for that instead of just enumeration and a process um so again we see the detection screen with what you can look at as far as the detection goes ntds util can be used again legitimately across the environment but it's extremely rare um so you can still look for event id1 and then the process name name of ntds util and this is just one of our detection versions that has the um command or Parent Command that has those two quit commands in it also because a lot of times that's pretty much all the script has all right then we have use case three this is another Priority One threat another living on the land which is comspec modifying the registry so on the bottom there you can see comspec is an environment variable that that just opens the command line so if you type in Echo and then comspec all it does is open up cmd.exe um you can't really see what's going on here from the description of the command itself because it's base 64 encoded uh but using that conspect environment variable along with hidden encoded Powers shell commands is extremely sketchy I've not seen a false positive yet of this type of behavior and I yet because there's there's an amazing amount of really terrible software used in production environments that like to mimic threat actor activity yeah there's I work with some industrial engineering people like why are they basic supp this yes why they thought they were hiding as a secret and apparently they didn't know about cyers cheef right right and it it's weird things too like I've I've dug into Powershell uh encoding before and it's it's just been like oh we're updating the firmware on this scuzzy drive like what in the world why would you pay 64 that I don't I don't understand yeah 18 Bas 64 it's at least raise raise suspicion on it right right um and then kind of a sidebar on that when I was looking into some data for this uh for examples for this talk I was weeding through some of these poal commands and notice something really weird and normally when you see a bunch of words with upper and lower case you just assume it's like appucation yeah so I took that b 64 decoded it and found that it was Matt graber's reflection method right which is a a tech testing Tool uh that bypasses amsi so right away I was starting to worry I'm like oh my gosh somebody's using mat's like stuff in this network and they're attacking it whatever and you know it turns out there's actually a really popular tool that people use as an endpoint agent that for some reason I don't know if they still do this um but they were using these locally on all devices as I'm assuming is some kind of Let's test our own detections um but I'm not sure it's it's a legitimate like endpoint security solution um and it just made me freak out because I thought they were actually being attacked yeah which they were not so that's good uh so back to the use case is that com speec um and here we can see it's being used to run Powershell and install on a service um normally you see a random service name but this uh talk was originally created for RSA so I had to you know make the service name rsac con and then here we see the full command um and I've already let's see this is on um sh what V ID is this I missed it um let's go back oh I think this is V ID 10 okay here we go so this is yeah the sneaky way of calling command line and then you see no profile the windows hidden it's an encoded po shell Comm and um all flags that make you raise an eyebrow at it these are window hidden encoded okay there's something going on here no profile yes uh which are common ways to bypass Powershell execution policy which turns out isn't uh wasn't even created as a security measure uh execution policies were created as a let's hope these admins don't shoot themselves in the foot by running these powerwell commands um that's kind of like having to type pseudo when you do something on Linux so we're going to look into uh event id8 because we already looked into those other two that were there uh most true positive process injection attempts you can find in both of these uh Sison event IDs this is create remote thread detected and another one that you don't have the ability to see at all with plain Windows logs and it lets you know here that Powershell has injected code into dlll host and you can see it here in The Source process as Powershell and then the target is dll host and then the next is 10 and this reports when a process opens another process and that's usually followed by information queries or reading and writing that address space um of that Target process and during this attack in the lab there was a generous amount of both event IDs because the command was run uh uh and it attempted to inject into all of the processes uh so this is just lab data and it kind of shows uh that that that encoded content isn't exactly the same as our actual customer event that we had in the use case so if we dive into that we take that encoded command uh at the beginning of this use case which was B 64 and I decoded that and then I was really confused because it it if the top Parts where I Gra from Powershell on the on the left the bottom is where it got decoded to but then it looks even more decoded and I tried to like double decode it but it didn't work uh it turns out that it was a base 64 gzip file ah right and then after looking that on the left hand side uh I'm like I don't know what this code is so I started to look into that decoda gzip and it was Cobalt strike hm uh and this is one of the methods that Cobalt strike uses to avoid detection and another funny story um when you include this kind of thing in a presentation it's best to take a screenshot of it as opposed to just copying the code because it turns out that a lot of places uh it was very hard to email and share this presentation and no one could open it because the endpoint agents were freaking out thinking that I was trying to use Cobalt strike hidden in you know a keynote file that's that's great so you had to make sure it's an image yes yeah so definitely learning learning moment for me there is to always just take a screenshot yep um and then here's a summary really just look for comspec all right super easy no one ever do good is happening if is running no it's so weird it's just this environment variable that you'll rarely see um and is worth detecting on as a threat uh so now that you're you know everybody's a little bit more familiar with event IDs and sysmon we're going to tie it all together in a customer exchange compromise um I won't Replay that lab because you know that's going to be another million screenshots of Event Viewer yeah uh but this is uh an example of proxy log on which came out in 2021 proxy log on.com explains all of it if you really want to know but it's basically just a remote code um execution that as long as your exchange server was open on Port 443 uh people could attack you right so it it generated tens of thousands of breaches of exchange online uh well so many incidents yeah people's on Prem exchanges uh yeah uh so not too long ago after I mean I guess this would been a year ago now uh we had a customer come to us and ask for help around a notification they received there's two MSI files that are installed on The Exchange Server other commands were being run that didn't think should be running uh and there were actually 11 rapid fire findings and so you'll notice that's the next you know several slides as they all happen in Rapid succession first you see something along the same lines as that process in process injection we talked about in the other use case so we basically have pow shell being run injecting into dll host and acting alone with nothing else it could be something not super terrible because we've seen legitimate server uh legitimate software do this before but then you get something like this which is a discovery command again on its own not terrible but this is net group Dom domain admins which is a command to find all domain admins in the current domain uh helpful for admins right also helpful for attackers yeah if you only have a couple admins and you're all looking at each other going who's looking up the rest of us right uh you start to be like all right well that wasn't me uh oh that's that's not good um and then here you have a little bit more suspect activity granted this is something I've done before right you need to transfer files and you just decide okay I need to open up SMB and transfer from one to another not great especially if you're doing it to a domain controller from your Exchange Server um very bad right like that's definitely somebody just copying stuff back and forth even the fact that they could do that means that something is not set up properly um but yeah you know this is when they started to really freak out talked a little bit already about Powershell encoded commands but this is something that happens in malicious Powershell all the time is that net um net web client so you can you know a couple lines of code in Powershell call and execute strings like it'll download that string and execute it right super easy um it's seen all the time and then finally you see this IAS web service process spawning another child's process so in this case it was Powershell and associated with that full exchange compromise this did full turn into like a full IR response investigation but without the use of cismon who knows how long it would have been to see that compromise right it could have been after this happening oh yeah you would you would have just had had a ransomed exchange server at that point all right and then with that last um detection of IIs so if you have on Prom exchange um definitely be logging your IAS logs and your process logs on your I server um that w3wp exe is the exe name of IIs which um uh the what is it Outlook web access uses and it should never spawn pow shell or command line never there are no circumstances that is a good thing no there's unless you are being maybe being pent tested but above and beyond that that is a terrible thing to ever happen then even then you let them know we found you yeah yeah for sure so you know rip off your blindfold there's a lot of different things you can do um you know you can have a gap analysis and figure out out where you where your quality of logs are now versus where you want them to be um do you have critical devices that are not logging the right stuff you have to plan for a lot of these attacks which can be daunting because there's tax for everything right but one of the first things that you can do is improve the processes of you know include installing Sison right um it could be a change in configuration you can align these with company objectives like um almost everything in security can be tied to something in your business all right whether it's reduced downtime um uh money savings whatever you want to tie it to um you can usually align this kind of activity with bolstering like your security posture okay um something that we uh created and it's out there for anybody to use is called posham um one of the most difficult pieces we had in the beginning of starting blumera was installing all this stuff uh at scale right because you can use we use NX log right which has its own configuration file and cismon which has its own configuration file but if you're not paying for them you have to have like the the right configuration or it doesn't start or you have to have the right log being generated or it just like silently fails so we were tired of having to do that and figure out all right well this server has IIs this one doesn't this one's a 2016 server this one's a 2012 whatever uh this actually just does it automatically which is super nice um yeah it figures out what channels you actually have and puts it just shoves it into config file so it makes it much much easier you can you don't have to have bla uh to use this you could just you know plug in whatever you want but yeah it was uh much easier to just do that than manually uh and then pick a configuration um you know a lot of times you saw the enrichments that sisma modular gave us with all those uh miter ttps that it lists and the it it gives the ability to turn on and off certain levels of logging especially um cismon isn't made uh for all devices right one of the one of the things we ran into is if we had a a couple customers that had really old servers with spinning disc drives like their DCS were just you know some 10-year-old box on the floor um the amount of iops that cismon has to use to write all of those DC logs that are coming in kills the drive it makes it very slow yeah so if you have ssds you're probably fine um but just know that syisa modular is out there so for the slower devices that we have you have the ability to not overwhelm them if you really really want to and I always include this too always test your always test your sim um how whether you're doing it yourself whether you have a third party doing it having an active relationship with with your MSP mssp Sim vendor or whoever there needs to be some kind of regular verification that you're attack you're you're detecting stuff right whether you have a pentest or whatever there's just so many things that can go wrong to make detections break right service stops a firewall rule gets implemented something right um there's a lot of different things that you can do to tests like there's easy password password spray stuff out there um there's just real easy commands that you can run a command line to test it yeah it one of my favorites and I did this when I was doing a blueera demo was you can just go to who am I and you'll know this it flag that because this is not something commonly run uh the testing doesn't have to be that hard uh my favorite way to test is get some accounting firms and have them as clients there's a secret class they give all accountants that say click on everything that comes through your email convinced of this I don't know what the class is called but there's something about accounting firms great because you know they're they're click vicious as I have named some of them I'm like this person will click on anything oh that's terrible yeah accounting for I don't know what it is about that particular industry they uh I thought sales was bad but that's very interesting yeah yeah we need some industry stats on what's it yeah I know right that'll be a great panel discussion what's the worst industry for clicking on things yeah we have to ask no before I'm sure they have stats on that yeah um and then last but not least if you want to do your stuff in internally this is a great combination of tools to track what you're detecting and how you're detecting those things um and that's it yeah Vector aramic that's I make sure there links to everything uh so it's easy for you click on everything in the description yeah so this is me um thank you so much for having me on I love I love doing this stuff and I love you know preaching the great word about Sison well it's fun this is this is such a deep discussion on you know we started with some of the theory and then some of the Practical as you go through it and understanding the big picture if you will it's not just the commands that are run this is like the whole top to bottom of why you need these things why we need some of the alerting and how we're looking at it as security researchers I think it's just it's critical because I get that a lot with the as I'm sure you do with the younger people starting in newer in their career how did you see that or what made you think about it and uh that's where this Theory and anomaly detection really comes into play oh yeah yeah uh I definitely suggest reading up on if anybody ever has um uh walkthroughs on instant response right so like either they've seen an attack in their lab or they've seen an attack in a customer and it's been like Anon or whatever um they're always really interesting to read like the really techn well at least to me the really technical versions of those because you can actually see oh my gosh like they went from this process to this process and that's how they did that and the the better articles I think anyways tell you what to detect on in those areas right and I spend time reading differ reports you know it's good good reading there's uh some great sites for that one of of was interesting I I posted last year I can probably find a link and leave that down there um they had actually gotten a treasure Trove of a weird differ report because it came from the threat actor they were actually monitoring a threat actor so they got to walk through what their day-to-day was of how they were doing it it was really cool um I put it on LinkedIn in a few places I can't remember if it was recorded future one I think it was recorded future who actually published it it was a great read Because very very detailed and it just they had logs of everything this person had done how they made their notes on how they attacked what was successful what wasn't but this is how we update our threat modeling for what you should protect against because I see people kind of go off and really worry about this or that when they look at their threat models I'm like just read different reports you'll understand because you only have finite number of resources uh people get excited about some of the physical layer stuff because don't get me wrong go into a hacker conference and you watch someone like J Street give a talk you're scared you're Blown Away you're amused but honestly he is probably not in most people's threat model of it's just from our day-to-day with all of our businesses it's not physical it's always fishing emails and some of the silly stuff hey lock down use security cards don't let people tailgate I'll agree with all those things but if you only have so much resources putting a better door access system in probably um they're they're getting into fishing that's the most likely once you solved all the other problems then work on the other stuff yes yep yeah for sure well thank you for this this was a lot of fun I loved diving into these things and all that fun stuff links will be down below and thanks awesome thank [Music] you

Channel: Lawrence Systems

Views: 8,418

Keywords: LawrenceSystems, log management, server logs, centralised logging, syslog server, windows event log analysis, windows event, windows event viewer, windows event log, windows event forwarding, windows event collector, windows sysmon, instalar sysmon windows, windows nxlog, nxlog windows event log, nxlog windows, cyber threat hunting, threat hunting, information security, cyber security, threat hunting techniques

Id: 1ZOhviksEEY

Length: 57min 51sec (3471 seconds)

Published: Mon Jan 22 2024