SAINTCON 2019 - Daniel Dayley - Building your first SIEM with the Elastic Stack

there you go all right hello everyone welcome to our 11 o'clock presentation this is building your first sim with the elastic stack my name is daniel daly aka Chronos I'd and just to make sure you're kind of in the right presentation here where this presentation I was kind of gearing towards a kind of an entry level dive into what a sim is kind of an overview of the architecture we will go ahead and do a setup demo where we'll get one running and do a quick detection and alert if we get enough time we'll take a look at some visualizations but there's a lot of really cool technology in the sim space right now and not gonna dive deep into a lot of it I'll make references to some stuff for you to go look up later but this is really more about getting you introduced to what a sim is and kind of get you going or introduced to the elastic stack and how to get it in integrated into your sim and how to kind of use it to build a sim off of kind of an overview we'll talk about what a sim is we'll go over kind of some considerations when you're designing your sim we'll do an overview of the software that we're going to use in this demo to design and build it we'll get a dashboard going kind of go through that process we'll get some alerts generated through a last alert and then I'll talk about some things that you can do to kind of expand your sim and improve the threat detection capabilities of it so we'll start out what is a sim when people talk about sim it's actually a security information and event manager it's kind of a combination of two technologies that have kind of existed in some form or another in the past an event manager and an information manager both geared towards security events and threat analysis Incident Response basically what it is is it's gonna be a place that you can go as a security engineer to get visibility into your environment to be able to analyze threats handle incidents and just get a good view of what's going on in your network there's I think a lot of misconceptions is in the sim space is the market has kind of developed as to what a sim does not do the the first two number one thing is that a sim is not a replacement for good security controls good firewall rules ACLs your traditional network security techniques your sim is not going to replace those you still need those the sim basically gives you visibility into those but doesn't do the work for you and one thing that I hear a lot a misconception is that sim is just central logging just a place to collect all of your logs central logging is good and for a lot of companies that use it for marketing or research purposes or for customer database research it's great for that but a sim is takes the logging a step further to give you alerts and dashboards and things that are really tailored to you as a security professional to help you you know keep the bad guys out so if you if you have central logging you can actually expand that into a sim by adding that functionality so and then the number one thing that I think a lot of people have misconceptions with is in in a sim especially with a lot of the the vendor claims these days in the sim space is that a sim does not just understand your threats and vulnerabilities for you out of the box you still need to understand at some level as a security engineer the sort of threats and vulnerabilities that you are trying to defend against your sim can only help you as far as you know what you're looking for and you know what you're trying to stop it's basically just that tool to give you the visibility that you need to be able to make those intelligent decisions so there's a lot of terminology in the sim space and a lot of it means sort of the same thing from vendor to vendor there's slight variations in sort of the definitions of what things are and what things do but generally you have forwarders agents sensors basically endpoints that generate log events and these can generate raw logs or pre-processed logs things that is going to are going to give you information into your environment you have enrichment normalization and parsing basically the process of taking a raw log event and making it use of to the salmon to you as a human user and we'll talk a little bit more about that collectors in jester's aggregators are kind of used liberally across the sim space as to what they do and where they're placed they're basically endpoints that can collect logs either raw logs or enriched logs and store them either for enrichment or for long-term storage and searching so if that's all kind of the same thing and then your indexers or storage nodes and search nodes are basically the places where your long-term logs are going to be stored your long-term events and one that I didn't put on here is kind of the difference between a login an event an event generally refers to an enriched log that has the data that you need to make your sim work a log generally refers to the unformatted you know raw mineral sort of data that you get out of something like syslog or Windows Event Manager so there are a ton of really great products in the sim space each with their own excellent features and different topographies great use cases they all kind of flex a little bit on some of the things they're good at and some of their pain points here are some of the examples here we've got logarithm alienvault SolarWinds sem are some great ones an interesting thing that I put up here we've got Splunk and elastic search up here which I think are some of the more popular ones but those services by themselves are not simpler mass data products or structured data search products and the way that you use them as a sim is really just the way that you implement your enrichment and your threat detection and so that's what we'll be doing today with elastic searches will set up the elastic search cluster or well it's just a single instance and then we'll go ahead and implement some threat detection examples here so that I can kind of explain how you would use an existing log infrastructure to build a sim or even if you're starting from the ground up how you could build that on the cheap so what what we're talking about when we talk about the elastic stack it used to be called the elk stack you might have heard it referred to as that is the suite of software produced by elastic for managing massive amounts of structured data you've got beats which are generally those agents they'll go on your endpoints to forward vlogs from windows or from Linux to tell you what's going on in and you can configure beats pretty extensively to get whatever sort of information you need off the host you've got great options and windows for event logging and Linux has a great audit D tool that will give you a ton of great logs here log stash is an excellent excellent tool that I recommend in almost any sim space for enrichment because it has very extensible capabilities it's it has its own programming language essentially that lets you write your own detections for the events that are going through it whether they come from a network sensor or from an agent wherever you're getting these logs from Wallach stash can make them better elasticsearch is really that kind of the core of it that's your database or your where all the data is going to be stored your storage node there and then Cabana is really just your UI for that it's your interface into elasticsearch and so there's also a lot of great tools that you can add to this stack to enrich your sim and give it more Sims Pacific capabilities to that I'm gonna talk about today our Zeke formerly bro the network security monitor and a last alert which is actually developed by Yelp as part of their security infrastructure to do automatic alerting out of elasticsearch and give you any sort of alert you want whether it's email JIRA tickets or slack messages elastic or last Laura can do pretty much anything you need it to so the with the architecture that I'm about to kind of demonstrate there are some advantages and there are some disadvantages that I think you ought to be aware of when you're thinking about designing your sim the obvious advantages with using something like elasticsearch is that it's free and open source it scales really well either your weather your container izing it whether you just want to stick it on a big host it scales really well however you want to scale it and as I'll talk about here in a second it actually is the backend for a lot of open source sim projects that you've probably heard of as well so if you get familiar with this model of Sim you're probably gonna be pretty conversant and some of these sim technologies that you may have already heard about and a lot of these things like I was talking about logstash and beets they're actually compatible with most of the proprietary components things like elasticsearch can receive logs from other proprietary forwarders so if you have for example an anti-virus that you've distributed across your environment and it's dumping you know antivirus logs you can still pick those up and put those into your elasticsearch environment even though there's proprietary logs and it's easy to kind of build out and expand one of the major disadvantages to elasticsearch is the amount of legwork that's required to get everything running it takes a little bit of time and understanding to really get what's going on there's a lot of moving pieces as I've kind of talked about already it is very in my opinion it's difficult to update like you can have a major issue if you do not update it correctly and if enterprise support is going to be an issue in your environment then open source software you're always going to run into some trouble with that and the probably the biggest disadvantage to this is that your threat detection definitions are going to be all handmade this is where a proprietary solution is really going to shine because it will give you threat definitions and they'll sell you on those threat definitions and they'll be great you know threat detection ability there but it's like I kind of said earlier it's kind of our job as security engineers to know what kind of threats were up against and to know what vulnerabilities were up against and to do our best to kind of identify those and to build that sort of detection into our environment because what you're detecting at your organization is going to be different than what someone else is vulnerable to or the type of threat that another organization may have so you do still need to understand that these are three great projects that I recommend that are all elastic based Sims the first one Malcolm is actually started out at the Idaho National Lab which just released this summer and it is a great project that integrates basically most of the components that we've talked about here with some additional network security analysis tools that are really great security onion in the middle is probably one you've heard about quite a bit it's an all one package with kind of a its own interface to manage all these different components without needing to go to each one and configure them all manually and then the third one is a patterning Metron which I actually haven't had a ton of experience with it's a relatively new player in the game but it's a it's getting quite a name for itself so kind of a you know a general overview of how a sim is going to be implemented in your environment you're gonna have the you're gonna have network loggers of some kind whether they're network security monitors or taps that go to some sort of quality assurance analyzer or something and those are all going to generate logs you'll have workstations that will generate their endpoint logs and along with workstations you probably want to get your server logs as well with those and those are all endpoint logs they all go to some single place where they can be either enriched or stored or both depending on you know how many resources you want to throw at it and then from there you'll query it with your dashboards searching and reporting and things like that so basically the way that we're going to implement it today we've got Zeke which is that network monitor it's gonna take traffic from anything it hears on the network and start generating events say I saw a DNS query I saw the connection here things like that it's gonna be collected by our beats agent which is also going to live on the host and beats is also going to pick up our syslog messages and going to start sending those over to log stash where we'll go ahead and do some basic enrichment on those logs we'll put it into elasticsearch and then visualize it in cabana so I do have I have the demo here which I don't know if we'll if we can see it very well I've got the URL down at the bottom but basically we start out with a default just kind of Ubuntu box and a bun to 18 instance we'll install everything on this single box and we'll go ahead and and run it all in a normal architecture I wouldn't necessarily recommend doing that maybe it works for your organization but you're probably gonna have a more distributed architecture like what I was talking about earlier so the first thing that we'll want to do is go ahead and get the sources added for the elastic stack we'll just add it to our apt repo that makes it real easy to do the rest of the installs we just do a apt install elasticsearch we'll do an apt install Cabana an apt install logs - and then we'll do an app to install file beeped and all this goes relatively quickly what we're doing here we're probably gonna get it going in about 10 or 15 minutes it might take you a little bit longer depending on your environment this is having to have a blazing fast internet connection which was great but I'm gonna have the instructions for this as well posted at the end so if you want to follow along with the kind of what these commands are what we're doing I'll have all of those laid out in a walkthrough for you as well as links to the the recordings here into the slides so we'll go ahead and get those installed we're getting Cabana installed here and so this is the those those first four pieces of software we talked about beats elasticsearch Cabana and log stash and those all come from the same place so it's relatively quick to get going and then that's going to continue to download here you see if we just fast-forward some of this yeah it's still logs - all right so then we installing a last alert is a little bit more complicated we actually have to kind of install it from source and that's I mean it's not the worst thing in the world there's about six commands here that we run to install Python install pit to install last alert again a last alert is going to be that service it's going to check elasticsearch every minute or every 10 minutes or ever F and you want it to and it's going to generate alerts based on definitions that we provide so if you want to detect certain things and alert when certain things happen alas alert it's going to provide that for us so we'll go ahead and clone that down and start building it here that will take just a second one interesting thing about Alaska is it doesn't come with a service I go ahead and give you the instructions here on how to make a service so that it starts up automatically for you and so that'll be kind of nice here I'm gonna just move forward to make sure we don't go over here alright so we get a last alert installed so the first thing that we're gonna do here with last alert is create that service like I said that's pretty straightforward you just kind of enter the service details and we'll get it enabled here one of the I think probably the most difficult one to get installed here is Zeke which we'll go ahead and get that installed so we we're gonna clone down the the app to repo order I guess add the app to repo and then we'll go ahead and pull it from there in your environment you may want to consider building it because you but could potentially get performance benefits from it it does take a little bit to compile though just be warned on that so that will go ahead and install it may prompt you if you want to configure Zeke to send email I just go ahead and hit the default prompts and then it will not send email by default so all right so we get that set up now we want to go ahead and get configure we want to configure Zeke to be able to grab all of our packets off of this one host so I just went ahead and enabled promiscuous mode here and typically on your Zeke you're gonna have a separate interface that you want to have a tap off your network getting a copy of all the traffic that's going to and from your core router or your main firewall into your environment so you can kind of see everything that's coming in and out this could generate a lot of data for you and so that's why there's a you should take into consideration basically how many resources you want to give your Zeke box or maybe even a Zeke cluster to handle all of this data depending on how much you have but for this one we're just going to install we're just going to use the promiscuous mode on the adapter and we'll go ahead and and listen on that traffic so we'll change the adapter interface here to the one that we have and we'll move on to to changing the output policy this is actually a big one the Zeke has to logging modes one is JSON and one is plain text plain text logs or sis logs are just a pain in the butt to enrich and parse because they have weird delimiter - some of them have some of them have commas if you have a common parsable format something like JSON or llamo even XML like windows event logs is he really easy to parse because it's already structured somewhat so log stash can intuitively pull out fields and you can enrich those fields without having to do your own extraction you can do that in log sesh you can do anything you want and log stash with vlogs but it just takes quite a bit more legwork so after we do that we're gonna go ahead and start Zeke on the interface and when we do that should start generating logs for us what we're gonna do here is I just did a ping of yahoo.com and wait a second and then we just take a look at the bottom of that DNS log that Zeke generated Zeke detected that DNS query and put it in a log and said this is the IP this is the hostname looked up this is great for now we are moving into a DNS over HTTP world where this will become less effective so keep that in mind as you're considering some of your threat detection but for now it seems to work well enough and it's a great way to catch low-hanging fruit you sometimes have to use your imagination with your threat signatures on some of the things you want to catch and there's a lot of low-hanging fruit that you can catch like taking every DNS name that's queried and putting it against a known malware list I mean that's real easy way if you if you have some common malware that gets into your environment and it's querying out on the network then you're gonna catch it by just looking at your logs and seeing that it's hey it's squaring a known malware list so now let's see we're gonna go and setup file beat file beat again is that agent that was gonna live on the machine you're gonna put these on maybe your Linux servers or your Windows servers to grab event logs and send it into elastic I've got a config here that's already configured to send it into our local elastic stack or elastic search and again I'll have all these configs available here at the end for you to go ahead and read on your own time and I'll be available if you guys have any questions about setting this up we're kind of going through a lot of information really quick here so we'll get that we'll get that file be configured here and we'll get it restarted so that it's running the last thing that we'll need to do is make a quick change to Zeke's path so that it knows where to look for these logs well sorry I misspoke we'll change file beats path so it knows how to pick up Zeke's logs off the host and put them into elasticsearch and then so we'll just add those paths here and get those done then the last thing we're gonna do on the elastic sack is probably the most intensive and if you are using an elastic based threat detection system log stash might be where you spend most of your time it's the enrichment part of the pipeline where we're taking raw log events and doing things with them what's really cool about this is that you can you can go as far as you want with log stash enrichments you can use look-up tables you can use you can use rest queries you can do just about whatever you want to put additional data into your log and so you can end up with these massive massive files that continue to do this great threat detection so like Sasha is a great way of helping you organize all that with these ideas called pipelines and a pipeline is basically a step by step process that logstash takes to make your log better it has an input where do I get my logs from a filter what do I do with my logs and an output is and where do I put them something to consider that I didn't really have time to go over in this and I've mentioned it earlier is to put a message queue somewhere in here and all that the message queue does is it's a place for your logs to sit either before they're being enriched or after they're being rich before they go into elasticsearch and it's just kind of a great way to buffer when when your events throughput doesn't match for example if you have a normal logging ray and you're enriching just fine but all of a sudden you have a major traffic spike and all of a sudden you're getting three times as many Network events as you were before coming into log stash you know log stash you're gonna say wow this is going to take me a little bit and it's going to start to queue up and the rule of thumb with your sim and with log in richmond is to always make sure that your enrichment events per second the number of logs that you're going through and make better or is higher or faster than the number of events that you are ingesting or taking in from the rest of your environment this takes some calculation as to the number of resources that you want to actually throw at it because if you get to a situation where you have more coming in you may be able to buffer that for a period of time you know maybe you know that it at 8 P 8 a.m. every morning everyone's logging into their workstations you can have a spike of logs but that's gonna go down so you can buffer that for a little bit you don't need to throw a ton of resources at it but if you just are constantly having more events come in then you can handle you need to consider potentially putting a message queue in potentially giving more resources to your log stash instance so we've gone ahead and implemented our pipeline's here and I'm just checking to make sure that file beet is running we're gonna go ahead and modify log stash log stash is a really cool feature here where when you make a change to any of your configuration or any of your enrichment pipelines it will go ahead and reload that pipeline automatically so you don't need to restart the service you need to edit anything and I'm just enabling that here in the default config file so that will work for us so then our main enrichment pipeline we're gonna go ahead and build this and what we're gonna do is this is a quick and simple enrichment pipeline that will go through and look for any source or destination IP address that we get from our zeke logs and compare it with a table that i've prepared here of known tour endpoints now in our previous speaker robert was just talking about the great uses of using the darknet and and bringing it to the light and i think that's great i think right now in some of my use cases i there's not a lot of real legitimate uses for users in my organization to be on tour you know I think that more often than not that's gonna be a good indicator that something's not right in the environment so what this is gonna do is this is gonna tag any log file that comes in whose IP addresses has a destination or a source address that it is tor and it's just gonna let me know in the log so then when I can go through the logs and start searching I can say hey you know show me all of the connections to tor and it can go through and show me all the IP addresses when it happened any other connections to tor and this is just an example kind of starter threat detection example so we've gone ahead and implemented that config there and we were going ahead and and starting the logstash service here and now we're gonna be ready to we're gonna be ready to move over to cabana which is our dashboard it's kind of our visualization interface here all right I'm gonna kind of skip this part because we're running short on time here make sure we still got it here there we go all right so once we get cabana running you can pull up the web UI and this is going to be kind of your dashboard this is what you're gonna see when you first get in and right now we do we have been sending data to it through file beat and elasticsearch remember Caban is just a visualization for your elasticsearch elasticsearch bundles your logs by type into bundles called indexes and it usually bundles them on a daily basis there's other ways that you can do it but the first thing that we need to do to really use these is are to make index patterns so we'll go ahead and create two index patterns here that will help us distinguish our data and you know kind of keep our logs separate because not every log has the same events our system logs certainly aren't going to have the same fields in them that our network logs are so we'll go ahead and create those two indexes here those index patterns here this will just basically help us see all of the indexes that elasticsearch or that log stash is going to make when it puts them into elastic search and search all that data all as one so once we get those added I gotta make sure I remember to select the timestamp for these we can go over to the discover tab here and we can start searching our data so let's see if we've got any data that we can actually pull up here we can see our events and we can see some of the data from the event so I'm gonna switch over to Zeke because if I want to look at our network monitor these are all of our network log events that are coming from Zeke so now we can check and see if any of our data that we were just setting up with logstash is working first of all I've got this destination dot tor filter applied so we can only see events here that that have a destination of Tor so I'm going to go ahead and make a connection to a tor example just using netcat just to to kind of prove the point here and this will this should generate a log that's detected by Zeke and that has my IP address and the IP address of the IP address of tor in it and then that will show up as a as a flagged log for us to review so we'll go ahead and refresh that all right there we go so you can see these three events that just showed up here it says the that these are indeed tourist destinations so if we come down we can see that was you know what the IP address of the machine was what its name was when it was that it connected and that it was indeed a tor endpoint so we're kind of at a time here so I'm gonna go ahead and give you some further reading or some things that you can do to continue to build this out as well as links to our slides things that you can continue to do are just enable control and elasticsearch elastic made that free as of May you get free user accounts for HTTP all included they just bought a great company called endgame so they're building out their their sim and they do have a SIM module in here I've used it and it's still in its infancy it's still really finicky to get working ID but I think that it shows some promise I think in the future they're gonna have some really cool stuff with this so that's something to keep your eye on just understand what data that you have try to understand where you aren't seeing things where you need to see things and shave down your false pauses so that you're only getting alerts on things that you want I'll go ahead and give you the the links to the slides here so you can go ahead and review that the walkthrough is posted here if you want to go through and finish that up and watch the last alert section here and the slides are here as well so that you have all of that information and of course if you have any questions feel free to hit me up here at the con or on twitter my handle is chrono side and thank you so much for coming [Applause]

Channel: SAINTCON

Views: 3,571

Keywords:

Id: jyl13RQniDY

Length: 29min 35sec (1775 seconds)

Published: Thu Oct 24 2019