BSides DC 2017 - Open Source Approach: The Next Best Thing in Cyber Incidence Management -

the besides DC 2017 videos are brought to you by threat quotient introducing the industry's first threat intelligence platform designed to enable threat operations and management and data tribe a new kind of startup studio Co building the next generation of commercial cyber security analytics and big data product companies grab a seat everybody I'm gonna try to get right into it since the times kind of tighten I got a lot to talk about so my name is Chris NZ chief operating officer at Dunbar cybersecurity I'm also the project lead for the syphon project which I'll talk about a little later in this presentation but really what I want to talk about today is incident management and how we can take and more formalized I guess the way that every organization is doing this by using open source community driven model and you know just to start this off a little bit my background I've been in this InfoSec space for probably going on 20 years now most of my backgrounds been in DoD Intel world cleared type of work and big projects doing security for different defense systems things like that more recently I've been sort of working the commercial space and early it kind of started to look at where problems were emerging inside of commercial enterprises where they were managing high volume of incident and that could be anything from like events alerts whatever it may be but nobody seems to do the same way they certainly don't do it consistently and there seems to be where a lot of the problems sort of bubble up that catch people off-guard so this talk is gonna be about how a we my current organization is observe this I think be what we've tried to do to help address it and how that applies to security operation center environment which is what we live in every day so we run a sock as an MSSP and that's basically about as much as I'll say about that so really what I want to start with is what do we define as an incident and so I'm gonna categorize this really into two specific areas so pre-processed type of content and then post process content and to dive right in I mean we look at often is you know we get a ton of data from logs and events we've audit trails that are out there all that I'm kind of defining is pre-processed like incident content and then the post process stuff is really what comes to all of our desks and that's where you get a trail head or you get some sort of notification that is fundamentally actionable and and really this is high-volume even at in of itself so that pre process stuff is like millions and millions of events the post process stuff is prized of thousands of events and we're suffering from like an onslaught of this every day so if you're like me you got your inbox and it's completely annihilated with just crap that you've got a somehow filter out and what end up happening is you dump all that noise into some sort of subfolder and the the bet the downside of this is you just never see stuff so it's gonna get filtered by some sort of generic criteria and dumped into some sort of subfolder um this unread message count that keeps going up and up and up and eventually you just call bankruptcy and you just wipe the whole damn thing out and that's what everybody does it's every organization suffers from this and so take that into account in a team environment and you have peers you have you know your IT department or your InfoSec department or your IR team whatever it may be and you try to do any kind of accountability with that you never know who's gonna address what you're always saying hey you know Steve's the guy that does Vera antivirus stuff so he must have taken care of it and so emails crappy in a sense because it fails us all the time it's very binary it's read or unread so addressing these problems everybody says go buy sim go buy Splunk well that doesn't really work either because unless you go out and spend a chunk on enterprise security or you've got some sort of way to you know manage who did what and what audit trail or or set of accountability measure if you want to put in you're gonna have to build a custom app on top of that and sort of roll your own solution ticketing systems obviously are compute a complete failure because the second you get anything high volume you got to just basically wipe everything out anyway it's completely non actionable so with volume of issues that all of us Syria how are we able to really deal with this in a consistent manner that kind of enforces some set of policy and control so in our organization we as an MSS view were trying to face this from not just one organization but every organization we work for so we do sock services for them they sent us all their data all their alerts from all their devices and we had to deal with this in a multi-tenancy scenario we got to deal with it at scale so we started to looking at all the commercial options that were out there back five years ago we first stood up our stock and we ran into a lot of problems with the options in the marketplace or even really in sort of the open source community so sim tools just in general you know everybody's gone out and price won't probably at this point you know the story there high volume pricing you you have to get these kind of additional premium at all it's really getting a lot of the great benefits that can address some of the workflow that you're gonna have to do day in and day out and then finally you now you've got this this big big behemoth infrastructure you got to maintain as well it can get a little cumbersome and I've seen a lot of Sims get deployed and a lot of organizations and more often than not they're just a repository for events and logs and nobody ever looks at them orchestration tools are the kind of the next big thing there's some really good products coming out I think the markets a little untested in terms of those solutions interoperability is the big issue I kind of lived in a world where business process management or BPM tools were a big thing for sort of a more web services oriented orchestration about ten years ago and the findings there were yeah it's great once you the first time you set it up but then six months later when somebody upgrade something everything breaks so I think we're still going to see that I think there's a lot of meeting this especially with tools like phantom and a mule ESB and some of these other products that are out there now now Enterprise ITSM really not set up yet for security the way I think a lot of organizations want it to be and then finally threatened threat hunting tools they're really great tools they give you a lot of horsepower and capability a ton of analytics and pivot capability but sometimes it's too much for just general actionable tasks so we were looking to try to find something to manage what we were doing we kind of had a set of criteria we were looking at really the whole purpose for us was I don't care how the event was fired I just need to be able to manage every event I need to be a look at everything and drill in and find out what what happened how it happened do I care about it can I dump it and you know really enable team collaboration was the top of this we have a relatively good sized soft team they're all gonna be working on different tasks they all have different skill sets so let's enable that let's make something that refined something that can give them the ability to work as a team it's not slack it's not chat but it's something that's actually retaining some that knowledgebase we want to track accountability who did what when how why we wanted to find something to help us enforce consistent innocent management process you know we didn't want to invent it necessarily from the ground up we wanted just to kind of be able to take some best practices and pieces together so that we then created a standard we followed every day and then create a knowledge base we're gonna entertain this information forever make it actionable and searchable as well that way we can you know do some regression analysis look how people made decisions what decision may have been bad versus good and then trendline that over time we want some light orchestration I'll say yeah not go over the moon in terms of automating every little incident and you know sending things out the virustotal and you know changing this parameter in the firewall on-the-fly wasn't really something we were concerned about as much because of the fact that we have we can't decide what our clients are putting in their environments it's not up to us it's up to them we're just trying to make sure that everything at every event is being managed and then automating prior is a shin and alysus and then we have a million different source types so it has to be really flexible so we couldn't find anything long story short and we had to just say okay let's try to build it and we looked at the open source community looked at tools her in development took a little bit of idea over here from an orchestration product sample something out of a ticketing system over here and started to piece together some use cases so we came up with it with a project called siphon and about four months ago opensource this so it's available out there encourage everybody check it out I'd love some feedback as we go through this I'm gonna show you just some details about what we built contributions of the community been really helpful in that so what we decided to really address was this kind of workflow from left to right so we have information coming from a million different places it's all different formats how can we get into a system that can help us to manage the alert flow and you know starting with can we bundle things so it's not high volume a thousand alerts coming at once flood the gates and then nobody can see the other stuff so let's smash that together get it together so it's it's actionable categorize them where did they come from what's the priorities alerts and then finally connect them to other data sources that actually helping us to better analyze what's going on like why did that alert fire what action was taken what action should be taken next so in that team environment being able to show this and work with other analysts or other people on your team is absolutely critical so we have to make a capability that can connect the dots from one skill set to another on the team pass the ball back and forth and make sure things get done to some sort of level closure and then track that transition process and then finally wanted to build something that was flexible enough to respond in a lot of different ways so in a default use case I'll show you later it's really like a create a next level ticket right so we talked about ticketing systems they're only really good once you decide something needs to be done and so kind of being and we'll pass analysis into a ticketing system was the first orchestration use case we wanted to address so that was the first piece then we have open framework basically that can kick off other processes call a restful or web service API even if we have an integrated API that can be a part of a parameter solution maybe call that the block and IP so make all that that possible this is what it looks like and I apologize for the kind of washed out screens lighting in here sucks but uh all things told you can see it online so we designed this basically for sake analysts siphons and completely open source platform we've got the ability basically there to triage events as they come in we can you know give some relative level of dashboarding analytics we're working on improving this right now and we're also monitoring event flow so the big thing for us was great that we're getting actionable alerts but what happens when that device stops sending us stuff we want to know so Sims not that great at that they're there okay as long as you can set thresholds but you know sort of getting data versus not getting data isn't something that's really unless you're actually monitoring visually they're not really good at telling you that all the time so we had to solve that problem we created a priority rules engine framework that is customizable for every type of event coming through the system so if you basically say hey when X happens I want it to be this priority or when these couple of different parameters all together work in some sort of equation then I set it to this level of priority so you have that ability to custom code all that and then really our team's really doing the maintenance on it at thumb bar cybersecurity but we're really starting to see some good uptick in the community it's being built around this so I encourage you to check it out like I said before what it looks like is basically there's a waterfall as I call it in the display and that that gives you the ability to basically drill into every event as it comes through the system and within this waterfall you get alerts from really any tool you're using today at any level of your your security architecture so that could be everything from a be alerts that are just coming out of whatever product you've deployed on endpoints to maybe you've got snort or bro and bro running on your your network sniffing traffic you may get alerts from things like smog that you've already created custom queries for that are gonna fire into this dashboard so I want everybody kind of think about this as a kind of fascia above all these other tools have the capabilities that you have already deployed it's not intended to sort of do user entity analytics or you know some sort of crazy you know connect-the-dots capability that happens in a very specific type of attack pattern it's designed to get all those from all those different tools and put them into one place so you can sort of work through them and then look for correlations within that the higher level order of magnitude and then track how things are reacted to how they are managed and so we've also done some things with operational IT training like cloud watch metrics inform AWS we're working with integrating other like outbound calls to Web Services for things like look-ups for blacklist watch lists hashes things of that nature so growing the capability over time so kind of the use cases here so alert management so really anything comes through the system you can customize the alert levels users say hey I think this is gonna be high high high great alert or a critical alert I guess I can assign that the different analysts if I'm a manager on the team kind of ship that out to my subject matter experts for their their tasking throttling those alerts so get twenty thousand of the same event because of a port scan or some denial service attack that's going to be one event in the system one-line to address it so really squishes that all down and then finally tagging we're doing entity extraction out of every alert and I'll talk a little bit why this is critical but what we see in the future something we're trying to build out is an ability to do some machine learning here based on tags that are getting extracted from events so that we can basically say as an alerts managed by a particular analyst with a certain expertise or skill set these tags can be relatively similar or identical to that next learn to come six months later I can now elevate the Gir the the past analysis to the eyes of that new alert so the analyst is immediately informed of the best way to manage it and how other things been done in the past to address it so those outcomes are being tracked as well as are all customizable based on whatever preference you have for defining outcomes and then comments and collaboration so analyst comes in here starts working on this issue does some second order analysis documents it decides hey maybe I'm gonna pass the ball to somebody else if somebody else comments on that they're gonna get a notification to come back and revisit it so really keeping them engaged and then finally actions I talked about sending things like tickets out that's another area where we're you know obviously able to take some action based on every alert any event you create a ticket it brings back that ticket number so there's a direct linkage between your ticketing system and and siphon itself so pivoting so every act every piece of event or alert will be in the system obviously there's other stuff happening so not only are we capturing those post-processed events those incidents that are important to manage we're also capturing all the pre-process stuff the raw logs the stuff you find in a sim so on our architecture the backend of this is elasticsearch Postgres built on Python and react for the front-end so this is all our ability and yeah watch that slide to basically give us the ability to dive into very specific details of what happened within that alert but also create predetermined pivots or context lookups on every alert that comes in so if I say I got a endpoint alarm that went off because of some malicious unwanted application running I want to see all the net flow from different perspectives throughout the entire set of metadata that I captured from like my bro feed so we're shipping all that bro content into our sim I can then carve it up look at what actions were happening what different requests are being made DNS calls whatever it may be and for analyst it's relatively new or wants to do this really quickly they can just take advantage of these context lookups to pull data basically with these predetermined searches so that's a really quick way to like dig in a little deeper kind of find that second order of research and then move on to the next thing and honestly within these events the rate at which were managing them you really only spend a few minutes with each one that determined hey is this actionable enough to take to a deeper investigation or a deep a deeper incident response so from a standpoint of the the next layer of this we wanted to wrote more robust comprehensive search as well and when I first started this project with our team we were like alright this isn't gonna be a sim it's gonna be kind of complementary to that and unfortunately as we started really using it in our sock the having to jump into another tool was starting to become a little cumbersome so he said well we're already shipping the data into the backend elasticsearch database we have all these indexes as well as we're all data let's just create a front-end for that and so we've created a pretty robust search front-end that gives us the ability to not only kind of dive in to the alerts and then search the entire context of every alert that's ever been captured and all the analysis that was done and any comment that was made but we also can search the raw data on the backend that was associated with that so if you're gonna point him broke connection logs or connection files you can then like look at those session tags and actually track the whole thing throughout this back-end it most like a front-end for those things I'm sure some folks in here are using like security onion or you know some of these other other tools in like your lab environment or whatever it is or even in production this is a great way to actually manage some of that content ship it in with a file beet send it up here and then get one place to view all of your your different sensors that are out there so this is just another view of just sort of other data sets coming in on the raw end so as you'll see we're managing like millions of events through the system scales really well doesn't require a significant amount of horsepower to use it I mean we're leveraging elasticsearch sort of as the fundamental backend of it's really helps with scalability performance ability index really rapidly so anybody even questions feel free to jump up and and ask other tip today is set so we're also bringing really other context information so necess is an example pulling that into the system making that queryable right from here so you can kind of really pivot off a different data sets and and use other complementary information you have in your set of tools to make that analysis a little bit easier so we're looking mostly at identical content so you can define in and I'll get into the weeds I'm like the siphon at like language at the watchdog's that you define that actually are pulling in those alerts they're gonna use the the similar level of content to define the throttling and so you could set up those muzzles which is what the throttles are threating components are called to essentially look for those things and give a window of time that you want to throttle so if something only happens once an hour you may not throttle it but it's a 20,000 times within an hour yeah you'll throttle that down to a single event okay I mentioned before we're doing some light orchestration with this so right now we do integrated yura for our purposes and you know looking for other opportunities to expand the capability there we're looking at kind of shipping things out the virustotal Joe sandbox calling out blacklist checks another thing I'm really interested in doing is some some of these workflow engines that are out there like phantom and others I think there's some interesting opportunities to connect what we're doing is siphon directly to this tools so we have this content in these messages and alerts that are coming out and we have the ability to parse that out and then call any web services framework from there so we seriously just take some of that content and ship it into one of these orchestration engines and let them take off we don't we're not looking to build necessarily that piece of it but there's some cool ones out there walk off as an example that could be potential good integration points that we're looking at as well with insight and there's a whole administrative backend that makes us a lot easier from a standpoint of configuring your alerts parsing data coming in setting up alarm setting up monitors now one of the things we'll talk about here in a little bit as you know sort of how do you connect this to some of these other data sources you can also configure all your context lookups and some of those pivots that you want to have defined in the system and all that's very easy to configure within this this backend so where does this sit so really from a fundamental standpoint of like incident management workflows we are trying to get data into this tool from a lotta different pieces of the story so only think about security operations for sort of bottom-up we're enabling you to sort of dump a lot of this content into siphon from really anything from just a collection oriented side of this all the way up to that mitigation point so if the vision for us is continue to feed data in track sort of the interplay between all these different buckets and document that and if we can help with sort of you know one step to the next in the automations overbuilding I think we can makes a lot of this independent tool management a lot more easier and effective so you know it's really taking this as an example you know we see stuff that's coming in from raw logs we've got events that are firing from all these different detection devices we're running rolling queries through our sim and those are firing events and alerts bringing that all together while allowing analysts to do correlation in siphon and then taking actions that follow on to the remediation so that's really the vision and goal so we do support some other use cases as well so within siphon and we've actually integrated Fred Intel management so we're feeding a ton of data into it we've got some like long-term plans there to really make it easier to try to manage a lot of different threat Intel feeds of different formats and quality we've actually originally built siphon to do social media management so our parent company is Dunbar armored they have like 90 some odd locations of across the country and as an armored truck company the last thing you want to see on Twitter or Instagram is somebody taking a selfie with $100,000 of cash next to them so we started really tracking the social media and we built safe in the original framework for it was based on actually crawling all these social media api's and both geo-tagging are serve geofencing our locations but also looking for different topics and keywords so we did that sucked all that data in and then realized it was scaling really well and then we did it for attracting drug dealers all across miami-dade County in Florida and that was really interesting and we got like hundreds of gigs of data and for that and we said wow this is really scaling well so then we pivoted and went to a more security or use case that's more specific to what we were doing at secure Operations Center so we're also using this in terms of monitoring DevOps you know different triggers and events that are going off within our clusters and then physical security how do we tie this into access control systems cameras that have metadata and alerts that can go off also say for example we see you know other types of alarms that we want to take action on and we can create different actions that can be called into to other tools that we have so back to the threat intelligence use case threat Intel is sort of very dynamic right now there's a lot of different format types you get these notifications from InfraGard it's an example that come out in PDFs and some folks get them in an API but other folks just get these these kind of publications on a regular basis sometimes you get it just as block lists that you can download critical stack puts out some really great ones for bro as an example that you can integrate right into your sensor but there's a million different ways this data is coming out and there's some really good projects going on and sticks and taxi is an example that are trying to standardize the least the distribution and the formatting side of it so that in addition to the stuff you're just going to read in a forum or reddit or whatever it made it may be all that data is relevant and all the closer we can get that thread Intel to the stuff that we're doing to manage incidents the better off we're going to be at large so part of what I'm trying to steer the team towards with syphon is how can we take the things we've learned in processing thousands and thousands of events every day and use that psiphon through just jamming all the threatens hell we can get our access to ended siphon and using the auto tagging that we're doing to connect the dots between really good threat Intel or even decent threat Intel and the events that are coming through our system so we're starting to build the plumbing for that today and granted I have a really small team so it's gonna take time but so that's why the community hopefully engage you know if you can develop help us out but we're looking at how we can pull in more api's stream more data in the system build up this corpus of knowledge that sits right alongside of the threats that we have so hopefully we're making some good connections there so we're also trying to make more of a intentional use case on siphon and with the new different capabilities we're building to support threat hunting so more than just being a buzz word you know how can we take the tools you're already using to do threat hunting and support them support the process of using them by way of siphon and also document the outcomes so in our architecture sort of the the bulk of what's going on here is already happening at least the way we use siphon internally managed managing our intelligence you know doing the hunting itself with other tools but also documenting what those hunts how the hunt outcomes were maybe using that as a way to initiate a hunt from different programmatic api's and then finally a tracking response so we're gonna build this out more to support different expansion on this use cases different tools that are in that playbook that toolkit you use today whether it's you're using OS query or OSA core grr or whatever it may be we want to be able to feed that data any actionable information into siphon continue to build up this corpus of knowledge that helps your team understand where problems emerged what the the efficacy of it was in terms of the tools you're using today if we're seeing you know some analytics out of siphon which are saying hey we're seeing like mostly false positives for this one tool that's information that's beneficial to you so we're also going to work on I think in general trying to improve the analytics that we're reporting on within the data set we're building easy deployment options that we may available docker obviously everything's a docker compose file you can download the the scripts and everything and run it or VMware we do have a VM that's out there if you just want to do a quick and dirty build that's relatively straightforward and easy you can download it on our websites I filmed at i/o and the docker builds are on github so is recommend going their latest release that just came out this past week we ought to added the data taggers which is you know kind of coming back to the data Tigers we're automatically doing entity extraction from every message an event that comes through the system in addition to analyst comments and analysis so if we find something that's a keyword or something relevant the auto taggers are pulling that out and the core piece of that is that we won't eventually make them into articles and so we also released in this latest build the articles capability it's just really a scaffolding for it there but the idea is every tag can be associated with an article which can then be a knowledge repository for analysis are doing their investigations so you know this signature ID is represents these types of findings this port's commonly associate with these services those are the type of tag articles that we're starting with and we're gonna expand that and their goal is hopefully over time - as we start pushing more threat and telling to siphon that that's automatically generating some that article content on the fly so you're linking those pieces together really efficiently also the system-wide search and the front-end for articles and support additional actions rest and taxi that's probably gonna mean the one that six release some part of that so we got a lot of work to do but it's it is coming together so as far as learning more we've got a github repository seeing a lot of folks getting involved there get er chat if you've got support questions our team has been engaged there and also there's some really great people in the community that are helping to support each other there which is really cool to see we've got actually spent three months took a whole summer to just to do documentation so the document unlike a lot of projects is actually okay it's pretty comprehensive I had the team did a great job and obviously we have our landing page on the site from that i/o so I encourage you to check it out you know as far as just some shout outs three developers and this being recordings hopefully they see this Leela hajeck who's our lead engineer on the project has been working on this for like the last two and a half years and chase brewer who does our UI and then David Gidwani who's our DevOps guy so it's a really small team that's not a managing it today I do a lot of the just kind of idea stuff and they have to deal with that but uh generally speaking as the community's been growing we begin in good contributions some flow requests so I'm hoping to see that continue and like I said it's it's there for you guys to use and if you want to download it and kick the tires on it and try it out in your environment and you know hopefully add some value but I think it's it's something I've seen that it's been the missing piece and a lot of organizations especially where there's a large team they're trying to manage this use and ensure that nothing is missed so give us your feedback and if we've got features that we're missing and you think would be beneficial to add please tell us to get working then we'll try to get them in the in the queue or help help build them so with that I think I went through that kind of quick but happy to answer any questions you might have all right well thank you [Applause]

Channel: BSides DC

Views: 1,479

Rating: 5 out of 5

Keywords: BSides, DC

Id: aVgRf3F8JJs

Length: 32min 7sec (1927 seconds)

Published: Thu Oct 12 2017