Cisco Secure Alert: the latest from Talos on the Log4j vulnerability

hello everyone my name is tazine khan and welcome to this live stream broadcast brought to you by cisco secure and talos on any normal monday i'd ask how everyone spent their weekend to learn about their methods of relaxation but i'm not sure that's the case for everyone today as you all may be aware on thursday december 9th a zero day exploit in the popular java logging library log 4j was tweeted along with a proof of concept posted on github that targeted any applications running vulnerable log 4j code today on this live stream we'll be breaking down the impact of this vulnerability discussing why this is urgent and insight into what we expect to see throughout the life cycle of the sex boy i'm joined today by my incredibly brilliant talos colleagues matt olney director of threat intelligence and interdiction amy henderson head of strategic communications and vitor ventura senior threat researcher so let's jump right in amy as our strategic comps can you share a little bit about this vulnerability and give us the lay of the land yeah thanks taz so late last week apache disclosed the vulnerability in the log 4j library this library is used widely across many software platforms so it is a big deal for all of us in the threat hunting and threat research industry since the disclosure and the the patch release we have seen the pocs released quickly threat actors are scanning for the vulnerability we've seen escalations on exploitation attempts since friday including the mirai botnet and we expect this vulnerability to be long lasting specifically because the threat surface for this is so large uh vitor i'm going to pass it off to you you can give more details on why this is such a high priority for us thank you amy thank you amy so this vulnerability is it exists in a logging library which will be deployed in several different applications right so this represents a big issue because it will be in on the underlying of the applications and it may may not be detected at on at first stance so the mobility itself and that without getting into much technical details it exists because it allows uh developers to put any kind of information inside the log lines but this can be also code so what what this means is that an attacker can actually send a request for this piece of code to be downloaded from its own server allowing them to run malicious code on the victim's device so hence this is a really really high high risk vulnerability because of of what it allows the attacker to do matt do you have anything to add yeah this is going to be super challenging for our customers to to remediate and look at um what we have is a single vulnerability that exists in a key library that is used by vendors and developers all over the world so companies are going to have to worry about their own in-house develop applications and patch aggressively there but they're also going to have to be working with vendors across the board to determine one if the library is present and two if that library is in an exploitable condition uh it's been pretty fascinating watching the telemetry coming in um on the attacker side where we're starting to see attackers put in these triggering conditions into all different points and emails in web requests trying to find something out there that that will succumb to that exploit i think we're going to be seeing like the echoes of this for a number of months to come as this shakes itself out yeah um it sounds like this has definitely been escalating throughout the weekend into today um now are you able to share how many attack factors are possible here that people should be aware of i don't think we're going to be able to actually definitely say that i think that's one of the reasons why vendors across the board are being very careful as they analyze their own in-house products and their own purchased services because you have to you have to go through every line of code to understand how it's used and and it's difficult to express how widespread this is but by way of example one of the earliest ways that this was seen was actually in the minecraft game which is based on java uses those logging characteristics so it's it's something as esoteric as that um but also key to a number of very high profile cloud services and um and end user products yeah i think one thing to add here is uh vulnerabilities are not new right but we have to prioritize how we approach them and what matt said this is very wide reaching right the the kenna risk score for this vulnerability is actually a 93 and that puts less than one percent of their entire library of uh i think it's 165 000 plus vulnerabilities that they've scored as higher than this so if you think about you know a large scale of vulnerabilities we're dealing with this one is at the very very top um of of the risk score yeah one thing i think it's also important to mention as i was explaining a little bit the vulnerability i was saying that the attackers will download the payload but it's also important for for everyone defending understand to understand that the attackers can even exfiltrate information without ever reaching their own server through dns and that's an important point that all the defenders must be aware so that they can look also on that possible vector of exfiltration of information we have actually seen that in the wild right now so it's important to mention it wow um do you all uh expect to see or actually what do you expect to see over the holidays and into the new year so this vulnerability is even scarier given the fact that folks are looking forward to go into holiday what do you think the life of a security professional might look like going into christmas a lot like this past weekend probably vitor vitor and i were just chatting about this actually on the side um we've seen this evolution occur multiple times in terms of how um how actors adopt uh rapidly emerging vulnerabilities and so for whatever reason and i can't tell you exactly why it invariably looks like this over the past few years there will be a few incidents that we see behind thursday so going back i think the earliest that we that we have seen is december 2nd reported it's december 1st there will be a few smatterings of incidents where early early adopters had that and then what what has happened is you'll get the coin miners are very quick to adopt and we immediately saw that in our honey pots uh light up with coin mining um on stuff uh then very close behind them will be botnets like mirai um adopting them and then what we're going to see over the coming months is actors with different pace who have different objectives whether it's finance or espionage adopting this vulnerability for their ends and they typically are a little more concerned about being quiet so they may take longer to test in-house but we will see them rolling out so even though there we've seen um seem actors adopt this now we certainly are going to see very dangerous attackers using this over the coming months so our customers and people around the internet continue to be in a race against those level of actors in terms of getting patching or other remediation in place ahead vitor do you have any input well on this what i would say is that as defenders go along and patch racing patent and attach all the internet exposed systems we should never leave behind the lateral movement vector which may because well people will prioritize their patching and will patch first what's whatever is exposed to the internet but internal patching may lag behind but that can this can still be used as a lateral movement uh vector so people and all organizations must be aware of that and keep in mind that patching and mitigating internal systems is as fundamental as patching the ones that that that are exposed to internet of course with a different uh priority based on the risk assessment that was done previously unfortunately i don't think this uh winter break will be any different than previous years for all of us here we're used to this this pace and and this vulnerability i think was on everybody's mind uh going into the the holidays in terms of what's going to come up and and this this came up i think there is a there is a piece of telemetry that we saw that i think kind of illustrates some of the problems that we're going to be seeing and why why i think we're going to be seeing this over a few months um some of our partners who are doing um aggressive scanning out on the internet are reporting that they will they will deliver their scan and then won't get the results of that scanner won't get that connect back from that scan um until much later um in internet terms so minutes to hours later and what that means is they delivered what would be an attack but it wasn't processed until it got into internal systems that were handling the logs checking the logs for various things so there are there are aspects of this that aren't just internet facing systems but also back in log processing systems that are often frequently um hand built by defenders as they as they customize how they're looking for things um that are gonna be showing up for for months coming forward um that goes right into my next question and perhaps you all can expand a little bit but what are these long-term potential consequences of this exploit that we can anticipate or expect detour well one thing that we we haven't seen a lot but we need to be aware that may happen is the development of a worm based on the on this exploit because it's extremely it's it's an exploit easy to write and it it's easily wormable and if that is is something that will happen that will that will lead to mass exploitation exploitation there will be a lot of of consequences around that and that's that's not based just on on on the vulnerability itself but if you look at past events where this kind of vulnerability was it was uh disclosed these kind of things have happened so that's something that we might be must be aware of for for the coming for the coming future amy i don't think anything to add um what matt and you said right were seeing a escalations of exploitations of this vulnerability we're going to see higher level threat actors looking at this folks that are willing to wait take their time testing and we're going to see that happen over the next few months right this isn't a vulnerability that we're all going to patch and it'll go away there's a lot of different ways that this vulnerability is being exploited the threat surface is very large so that's why we continue to post new coverage as we uncover different ways that it's being exploited we continue to update our rules and it's going to be a continuous process for us as we see what uh is coming out of our telemetry yeah that that size of threat landscape is kind of really the defining feature of this so um we're here because everybody's affected like every organization out there is going to have some exposure to this most likely very few will escape um but the fundamentals of security practices are still in play here there's nothing um technological about this development that changes the game in that way so all of your internal process reviews all of your lateral movement monitoring all of your threat hunting um all of your exploration protections all of your post exploitation mitigation all of your segmentation all that is still valuable and in play um you're just because this is happening so quickly because it's so broad is you're highly suspect that you're gonna be in a situation where you're depending on those secondary defense mechanisms as opposed to blocking the attack outright and that's kind of where we are right now we're missing kind of that front layer protection from this attack yeah that is um thank you for sharing that and especially taking us back to the fundamentals right that's always important to know that we as an industry know that that is there to help us through any of these types of attacks there are a lot of people that are still coming in and i just want to reset the room a little bit um we are here joined by my amazing colleagues at talos and we've walked through um you know the lay of the land of this vulnerability and we've talked a little bit about its impact and i think um it would be helpful to walk through though what do we do now to help folks in the audience kind of get through this mitigation process um but to take it back a step do you all mind sharing how folks can understand or know if they are vulnerable and then we can go into talking about how they can protect themselves et cetera victoria you had some good analysis on in our reality so um first of all folks need to assess their their environment they need to understand if they have the vulnerability if they have if they are using that that that library where it's being used what's the exposure that they have it's it's this is the perfect vulnerability to actually use a risk-based approach because there there are so many possible um systems that are that are vulnerable that you need to take a risk-based approach approach so that you can actually prioritize what you need to to set to to patch and to mitigate first so that would be the the first the first the the first thing to do is to assess your environment and know exactly what what is what you have to do so that you can then distribute your forces around and and and do the patching and the mitigations yeah the this is one of the situations where you know for all that we're a product company um i believe that that the people that you that you have to help you build your defenses are critical and this is where that creativity comes into play and kind of adopting what is seen on the intelligence landscape you're going to be in a situation most likely where where you have vendors who are laying patches or or or um or you're uncertain about this condition and you need to be in a position where you can start um isolating and putting in additional layers of protections and working with your defenders to understand what the risk is and how you mitigate that risk is key and so having those traditional security foundations in place gives you those levers and mechanisms necessary to respond to this this really is going to be a challenging period where where you have to depend on your internal developers to appropriately assess their usage of this library to ensure that it is uh it is safe and i and my guidance there would be i have seen in my career in security developers fail at this task so if there's a way just to go ahead and patch and not say we use it safely that is the safer path um from that perspective uh and then you're also gonna have to be working with vendors directly um and so as those vendors and you know and they're all all large vendors right now are kind of going through the process of evaluating their code base and trying to figure out what to say to you about what's going on and being very careful and diligent on on evaluating how they're using those libraries and so as you go through you need to adopt okay i have this piece how do i protect myself with this piece in place and every one of those rich decisions are gonna have to be based on your environment and what protections you have in place i think to add on to that uh what makes this even trickier as well is this library is used again not only by your own internal developers but across many software platforms so you also have to take a look at your third-party uh platforms that are in your environment your partners that are in your environment everyone is aware of this vulnerability right now but talk to talk to your partners right talk to your vendors see what they're doing make sure they're patching um if they're not because they think they're not vulnerable make sure you understand that so i think that assessing your environment um but also that includes all the third parties that uh that have access to your environment as well there's been a lot of discussion recently um in policy circles about the software bill of materials um and where it would be useful this is this is your example um if every one of your your devices came with a list of all the libraries that they used it would make it easier for you to make these risk-based decisions so as we move forward in the security world and we're looking at adopting software bill of materials this is the kind of thing we're talking about where understanding risk can only happen when you're informed about what your exposure is and that's where these things come into play that makes me feel a lot better about the hours i have put in developing a list of bill of materials so i'm glad that this hopefully can help folks out there um so i know we talked about the third party vendor landscape a little bit but can folks assume that they're protected from their security products or is that a false assumption to make right now didn't quite understand the question are you saying are they protected from vulnerabilities in their security products yes oh no never assume that right so always like like everything that that you have that's a general computing piece of infrastructure you you have to treat as a potential risk like that's the right way to approach it so whether it's whether it's you know your word processing software or your antivirus software or or your your accounting software uh whatever it is you have to account for its safety so so certainly you need to to be engaged with your vendors to be like are these are these safe are they patched do i need to do something is there mitigations in place that i need to take uh to protect myself for everything in your environment there is there is literally no part of your environment where i can't think of potentially having a java based device would be something that you could see so so yeah look look uh like work aggressively with your with your vendors to to kind of get to that point go ahead vitor sorry i was gonna say this brings us back also to the to some of the basics so think things like um not all servers not all systems need to reach the internet and this goes back to my initial explanation of the vulnerability the attackers will need to download some kind of payload so we network segmentation would i cannot say that it would block 100 but it would mitigate the damage that an attacker can do with vulnerability and this is something that is going to go back to the basics like matt was saying that we need to put segmentation around the vendors and our own our own internal development as needed and that that that is really a really important point also to keep in mind yeah blocking blocking outbound ldap queries like like for example right now if you could do that not a bad idea does it protect you 100 no does it protect you about 99.9 of what we're seeing currently in telemetry yes so if you have questions in your enterprise looking at that kind of aggressive remediation just to buy yourself time to conduct the kind of analysis that you need to do that's what you need to do ultimately you're looking you you have to complete some amount of work to protect yourself from these attackers buying yourself time to complete that work and not just decimate your people trying to get it done is an excellent approach i think part of what we do here at talent not part of but the main role of what we do here at talos right is build those protections into our products so i think that's a different um way to approach this question so uh matt and vitor you guys are the experts here how are like i'm pushing this off to you guys but we've built uh several new rules right that go into our different cisco secure product line um how have we looked at those how has our telemetry led us to to build those rules what are we seeing and how do we approach it i think you're starting to get there matt so i mean there's without getting into the without wandering into two nitty-gritty of the details um i think the way to explain is there's a very straightforward and and simple way to execute this attack um and so right out the gate we rolled out those protections but immediately the analysts were like there's about 100 ways to obfuscate and kind of make this attack look different than it normally does um and that's kind of where we were focused on the inbound side on on the ngfw side so looking at kind of providing a network based detection uh for customers is the first piece uh that that team did so we first we put out that very simple approach and then we started iterating how would we change this attack to get by these defenses and there we have rolled out consistent updates over the weekend and i believe we were continuing to roll out uh defenses or additional changes today um as we identify uh both either internally or through telemetry different ways to obfuscate those attacks additionally what we're doing is looking both in terms of what we see in telemetry back from ngfw but also in umbrella and honeypots and other intelligent sources that we have um shares from intelligence partners um externally observed from open source intelligence uh identified pieces or of of infrastructure the attackers are using so i p addresses host names hashes etc and all of those protections will be driven into umbrella and uh secure endpoint and all the other cisco products so if there's a cisco product that can analyze a url or look at a domain or look at an ip that product now knows okay all of this is part of the the attack that we're seeing on log4j all of it gets blocked even though this product may not be aware of what life 4j is at all so putting in all those different pieces and driving the intelligence into the products is kind of what we're focusing on right now we also added sorry just one more thing we also added to our blog post uh some domains which may not be malicious but are being abused by the attackers to leverage their attacks so people might want to look into for those domains in their logs that might so be an indicator of possible exploitation i think the big key here again we've said it multiple times is the threat surface for this vulnerability is very large and we're going to continue to see different ways that it gets exploited and that's why matt talks about our approach of what are we seeing in our telemetry how is this being obfuscated we're going to continue to push out new rules uh into our products we're going to continue to discover new ways that threat actors are using this vulnerability so over the coming months into next year will continue to be updating uh our blog but updating our projections as well and there are there's things that we've learned like this isn't the first time we've had kind of an emerging thing lately so we think back to hafnium for example which was a similar kind of uh of pace in terms of rolling out of those exchange server vulnerabilities and what what what i would remind defenders i i think it's recent enough that that most of them are still have like traumatic memories of happiness but remember that it's not just at this point about patching it's not just at this point about securing or mitigating it's also a presumption of breach because there were actors in possession of this before that that patching came out and you are in a period now where you have variable levels of patching coverage over your products as you move forward so you need to be looking at at open source piece of intelligence like the talus blog and others grabbing those iocs and aggressively looking through your logs and looking in your environment your dns queries and everything else just double checking you know am i seeing something that tells me that i missed something or that i haven't caught up to something you're in a point right now where not only are you on one hand um patching like you would traditionally but you're also doing kind of a high-end threat analysis threat hunting exercise where you're just making sure that you're not breached as you try to get everything under control and and i i wish you all the luck because i know it's a very challenging environment right now can you guys share a little bit more of any of the resources you know especially from an incidence response perspective that we might have that are available for folks to reach us in case of an emergency of some sort absolutely uh you can always go to the the talos website callus intelligence dot com slash ir slash incident response you'll see our emergency phone numbers there uh i would say for all of our our customers on retainer right we are available to assist them uh any time that they think they've experienced a breach if they need proactive threat hunting compromise assessments i would also go back regardless of who you're working with brush off your incident response plans right that's why we do tabletop exercises that's why we practice for these events they happen recurringly so make sure that you have those even if it's just who do i need to call right make sure we know who um uh who is in your directory that you need to reach out to if you notice something amiss in your environment so we're coming here uh up on time but if there were any last words or pieces pieces of advice that you wanted to keep our audience with is there anything that you all would want to share so the first thing that comes to mind is good luck you will need it [Laughter] we are here to help in any any way that that we can of course and we will be keep we will keep updating our blog with all the iocs with all our our visibility and with new detections i think that's that's pretty much what i would say yeah when i was when i was in paramedic training they they taught us there were certain conditions um that you would only catch if you had a very high index of suspicion like you really had to be looking out for things to see these little indicators of specific issues that you would see in the field and so right now is a time to have a very high index of suspicion a lot of incidents start out with with an analyst saying well that's weird or why did that happen now is the time to run those things down um certainly uh i i think we don't have to tell you to patch we don't have to tell you any of that i know you're doing that but but those little things where you have a junior analyst kind of show up and be like hey the the uh the server in the dmz keeps pinging the firewall those kind of things have have triggered full-on incident responses in the past so now is the time to be very diligent um not to just rapidly close alerts really look in because you are behind everyone is behind the power curve right now and attackers have the upper hand i'm gonna go uh make sure you you eat your lunch and get rest take care of yourself um i know these guys know we've we've been at it all weekend we've been at it late last week um but you need you need to to take care of yourself as well and just know that the community is in this right now you know every single vendor every single partner uh we're all deep in this right now so there's a lot of information out there we're trying to keep our customers as up-to-date as we possibly can with updates to the blog when we see something we'll we'll include it in there so keep track of what's coming out um and get some rest i did have one actual other thing that amy reminded me of um so log 4j is an open source project um you know it's not run by any specific corporation largely a set of volunteers and they've had a terrible week uh and but that that software that they build is obviously critical to what happens in the world so a fantastic job to that team both in terms of building that product but also in terms of responding to this in a way that gets those patches out and all the information that customers need so very well done and i hope you all get some rest soon thank you amy and matt and vitor for joining us and the entire talos team and cisco secure for your incredible help and input and resources that are always available and i want to make sure to reiterate some of those resources so if anyone is experiencing an emergency or needs incidence response support the cisco talos emergency response number is 1-844-831-7777 and then the eu number is 44808-234-6353 um we also have our blog the link is https obviously we want to make sure things are secure so remember guys all the time secure websites blog.talosintelligence.com 2021 slash 12 slash apache log 4j dash rce vulnerability html and we will definitely have these links in the chat and available to you all we'll be sending information via our social media and like everybody said today please take care of yourselves this is the time to definitely keep your alerts on high but your community close right cisco has an incredible community i know folks are tapped into a myriad of different places but we are all available at any time that you need and we're here to support you um thank you again for everybody that's joined us um like i said the team's been working around the clock so be compassionate be gentle be helpful to one another and reach out to us if you need anything once again my name is tazeen khan and this was a live stream between cisco secure and talos

Channel: Cisco

Views: 2,843

Keywords: cisco secure alert, products, security, securityveh, talos, veh

Id: MMwUF4sHmBQ

Length: 32min 33sec (1953 seconds)

Published: Mon Dec 13 2021