Acronis webinar: Get Fast Facts on the Log4J Vulnerability; Act Quickly to Neutralize it Now

welcome everybody to the acronis webinar get fast facts on the log4j vulnerability act quickly to neutralize it now i'm your host james slaby director of cyber protection at acronis uh thanks everybody for joining us on short notice uh but as you see the cyber threat that we're gonna talk about today is a really bad one uh it's dangerous it's spreading fast and worst of all it exploits java which runs on an estimated 3 billion devices and services that's 3 billion with a b so there's a high probability you have it in your organization as do your partners and customers and cyber criminals have really ramped up attacks exploiting this vulnerability since it was first disclosed last friday so uh we all have to really jump on this one uh if you're an acronis manager service provider partner or reseller partner you can expect your clients and customers to be reading all the press coverage that's going on about it in the business as well as tech press and looking to you for help i think that by the end of this half hour or so you should have some answers for them i think as you're going to see um the big challenge with log4j is not about advanced persistent threat attacks or ransomware or how you're going to deploy machine intelligence in your cyber protection stack or where you're going to find cyber security talent in a labor shortage for it though the log 4j vulnerability does really underscore all of those issues and enable some of those problems really the root of the problem for most of us is going to be this kind of old and prosaic one in cyber security which is the issue of inventory you know the problem for a lot of businesses is that we don't know all the applications and services that we're running let alone which ones have jabba with logging enabled and you can't patch what you don't know is there all right let's let's get started with a few housekeeping notes the webinar is being recorded we're gonna email you a link to the recording afterwards so that you can share it with colleagues couldn't be here today your microphone will be muted throughout the event so please submit your questions uh through the zoom q a interface we have a couple of acronis experts manning that q a interface they'll answer some of your questions via text on that and they'll also hand some to me for me to present to our panel during the live q a session later in the event okay let me introduce our speakers uh our main man today is kevin reed chief information security officer for cronus he and his staff have spent several long days and nights lately uh battling blog 4j uh which is sometimes he referred to as log for shell that that's really the exploit most of this half hour will be kevin's show on the zoom q a and maybe popping up on screen to answer some live questions later our topher tebow is a senior malware researcher at cronus and colin apodac who's the manager of solutions engineering here at acronis all right let's get a quick look at the agenda it's a short and to the point one our ciso is going to take you through a kind of a crash course on the log4j vulnerability what it is why it's so dangerous how the bad guys are already using it to inject malware into java apps and services the types of attacks and damage that those malware injections can cause to happen and the steps you need today to protect yourself and your clients against it uh kevin's also going to spend some time walking through um acronis's own internal response uh to the threat and how we vetted uh acronis cyber protect cloud and other acronis products to make sure this vulnerability has been neutralized with that i will hand it over to our star speaker kevin reed over you to kevin thank you james and hi everyone and indeed that's not going to be a sales call i just will share some of my experience what we've learned and what you can use to protect your organizations or customers so let's first discuss what is involved for j and what is love for shell vulnerability love4j is a library that is being used for login in java believe it or not login is complicated sometimes so there is a full library dedicated just for that and it's quite flexible if you look at a modern application i have like a sample here um it consists may consist from multiple parts on the back end that we refer to as microservices nowadays and actually every microservice doesn't even need to be vulnerable or all of them needs to be vulnerable or written even written in java it's enough that somewhere in your infrastructure there is a component that would somehow touch this exploit and i'll explain later on how how the exploit itself works so let's let's first consider what are the problems the two versions of log4j version one and version two like two trains and so version one training version one is not affected by this vulnerability maybe affected by others but not by this one however almost all like actually all but the very latest ones versions from the from the second train are affected by this one so if you have any application that is using uh log4j version 2 to something that you most probably are vulnerable to that as james mentioned many companies many organizations do use this library and there is an extensive list of those organizations started as a i believe a github report and then is being picked up by cesar at some point and there is a link available to you so to understand how it works how the exploit works let's just consider uh this modern application as i said it may consist of a couple of micro services on the back end and each of those microsources talk to each other while some kind of rpc it could be rest which is very common could be apache threat could be grpc if you are like kind of hip and modern um could be anything now um of course some of those applications may want to log the data that we receive from our clients and then one of the most common examples would be to log a user agent which is like a browser name and version so that you can have stats about your audience now consider a situation when an attacker sends a typical request and then uh they supply uh the exploit within the user agent http header and then that request get processed by the web front end which is not vulnerable with the nginx even apache web server which is not vulnerable by the way and then forwarded to one of the internal microsources that doesn't again even to be need doesn't need even to be written in java it could be anything python php girl and um and then for rather another one and that one could potentially be written in java and that second microservice would decide then maybe to lock down the user agent that was supplied with the initial request on the front end and that is the moment when the exploit hits the log4j library and as a result the application will open and back connect in this case you can see from inside of your network back to their attacker's site in this case to their malicious ldap server or even the other application does not need to be vulnerable at all the application set could write log to a file and that would be totally fine and don't exploit executed there but then this file is going will be paused through i don't know some uh some transport with the apache kafka could be something else and then eventually it could heat some low forcing application in your infrastructure and very common one is uh elastic lock station elastic lock stash as we know now is vulnerable to the um to the city and so again elastic logs that could be exploited and then from inside of your network deep from your infrastructure it would open and connect back to the interpreter and receive comments from there and then the data may be written to the log store but by this point it's already too late the important point here is that the time between the actual request and the exploit triggered it's not immediate it doesn't need to be immediate like your locks could be delayed uh for a matter of minutes or maybe sometimes even hours maybe you like combine them together and then you do a batch processing and only then they explode exploited the trigger so you need to take this into account when you're trying to scan your premises with the scanners that now are available publicly if you don't get an immediate response uh it does not mean that your application is not vulnerable at the moment could be that the exploit will be triggered at a at a later date maybe next day or maybe by the end of the week when you do a batch process now when we understand how the exploit is triggered let's look at exploit structure itself java is a wonderful language that was built on the mistakes that we learned from c plus plus and there are a couple of things um and simple spots that were annoying to programmers among them there was memory management like it was a terrible thing and maybe everybody here on this call might try to recall more than one exploit because related to uh buffer overflows it was a very common problem back then but then it was another thing like everything should be an object like java is fully object-oriented language and the third problem that uh simplify did have as a was very um weak and not expressive standard library back then and so java decided that okay the javascript library like the ability to do things stay in the thin server would be much larger and there would be more flexibility and more things to do there so one of those things would be what is called the java naming and directory interface gndi gndi is uh an ecli buffing java is part of java that allows a java program to connect to uh directory services and it could be many directory services but the most popular is obviously ldap but there are others dns is a directory service corva if someone still remember that name i had a directory service built in which was called common object something i guess common object servers and so on and so forth there are multiple so uh interestingly um lor4j allowed to include to parse the gndi references within the strings that were passed into the log and so if you if you decided to log a user agent and this user agent eventually hits a log4j library and within the user agent itself there is a line that looks like a gndi reference and you you see one on your screen then java or log4j will tell java okay i want to force this gndr reference and i want to resolve the object and so the way how it works is that you have the reference to in gndi this is the first part of this line and then the second line is the second part is the protocol to be used for the for accessing the object as i said java is an object-oriented language and so everything is an object there so the second part of that is the protocol that you're going to use to to retrieve this option and the third part is actually the reference to what is called the lookup api within gndi that would allow you to look up for this object on the host that you are referred to now in as i said everything is an object in java but when you receive uh a few words or maybe a bunch of boys over the network uh you cannot consider them to be an object so there is a there is an extra step in between that is called visualization that converts this stream of bytes into something that java can interpret as an object so normally that would be an object that could be in a normal situation it would be an object that would contain an uh ldap response like ldap object ldif but unfortunately the way how it works is that it could be a java class and java class itself is an object because again everything is an object in java and java class is basically an executable so if if java encounters a java plus reference it just executes it so what happens in this situation is that you're trying to resolve this gndi reference that you receive from somewhere else you connect to this side and you expect to get an object back and you indeed receive an object unfortunately this object is an exhibit sorry uh is a java class and so you executed your application executes it and this is why we get this remote comment execution so how this whole thing could be used by the bad guys there are a couple of things that we already see and other cyber security companies see uh happening the most obvious ones is you have remote code execution within your perimeter like deep inside of your infrastructure that's a great entry point to start stealing your data it's the most obvious thing but also um it is something where you where the attackers the cyber criminals could start by uh doing their reconnaissance before they deploy ransomware or or do whatever they want their lower end guys let's just say that they're not uh sophisticated they just deploy crypto miners right on top of the box like they compromise the box but they don't care what is what is running there or what organization they happen to land in they just start a crypto market and they're trying to mine bitcoin or theory them on an era or any other cryptocurrency or alternatively since they land up usually they'll end up on a fed server and so it does make sense to use it for as a part of whatnot maybe for ddos attacks or something like that and so what we saw on um on our services we obviously monitor them and on on this slide you see a real graph uh over the last one two three four five six this is six days starting last friday um after exploits that we observed trying to uh heating acronis promises so this is a logarithmic scale which means that um friday you can see and every every bar is one hour so on friday the slow ones as uh just single digits hits so like we had under 10 hits in one hour and the larger war is like 100 dozens maybe and as you can see over there over there on friday we had maybe a few dozens of those exploitation attempts and then over the weekend suddenly started growing and we quickly moved into hundreds and then eventually to a thousands range and why yesterday you can see someone came up with a great idea to uh launch dns back not all that backed gndi exploits but dns like gndi exploits and those end up being in the range of tens of thousands per hour so as you can see this is quite massive um and um so we have to respond quickly let's now talk a little bit about what did you do to respond to this thread and what lessons did we learn and what you can use to protect your companies your networks and your applications so i am based in singapore which means that by the time the news about the love4j vulnerability became public on friday evening i actually was already offline with my family and i didn't catch it um and so on saturday morning i woke up to like this media storm everywhere uh and i thought okay like that is that is going to be bad and we definitely need to do something about that to my um surprise to my surprise but i was pleased to learn that my team in europe uh that is that works uh with eight hours later uh picked it up and took ownership of their whole situation and they started to uh do work to walk streams in the pro so one thing um they uh started looking at acronis applications so we could help uh and advise our customers about their security and also the second box three was to protect the cronus and uh especially acronym cyber cloud from the potential effects as you can see there were quite quite a few of them and so we did it in two stages the first stage was a walk around with like there were no response for immediate response from vendors and uh the vendors themselves were evaluating uh this situation and uh releasing their products so at the vocal room stage we understood that in order for exploit to function the application needs to need to be able to connect back to an attacker's site which means that if you block back connects from your network to the outside you can prevent exploit from running so the application will try to connect but it would not get the java glass back and so the rc will not trigger so that was the first thing we did and it was quite successful um when you do this on linux if you need to protect kind of stories they're always choice to do that it would be iptables so in our situation uh obviously the whole layout was more complex you can easily start with promoting all internal communication all internal egress communications and then denying everything else and then most probably will be sufficient as a very first response maybe we'll need to enable another couple of course or a couple of external hosts that might happen but as a quick walk-around this will work for a few hours only a couple of days and then i think by monday timeframe news came out that there is this second vector besides zelda that would allow an interpreter to leak the information about all the environment variables and so we looked at that and we reviewed all uh environment variables that would uh potentially leak if the exploit triggered and here on your screens you see a way how you can do that in your linux application in windows you can just use process explorer or something on the silver current to look at the environment variables and so in our situation we concluded that actually the applications do not have any sensitive environment variables within the memory so there will be no gain for attacker if they even execute exploit successfully however in your situation could be different i've seen reports about attackers stealing the aws secrets that were available as environment variables so uh take a look into that and then uh in one specific environment for a specific customer we actually just completely removed gdi lookup class from the log4j library again you can see a way to do that and uh those those three things together help us to function until the official fixes arrived from vendors so at this point we moved to a fixed stage and elastic for example air fixes by the end of monday and then we started applying them uh one by one in our production we're doing some testing uh we also we're running uh by the by the moment when we're sure that the environment is secure we actually run a hunting round for potential indicators of compromise in in our service and acronym cyber protect is a great tool to do that uh it allows to detect a wide range of malware as i mentioned scientific is just just deploying cryptominers and according inside over there has a crypto minor detector module both for windows and linux so you can use that if you prefer to use uh open source there are tools like rookie handler there is very generic one and then it can try you to understand whether it is some malware deployed on your host however if you want to do it at scale use some automation um and also you need to understand what you're doing because we'll get tundra output requires some expertise in how linux works to properly interpret it so if you don't know what to do i think anti-malware solution maybe is a better approach we did not find any signs of successful uh exploitation of this vulnerability cloud that we didn't find anywhere uh the situation under control until 17 hours ago when um news came out about the new vulnerability 45 0 46 which affected the most the most recent uh log4j library that was updated by apache data before so i thought okay this is back to square one and we need to re-relate everything but it turned out that actually no and in our configuration the job is mostly done and we are safe in the meanwhile uh i mentioned there was another block stream our application security team uh looked in our products and worked with r d team to understand what is uh what might be vulnerable and what is not vulnerable and at this point i'm happy to report that he already concluded our investigations and uh all across on-prem products are not vulnerable to this city and as i said uh the things that were dangerous in aquarius cloud are already mitigated at this point and with that back to james thank you so much for that kevin you know there are times when i thought it would be cool to have your job and then i see like up this past weekend and i'm pretty happy where i am uh audience i hope you can stick around for about 10 minutes while we take your questions live uh keep uh them coming to us through the zoom q a interface uh topher and colin have been feeding them to me and uh they're going to be joining us to help answer your questions so let's see what we have here okay audience question number one if my application does not have log4j uh but integrated with third-party applications that are vulnerable will the vulnerable application be able to query my application and pull the data out who wants to take that one yeah i'll uh jump in really quick i think thank you topher those awkward pauses are best ended as fast as possible absolutely yeah for sure uh yeah i mean as far as you know uh you know what one application being affected by this if it's not specifically using this it really essentially um you know obviously there's going to be interactions between different applications if there's uh any sort of a dependency that that has uh you know log for j uh you know in within that application uh then that's where you're gonna be able to see you know the potential use of this you know against uh you know anything that that may not directly use the library so yes it is possible all right we have a kind of a related follow-up question uh how can an application that is not vulnerable protect itself from third-party integrations that are vulnerable right and that's where this one really gets kind of sticky uh you know it it's you know obviously you can remove any integrations that may you know work with something that's using log4j uh you know but of course then you start potentially breaking some functionality uh you know it's it's not necessarily an easy quick fix um obviously you know if you can find the the log4j library you can update to the the latest version uh you know the that has the patching in place but uh you know it's not necessarily a quick just you know update it for you know uh just the computer as a whole uh you're going to have to look into each application and and keep your applications updated as patches are coming out from vendors all right thanks for that i i love this next question it's the appreciate the default deny mindset here uh can't all of this be avoided by removing java from your host pcs talking again about the uh the broken functionality broken applications i mean i think yeah the idea here is like let's just nuke everything and is that how do you how do you feel about that approach that's not possible like minecraft that is like the the most the most critical application at my home and i just cannot i have to guess and uh i just can't get rid of it yeah i think three billion devices and services worldwide it's kind of everywhere in our lives and you know disabling it in one place could have potentially on unintended consequences in others all right i've got a see here regarding the uninstalled java topic if the java application is not a fundamental application for your business and one can remove it then can you explain how it may create other problems so that's great that's a follow-on to my last point colin you want to take that one sorry about that um james i think that would probably be a little bit more appropriate for our buddy topher over there topher uh and i apologize i was trying to read some questions myself sorry we're all multitasking here right yeah um so uh what was that one again so it's basically uh if you decide that the application that has java with logging isn't fundamental to the business and you can remove it you know how might that create other problems uh uh yes uh yeah i mean you know again it's it's what's depending on that uh you know what you know if it's not fundamental you know obviously it was there for a reason uh you know so you know one you're you're going to have to you know figure out what are you doing instead uh you know it's it's not just a simple like oh you know we can live without this let's remove it um you know and then you need to make sure you know again if you uh remove the application but the the library remains on the system you know you could still potentially see it being exploited um you know especially if there's something else you know uh again you you basically have to find every you know every instance of what's using this and again with so many applications so many systems using this particular library it's it's not necessarily an easy task um so you know there's gonna be time and resources put into it there's gonna be uh you know dependencies that get broken and you're you're going to end up with uh most likely you know some some functionality that your your people are used to that uh just suddenly won't be there and it's going to make you know jobs difficult to uh you know to perform uh you know in addition to any other potential uh you know software breakage all right good stuff all right here's one that i can actually take uh how long can we expect to deal with this and the answer i would have there is this really has echoes to my mind of eternal blue which was the exploit that led to the wannacry and not pecha ransomware attacks of a few years ago that kind of brought the world's attention to the ransomware threat and internal blue basically exploited windows 7. microsoft issued a patch for it really quickly but we kept seeing recurrences of these attacks for a couple of years afterwards just because there was so much windows 7 out there even though it was end of life um uh people just couldn't seem to get it out of their networks they were who knows applications that they didn't were afraid wouldn't work on later editions of windows for whatever reason maybe in many cases the fact that there was windows 7 systems that they just weren't aware of and i think we're going to see something similar with this particular exploit because of the ubiquity of java apps and the fact that inevitably we're not uh there'd be a lot of businesses that won't find them all won't patch them all so cyber criminals are going to continue to quietly exploit it and one of the ramifications we've seen of that already is them using the exploit to plant hooks for later long-term apt-style attacks so they'll get a foothold in your network you may have patched up the vulnerability but that remains their lurking and provides a platform for additional multi-stage attacks so i i think we can expect to see the kind of echoes of this exploit even though we very quickly got the patches for it uh happening for uh for a couple of years easily all right um a question from the audience next is there a test tool from acronis where a specific application or url is vulnerable to log 4j i was gonna say you know we we do have uh you know our vulnerability assessment patch management tool uh that that you know can at least check for outdated versions of applications and help keep you up to date uh you know that that's something that as patches are rolled out uh you'll be able to start uh you know seeing uh what what applications you're running that are uh you know older versions that may be vulnerable to uh the log4j vulnerability yeah absolutely and adding to that as well we also have hardware and software inventory in the platform so now we can actually do scans on these assets and do reporting on that to determine which version we actually have running on that system so we're able to kind of approach this from multiple levels right and if you do miss something um everything else that's in cyber protect uh cloud uh to for instance stop ransomware attacks automatically recover from any damage they might have wreaked before we detected them and so forth another audience question has anyone reached out to java to find out if they are patching their software to prevent the log4j library from working okay let me take this one um the funnel ability itself is not in java you can argue that it is like a uncommon behavior to have this uh as i mentioned this ability to actually refer to an adapt server from within a from a thinner logo but that is not a java problem itself it's uh this log4j library problem and log4j library apache software foundation that develops log4j library did already created a fix and this is a log4j version 2. yesterday it was total of 15 and now it's 2016 because the 306 didn't cover all the cases so the fix is available from apache software foundation and uh if you want to uh update your application that you develop uh by all means you have disability right now right and i also advise you to consider some of the tactics that kevin reed talked about uh that acronis took uh in the meantime is just uh shutting down your it's the ability of that to communicate externally while you figure out what else you're doing all right uh another question that i'll take here what other kinds of threats do we need to worry about next year yeah i know we promised in our invitation to this event that we would talk about future threats we ended up deciding for the sake of time to focus only on uh what you need to do about this particular one but we do have a very cool recent research that we just published in the form of the acronis cyber threats report 2022 which talks about what's coming next that you need to worry about be planning for uh in your own environment and in services you might offer to your customers i'll i'll point to a link to that when we get done with the q a um another audience question how do i know if i've already been attacked using this vulnerability uh topher you want to take that one yeah so i mean the the you know obvious answer here i i think would be to you know check your log files uh you know you you're going to see in there the the string uh you know that kevin actually shared earlier uh you know basically where you know it's the um you know jndi uh you know with ldap or dns or whatever uh and the attackers you know ip address or or you know web domain uh you know so basically that that's going to be the main thing that you're going to look for um you know if you are seeing any you know jndi uh requests that you know aren't expected uh you know that's where you're really gonna want to you know run that anti-malware scan on your systems and and really uh you know start you know essentially triaging the issue and uh of course obviously in the logs you you're going to be getting the information for um you know what to block in in your in your firewalls and applications all right we're uh running out of time here let's try to get to a couple more questions uh what part of a kronos cyber protect cloud is vulnerable well that's a biggie i think we need to answer that one so no part of a prominent cyber cloud is vulnerable at this point the application that constitutes the server cloud itself was never vulnerable it was secure from day one and so the similar application cyber protect 15 that is deployed on prem is also not vulnerable we did have bits of the supporting infrastructure within their cyberpunk which are not part of the problem that were vulnerable and as i explained by by today we actually have mitigated them already we mitigated them by monday actually well that's encouraging i have a feeling we wouldn't be doing that wouldn't be on this call if we didn't have a good answer to that one uh okay um next audience question are only apache servers affected or can this occur on iis uh iis other web servers from my understanding its going to be any server that that has java running uh you know if uh as long as this library is included its something that i understand it would be vulnerable uh most of the the servers that you're gonna see this on probably are going to be uh you know linux so you know apache or uh something along those lines but uh that doesn't necessarily mean that windows servers aren't you know it doesn't necessarily mean that they're excluded yeah and i think it's also um just worth adding to that really quick uh the reason for the severity of this vulnerability is because it impacts so many systems out there so you will be extremely um you know probably concerned or or enlightened about the fact that you have java probably running on systems that you didn't even know in all honesty so it'll be a good exercise to go through your environments anyways and you know triage just do a little cleanup all right that seems like a good setup for the next question which is how can we actually ensure we don't have the log 4j vulnerability in our environment so given that this is mostly going to be on linux environments where you're going to see this you know you can do a search uh and i mean obviously you can do similar things in windows but uh you know linux is kind of the easy uh the easy one for this because of where it's uh mostly going to be um but you can actually uh you know essentially just do a search on your systems for the uh you know log4j directories uh and you know find the version uh there's a great command that you can find online uh that people have already been putting out their uh you know simple one liner that'll show you anywhere that you've got uh the log4j uh library and uh you know from there i you know it's just a quick uh look to see which version it is and update that if needed all right we've already gone about 15 minutes longer than we intended to so i'm gonna say that that's all the time that we have for questions audience if we didn't get to your question we promise we'll follow up with you directly uh before we go i wanted to point uh well first let me say thank you uh to kevin colin topher uh all our folks backstage that helped put this event together in a very short order uh i wanted to point you to a handful of additional resources that you might find helpful there's an e-book that i wrote earlier this year on how to reduce your technology supply chain risk um it's basically a kind of a miniature assessment framework that can use on your own organization and on your tech vendors uh to try to understand what your exposures there might be we have a couple of technical reports on the log4j vulnerability uh there's the full security advisory and an incident report we also put up a more general purpose piece up on the acronis blog you'll find that probably at the top of the page still uh today uh we also encourage you to download the brand new acronis cyber threats report 2022 this is the one i mentioned earlier which is the look ahead that we've got the threats that you're likely to encounter next year ton of great very fresh research in there uh really useful for we think for our cronus partners uh last link on that page is to the acronis cyber protection page that has a whole collection of educational pieces links to upcoming events like this um other articles analyst reports white papers so that's kind of a good uh first stop uh if you're looking for more kind of background information to help you address this particular issue and others like it next i want to personally invite everyone to join us at our upcoming chrono cyberfit summit in singapore on february 17th and 18th if you can't be there in person you can join virtually uh we always have great external speakers it's a very thought leadership and educational type of event we're not talking about acronis too much here very partner focused uh i've talked to many of our partners who attended um the previous versions of this event or just in the past weeks in miami switzerland and dubai and the the feedback there is that it's it's well worth your time if you're an acronymous partner so i hope to see you there either in person or virtually finally um i'll say if you've got the means and the inclination please donate to the acronym cyber foundation this is our non-profit project that builds schools and provides technical training to young people in needy parts of the world with the sweat and free labor of acronis employees using their own time in many cases your money really goes far there in places where it's really needed with that i'll say thanks everybody for taking the time to be with us today be safe be well be secure we hope to see you soon at another acronis virtual event very soon thanks again everybody

Channel: Acronis

Views: 176

Keywords:

Id: 9klSDAvviKU

Length: 44min 59sec (2699 seconds)

Published: Fri Dec 17 2021