BHIS - Talkin' Bout [infosec] News 2021-12-13 | Log4j | The Floor is Java

show hey everybody yeah this is this is not the uh this is not the webcast this isn't even a webcast a youtube stream what do we call this like we caught it in an emergency newscast that's what we called it today [Music] i am so glad we got the opportunity to bring that back yeah right right we haven't been here for a while so there you go apparently some other bhis people are on the actual news so there's that yeah all right yeah some actual phis folks are like what's going on why are why are why are deb and jason here [Laughter] this is a special broadcast it was kind of you know we're not we're not starting this is just the pre-show we're just pre-showing it um but it's kind of fun to watch this thing blow up on twitter over the weekend oh yes it was so when you say fun like how how would you categorize that word i i think i categorized it as i've seen this before and a number of us have been around a long time and you kind of predictably know what's going to happen um as somebody releases like a proof of concept for an exploit and a specific product somebody's going to rip on them like you shouldn't be releasing that exploit publicly you should be going through responsible disclosure and it's like well the internet is burning and everybody is panicking right now and responsible disclosure means the lawyers the company are happy um so it's kind of fun to see the same trends show up and i'll talk more about that at the beginning as i quickly try to recap what's actually going on well the other thing i've seen too is that you know we create content there's a lot of people now that create content and so when this thing arrived it was like oh wow like there's so much content that could be created and the it's great and also noisy yeah there's just so much like i i mean from friday as soon as i opened up twitter it was log 4j and i was like i have no idea what that is right now i put that on my tweet deck and it just flooded the tweet deck which is written in java then yes yeah my favorite thing is like is it is this vulnerable is this is this vulnerable yeah is this vulnerable yes yeah yes yes so someone's probably on twitter just replying yes to all of them like yes it's a twit twitter bot yes it is yeah so john you realize that there's gonna be people who use today's newscast to then talk to their leadership and the c-suite about what's going on and hopefully you don't feel that pressure right now so that's your polite way of telling me to swear less i was just thinking to myself oh god oh my god that's pressure the other reason why it was kind of interesting is we very rarely do these types of things because i feel like it's ambulance chasing and jason and i spent a lot of time talking about that when a new vulnerability comes out and i'm going to talk about what i think is a fundamental difference between something that's like everybody panic like this um or like an ambulance chasing type vulnerability it's like if microsoft releases a patch for exchange servers i'm not going to do a special webcast on that because that's just dumb i mean it's like the answer to that problem is pat your exchange server i mean that's not it's not a long conversation something like this i think is because it's uh it's very much not an easy thing to deal with we get to find out how deep the rabbit hole goes with java all the way down man turtles how deep does java get integrated into everyday apps and devices let's find out some billion i think about that number up somewhere yeah you know that there was a conversation years ago like when they first released java many people don't know this but java was like training wheels c and c plus plus for college students um it was we're gonna come up with a coding language that's not as bad as c to ramp you up and to get used to writing c code and um ultimately what happened is somebody was like you know what we should just use this crap for production and you know there was somebody that said a long time ago that doesn't seem like a good idea to me at all and they didn't listen to that guy at all so yeah that install screen that says over one billion devices run java it's like oh god juicing those protesters were like delightful is that like mcdonald's like over 99 billion served it's like how many hamburgers that's a lot of hamburgers that's a lot of cows that is a lot of floors the floor java is fantastic well done you're welcome uh so we're not gonna we'll start in about five minutes so if you're here you're here early thank you so much for being here we're on discord if you want to continue the conversation with your fellow attendees we're on youtube if you want to talk there and then discord youtube somehow talk to each other most likely through java to have that conversation it's javascript jason it's totally different yeah but now cory's going to find out hate mail somebody's going to type in just you know javascript and java are not the same thing it's all javascript always has been but there's also java apparently which is yeah apparently is a thing uh i saw a tweet today that said hey just for a reminder that today is day nine for most of your infosec professionals who started their week last week and it's just continued on and to be kind to them and so just a reminder be kind to the infosec professionals and it's right before christmas like there was one tweet i saw it's like things will ramp down in december they said it'll cool off and we'll have time to relax they knew no wasn't just way worse than that christmas tree burning no way oh tannenbaum was this the same time last year where i took a week off because i was super burned out and then there was like an emergency webcast about solar winds was that at the same time that's usually where i pick up the phone and i talk to deb yeah usually when jason leaves something bad happens like wow we're moving the whole conference virtual good lord cruise ship let's right now happen and i'm always like yeah sure we'll be fine imagine working at solarwinds right now and how many people you have to tell we don't use java or maybe they do thank you but you know they're getting a lot of questions well it's funny that you mentioned that i'm going to go through the list of companies but i think cassaya was one of the vendors yeah i'm not sure there was one of them one of the rmm tools that was out there yep because he's already dead he's already stuff i'm beating him up uh three minutes everybody three minutes if you're here early thanks for being here early this is a newscast the news happens every monday even though i'm not normally here uh it happens every monday and it's fantastic and so if you're like the news happens every monday yeah the black hills does the news every monday if you want to hit the i guess i have to say smash that subscribe button and subscribe let's do that like we always feel uncomfortable doing this because some youtubers are really good at doing things like hey and if you guys could take an opportunity and smash that subscribe button and hit the notifications you would greatly appreciate that um and then i've been watching linus tech tips lately i know other people have been seeing this but he had trouble with system 76 and trying to get an everyday driver linux system and i've been watching it that dude is so smooth at like sneaking in advertisements like right in the middle he'll be like talking about blah blah blah blah and you know what i think hats are important i think hats keep our head in for him and it's just like my god the guy's a genius hey john i like your shirt did you get that at spearfishing i did actually no it's amazing at spearfish general store they also have these blue skirt fishing shirts the red team shirts they're all they're all sold out we're kind of sold out yeah we shut this we shut this door down this morning see what i mean we suck at this so double click the like if they double click subscribe will it just unsubscribe them like will they subscribe so do not double click this subscribe just do it once once hit it once if you'd like and also the worst thing about you hitting the subscribe stuff is we don't know what happens we just think it's important the algorithm knows the algorithm knows don't worry you know what happens is some java applet kicks off uh just uh we didn't shut the store down because of a java issue we shut it down so that the team could spend the next two weeks enjoying life and spending time with their families instead of shipping things so that's why we shut it down this is a weird coincidence though yeah it is weird how that worked out yeah all right in java what are we ready john it's 4 30 4 30. all right so let's do it like i said i'll do the intro some slides about 20 minutes we'll come back and then we'll bring in the panel so ryan are you ready i'm ready all right i'm gonna start sharing my screen so it's ready for you and then you can pick the camera or my screen or whatever do your thing ryan here we go [Music] hello and welcome to another edition of black hills information security talking about news this is a very special edition of bhi is talking about news because we're going to be talking about java and logs as i like to say in my slides there are two things that i i hate like a lot so we're going to set up we're going to talk about the vulnerabilities that are currently going through the internet in like absolutely everything and then we're going to bring in our illustrious panel to discuss it in a little bit more detail so let's jump in so as far as we can tell the first time that this started with some proof of concept code was released on twitter and on github which to be honest is usually where you want your proof of concept code for every vulnerability to be dumped immediately i don't think that tang xiao fang 7 um knew exactly just how bad this would actually turn out to be um can't really talk to them i don't know but it's interesting because whenever this got dropped and was released on twitter immediately there was a wide variety of people that are like wait a minute if we're looking at log4j and logging in java in apache i think log4j is used in this app in this app in this app and what we actually saw on twitter was an absolute crap storm of people going through and finding all of these different applications that were using java and using log 4j underneath the hood and i'll talk about what this actual exploit and what this vulnerability does it's actually a command execution vulnerability and we'll talk about what that means and how it works here in just a little bit but as i said it escalated rather quickly online um here cass van goghten um he basically decided he was going to do some testing in apple products to see if it's actually vulnerable and sure enough it actually was now when this whole thing started breaking down in conversation initially people were trying product after product after product and there were some people that were responding and saying hey you shouldn't be releasing quote unquote zero days like this online and cass has a good disclaimer he said the issue is already very widespread wasn't the first one to tell apple and it's a prime example of just how deep this actually goes i don't want to say this is a pig pile but it definitely seems like it's crowd-sourced enumeration of vulnerability and attack surface for the whole internet which is honestly the way that i think things should always be done but none of this is new and the beginning in the pre-show is talking about how i found this i don't want to say humorous but i was enjoying it and i think the reason why is all of this has happened before all of this will happen again so it's really hard to get completely bent out of shape whenever you've seen this type of thing show up again and again in history so if we go back in time dan gear releases a paper from at stake that basically says if everyone's using microsoft products we're a monoculture and now all it takes is one vulnerability that can take over multiple systems now it was funny because dan was actually fired over that article because the main customer for at stake at the time was microsoft a little bit of bad politics there but it at least started talking about this type of issue as a problem and it also showed corporate america's willingness to fire people to try to keep crap quiet move forward to sequel slammer sql slammer was actually very very similar to this particular vulnerability because it was a vulnerability in the jet ops access excel dot net early.net asp frameworks for how microsoft products communicated back-end to databases and it turns out that literally hundreds of products were actually vulnerable because hundreds of products used the exact same library and we actually saw south korea go offline for a short period of time literally millions of starcraft players screamed out in anguish for that one shell shock is another example shell shock was an example of how you can take input from the web something like a user agent string which is very similar to this one and you could use a variation of the linux fork bomb where you could fork and you could actually execute shell code on a remote computer system the reason why this one's close to this type of problem that we're dealing with now excuse me is you're running into a scenario where we don't quite know the entire attack surface so if we're looking at apache well apache was installed on absolutely everything all the way down to like home routers so it wasn't an issue of just patching an operating system like a microsoft patch it was literally dealing with hundreds if not thousands of vendors and all of their individual patches then we saw the same type of thing for heartbleed then we saw it from apache struts so the point i'm trying to get at is we shouldn't as security organizations in the industry we shouldn't be in a situation where we're doing the exact same panic dance again and again and again these types of vulnerabilities show up they're going to repeat we're going to see them again i can see the future this is going to happen again so every single lesson that we are learning as we're working through this particular vulnerability needs to be something that we institutionalize in our organization and in this opening part of this webcast i'm going to give you some tools and techniques to try to help you with this as well next how does this work well what this does is it's command injection in log 4j how exactly does that work well let's talk about how logging works okay whenever you send input into an application it could be a standard application that's standalone it could be a web application those web applications generate logs okay so they're going to generate these logs and any type of application library whether you're working with python or perl or java has libraries that can be used to actually handle and take care of that logging for you the unfortunate thing about how log 4j actually works is you can actually insert jndi commands in the middle of the text and then the logging service will actually interpret that on your behalf this gets into an input sanitization problem there's data that goes into an application like an input field a username a password something you're searching for and then there's code to be executed so if i actually say run bin sh on a web server that's code when the logging facility should handle it is just text it should just log it so if we go back to command injection vulnerabilities you could do gibberish semicolon ls semicolon gibberish and for many web servers it'll actually fork the command execution because that semicolon will be a command delimiter and it'll execute the ls command it's kind of similar to what we're seeing here where the logging facility in log4j can read this jndi which is java naming and directory interface it's actually an api and then you can tell it to do an ldap command in this situation we can say ldap go to evil.org 3839.3839 and you could pull down some evil what is that evil once again the difference between data and executable code is incredibly important because the logging facility in this example is going to execute that string at that first bullet point it should just receive it as straight log data and write it to a file and ignore it don't look at it don't touch it if you do it'll think it's people then it just won't stop so in actuality what does happen is when it actually sees that it'll try to execute it now there's some additional problems like evil we could download class files and then that tells the logging facility how it needs to help parse the logs which means we can now run code execution on these computer systems and yes all of this can be obfuscated and that's why i have this wonderful hide and seek picture here because it isn't an issue of just going through and looking for a specific string like jndi and then filtering that out on your web servers or any other applications at the perimeter of your environment it actually is far more difficult because it's a logging facility it has the ability of parsing and handling a variety of different types of encodings yes this gets better so you can encode it and obfuscate it which makes it incredibly difficult for parsing and string identification for an alert like an ids signature but it'll still execute on the logging server so this whole entire platform this whole entire api is designed to just handle logs but java also incorporated the capability for it to do execution for things like ldap corba rmi dns as well so the one that's actively being exploited is the ldap functionality um so what we're seeing here if you want to boil it down to a key component is data should be handled as data it should not be handled as executable code input sanitization if it's not done correctly will ultimately lead to command injection and remote command execution on a remote system so the mitigations they seem simple right like just upgrade to apache log 4j 2.15.0 okay well what's running that in your environment i'm going to show you a couple of websites here to get you an idea just how bad this is but the answer is everything if you look at java anything that's using java is probably using log 4j and there's a lot of things that use java we were talking about many people's first coding language in computer science degree programs is java so they just get comfortable writing in java and that's what they write everything in you can also set the system property format messages no lookups to true you can set the jbm parameter d log4j2 formatting message no lookups equal to true as well you could also remove the jndi lookup class from the class path showing all of these because once again i've been doing this a long time it is very common where a patch is released and then very quickly the offensive community finds a way to bypass the way the patch was implemented so you may want to implement multiple layers of defense if it doesn't actually crash anything in your environment because just implementing the patch sometimes especially one that's rushed fairly quickly sometimes attackers will find bypass techniques get around those so if you have multiple different examples of mitigations and i would also put in block if you can outbound ldap lookups from your environment because by and large most of the attacks that we're seeing are using ldap lookups do multiple things don't just issue the patch and think that you've got it covered because now the entire hacking community is looking at this and ultimately i can i can bet that we're going to run into another vulnerability in the very near future in this particular category so running multiple mitigations is also important so i also want to cover how you can actually find this on your hosts how can we actually identify if our systems are running anything that uses java well you have some things that you can work with the first thing you can use is microsoft endpoint mapper this is the updated version of microsoft sccm and it allows you to take an inventory of installed files and installed applications on your environment the downside for this is it tends to be a nightmare to actually get this installed and working properly it's not an easy task to just do over the course of like an afternoon so that would be a long-term strategic thing that you would want to implement in your environment once again going back to the core premise that this is going to happen again folks if you want to do it quickly though you can use wmic wmic is cool for a wide variety of reasons you can implement it in powershell scripts you can run it directly from the command line it's pretty cool and it's flexible for tactical situations like this you can do wmi speech space forward slash node colon at systems.txt now the cool thing about running this with systems.txt is you can put in a list of ip addresses and host names in systems.txt so if you can get a list of all of your host names or all of your windows ip addresses in your environment if you run this as a domain administrator it'll actually execute this command on all the systems and then it'll output it as a really nice csv file and then you can import it into excel and then you can sort it and do a search for things like java so we're going to dump out product we're going to get the name version and vendor if you have a product like tanium you can use that to query your environment if you have other software inventory tools going to throw a shout out to hd moore's company rumble they actually have some queries that you can do that does software inventory and then yes most vulnerability scanning platforms like tenable actually have plugins that'll actively check for the presence of this vulnerability as well also if you're on linux systems and you want to see if maybe your system was compromised over the course of the weekend hal released this really great tweet where he said you could do fine start at the top level directory and you could do a search on modified time or m time minus four now whenever you say minus four that doesn't mean back four hours that means back four 24 hour blocks easier way is go back four days but it's basically 24 hour blocks back from the time that you actually executed the individual command and the reason why you would want to focus on like var temp dev any of these different world writable directories is if your systems were compromised more than likely it was unprivileged so they got shell on the server you look in these directories associated with uh like variable directories for web servers or any applications that may be vulnerable focus on those directories and see if there is any modifications on those systems over the time that this exploit was released also if you ever see something that hal releases online there's this great website called explainshell.com where you can copy and paste what hal shares into it and it will explain it for you which i think is funny because really it's just parsing the man pages and putting it up there it's like a more complicated way of rtfming but still it works really well to explain what actually is going on here you can also find it on the network there's this tool you may or may not have heard of never been to a webcast with me we have zeke and we have rita um zeke is an amazing parser for network traffic and one of the things that it captures is it captures things like user agent strings that's key and i'll show you on another slide here in a second you can actually do a search for the word java in your user agent strings and hopefully identify different applications that are using java underneath the hood um so you can go through do a quick and dirty grep on that for java i'll talk about what rita can do which also will give you those user agent strings as well this is an example of a z-clog for a very out-of-date java implementation where you can see what the z-clock looks like you can see the request you can see the url where it was actually going then you can see the user agent string where we got mozilla 5.0 windows msie7 but you can see java 1.5.0 underscore zero eight so if you have zeke running on the perimeter of your environment two magic things are happening for you now you can quickly query if there's any systems in your environment that are using java you could even do a search for that like jndi string to see if there's any strings of actual exploitation and you can pull all this data and you can now run rita against it to do beaconing analysis as well so why would you do beaconing analysis two questions that management is going to ask again and again and again is number one they're going to ask what systems are vulnerable we can query that through our outgoing egress network connections with z we can also look on individual hosts on the network to try to hunt these systems down that are running java to make sure that they're implemented but if we're compromised well that becomes something completely different at that point now what you're looking for is beacons you're trying to identify if there's any systems in your environment that are making outgoing connections trying to get command and control one of the big things we've been seeing with this particular vulnerability and being exploited in the wild is a lot of attackers are using this to upload crypto miners to systems and by the nature of crypto mining they constantly have to check in with other systems that creates a very definable beacon pattern and tools like rita which can read in your z-clogs can tell you if any of these systems are beaconing out of your environment all of this is free like other than mentioning tanium and rumble it doesn't cost you anything so there's really no good excuse to not get out there as quickly as possible and start implementing some of these things in your environment if not for this current panic thing that we're working through then for the one that's going to be coming in just a couple of years where we get to do all of this all over again i want to do a shout out a number of different great write-ups that are out there one anytime stuff like this happens follow rob fuller or mubix on twitter rob does a great job of breaking down these technical things with wonderful little whiteboard illustrations this one shows what current exploitation attempts look like but then he shows you what some people are trying and just how deep this is going to go this isn't over folks many people are focusing on their perimeter and the little write-ups that rob did are showing how people are actively trying to exploit logging functionality because many of our logging servers are in the inside of our environment so imagine if you will you have something that logs in the edge of your environment and then it logs to a logging server a sim on the inside of your environment and it's somehow using java they're trying to take advantage of that as well so this gives you great visibility on where this attack is actually going so watch the space that rob rob fuller provides on twitter huge shout out for lunisec.io great blog write up talking about the timeline and how it actually works and also i mentioned hdmore and rumble they had an amazing blog post as well where they basically were talking about this particular vulnerability and they absolutely get into some marketing propaganda which is fine showing how you can actually find applications that are using log 4j with rumble for inventory management but below they break down some of the effective products and services so you can see apache cassandra druid drobo flink geode hadoop james kafka spark broadcom cisco elastic well right now elastic is saying their mitigations make it difficult to exploit i expect that to actually change here in the very near future hcl vmware but this isn't an exhaustive list there's actually much larger lists that are available and i'll share share these with you um in the chat here in just a little bit but this one right here this uh from swit hack these are all of the different vendors that have issued write-ups and advisories for their product being vulnerable to this particular attack this list is substantial i've been seeing a lot of twitter feeds online that are talking about just how bad this is this is very bad and almost everything that you're receiving and you're getting from people talking about how to identify in your environment from the stuff i'm talking about on zeek wmic using inventory management systems these are all kind of stop gap solutions we have to come up with a way where we can start doing inventory of software and hardware in our environment this is a hard problem in security many times i think that there's better things that we can spend our time on but we can't continue to ignore this because as i listed out at the beginning of this little presentation we've dealt with this problem before many times we're going to deal with it again i also want to just throw a shout out kasaya my heart goes out to you guys it has not been a good year for the kasaya team at all so that is my quick write-up now i wanted to bring in the rest of our crew so that we could talk about this and talk about some of the stories and some of the things that we've been seeing um we have corey we have joff we have steve um deb is here as well ben is here um so folks that are on with me tell me like when you're looking at this what are some of the stories for this particular vulnerability is funny or interesting i think my favorite hot take was that the people who are like logging is something that everyone says you should do not only for security but also for like just code and development practices but the people who just aren't logging anything are like i i told you we shouldn't log anything we'll never have any vulnerabilities but yeah it's just funny how it attacks a weak point where you have you know really you should be using these libraries like blog4j to log user activity user input and things like that but then they can bite you it's just kind of funny how the developers that were like oh i'm lazy i don't log anything or like oh i don't have to do anything i don't have to pack corey it's funny that you mentioned that because that's what everyone said after solar winds they're like well we didn't update or patch our servers ever so this just goes to show that we should wait six to eight months before patching it's the wrong lesson you're learning here hey john yeah i just wanted to say i i did a poll on youtube during your explanation and 97 of people said your explanation made sense uh that was about 600 or so votes so good job john i'll take that it's almost like i've done this before and i was waiting for you and deb to jump in i'm like oh what is that what is that what is that so like if i go to so here's what i think i gathered like if i go to a website and i can put something in a form i can just put this code in to it and then it just breaks it well it doesn't actually break it that's where it gets interesting it could absolutely break it but what it gets interesting is if you actually do that quote-unquote ldap query that's one of those functionalities that the logging feature has if you set that up you can actually get it to execute code on the system so this is really easy to do for people that even have basic understanding of dark arts of information security but like your general person just copying and pasting a string and breaking a web server and causing its crash that's not going to happen um it just isn't going to happen in most situations i say that with that caveat because i just literally scroll through well over 200 vendors and they will all be implement like they will all be impacted by this differently as well and then my follow-up question is does this only matter in english this is only no it it so it doesn't matter only in english because the as far as like the log functionality as far as what language doesn't care um as long as you put that string in the actual libraries that the logging functionality in java is using it'll recognize that as a string that it needs to execute as well so i had a i had a comment i wanted to make uh john uh part of uh part of this this whole sequence reminds me a lot of of the java serialization and deserialization of object vulnerabilities that we've seen in the past and uh one of the aspects that i've discovered in in exploring that area in past pen testing was that there were potentially multiple installations of vulnerable uh class libraries within one single java uh java uh deployment on the server side and so i i think it it's not only bad but i think it's bad potentially exponentially because actually finding it and getting rid of it may not just be in the obvious locations on your servers right as corey said earlier how deep does it go and most scanners may not even find this if it's deep within a web app or something right so this gets into something that i that i wanted to ask you all because i honestly don't know right and i know that on webcast should be like we have all the answers no um with this particular vulnerability where it's going one of the things that i think is interesting is we have the logging functionality but it's calling jndi and i think that what you're going to see a lot of exploit developers do is they're going to start looking at what other classes in java can actually make that request jndi and how they handle that data logging is used absolutely everywhere but remember whenever you're passing things around in java it's going to go through a bunch of different libraries of functionality i don't think this is over i think that once they found that this pathway exists with the logging functionality they're going to see if they can get other applications to trigger and do the same thing as well right and those applications could be on a windows environment they could be on a linux environment so that's what kind of makes this thing so dangerous is it can be executed from just about anywhere and behind your firewall like we were discussing earlier and if you're on a pen test or a red team or doing an assessment of an organization make sure you're checking um the exploits that you're using against those companies because i've seen people putting um you know their exploits on github and those can be modified or put to launch any kind of string so just make sure you know what you're doing check your code before you go testing this in an environment right so how would i how would i validate this in a non-destructive way canary token spoiler alert yeah canary tokens dns lookup something like that and it would probably be good to use your own burp collaborate collaborator server or something like that instead of sending out client internal network environment variables and stuff like that to third-party services such as canary tokens so where you can use your own service to yeah so to explain that for anyone that doesn't that doesn't isn't familiar with that concept basically because the host is doing a dns lookup when it when it's trying to connect to that external host you can basically monitor like a burp collaborator can do it or a canary token can do it you can monitor for a dns identifier that's unique that you that you have that you generated and if the generator or if the dns name is resolved then you know that the vault library is vulnerable because you just triggered a resolution to that unique id correct so i just wanted to echo also on on something that john said that that this is going to become a class of vulnerability in the future i i firmly believe that john's exactly right there i think we'll see continued exploitation exploration sorry and research in a number of areas here um it's kind of opened a door for for people to start looking a lot deeper very very similar to when we had that whole experience uh with uh openssl many years ago that opened this whole door for people to start looking around a lot more in these fine foundational classes and there's a lot here i mean we you know i i can tell you i have found vulnerabilities previously undiscovered in java class libraries before so this is this is a big deal it's also a big deal because no click no fish no user interaction required it's been a while since we had one of those there's no it's not like oh be extra careful with your inboxes um this time it's you know honestly it's more of an incident response thing just monitor like john said monitor for outbound connections things like that that you might already have been compromised you know if you didn't have logging so actually actually that that reminds me of a really good point corey and and that's something i don't think that that came across very well in some of the explanation to date and that is it's it's been a long time since we've seen direct remote uh code execution on a network listening service and this class of vulnerability certainly is one of those uh that may not be an entirely accurate statement let's just say we haven't seen a a level this wide spread exactly um and you know remote code execution via a connected network service is is really the worst form of of uh vulnerabilities that you can have because you know anybody can come at it it's not that you have to be necessarily inside a network there may be multiple pivot opportunities or lateral movement opportunities that that are opened up by it it there's just an enormous attack surface that gets exposed with these classes of vulnerability yeah and the data is passed around a lot log data is passed from one host it's not like traditional network segmentation may not work because it's not like you have to have a service listening out on the internet it could just be input is parsed from one application then sent to another server that does logging than sent to another library that does log parsing and the shell might come from the logging server deep within the network right so it's not like oh you have to have log4j open to the internet it's just it's a library vulnerability so it can go deeper than you might initially expect just look like it's not necessarily part of your internet attack surface it could be sitting on a box that's totally inside your internal network and protect it i would say i would say that you know this is hard for people to get their head around like you said this logging functioning functionality is called it's fairly safe to assume if you're running an application on the edge of your environment and it's using java it's vulnerable um would that that i think start with that default assumption right now i'd agree sure assumed breach i got a question for everybody one is does this need to be a new attack inside back doors and breaches and two is this going to be in mitre soon if it isn't already um i think backdoors and breaches we have like a remote code execution or a vulnerability in a web service that would fall underneath that category i don't think that we need to get that granular but i would say if you're if you're an incident master you can add life to that card by saying this is the type of vulnerability that was used and i think it'll be a valid one for quite some time what you're going to see with this vulnerability is you're going to see apache is going to be patched vmware is going to get patched all of the big vendors are going to get patched right and then everyone's going to think that it's over and it goes back to what stephen corey and josh were talking about for pen testing for pen testers this is going to be a gift that keeps on giving because there's so much custom java code out there almost every single application or every single environment that you test these days has some type of custom java app that they have written that does something and because it isn't in the news because there is an active patching being released because it's created in-house this is going to become an attack vector for pen testers because it's going to be very easy for them to whip up a proof of concept exploit for these other applications and we're going to see deployments of software that are not that don't that don't have adequate patching capability to actually remediate the issue i mean there's going to be a lot of those there'll be some places where it'll get actively remediated but there's going to be a lot of other think internet of things that that that just cannot be remediated and i'm wondering if this could be an additional add-on vector to something like phishing to where instead of sending a payload that's executing process injection or something like that it's just sending these strings out to get caught by logs and then get parsed and then give that attacker access to whatever server that was instead of the user's endpoint and that's i you know honestly if i was going to put this in i'm waiting for this to actually show up in burp as another script that you can execute within burp like with intruder or repeater um so that's that's on honestly for web app attack surfaces this is where it's going to be as well i do like somebody pointed out that my lips are chapped and like that just shows that my camera is probably working pretty well so my lips hurt real bad now i'm self-conscious about it it's like nice lips john let's need some chapstick um can i just can i ask a question about cost just as someone who is just struggling outside doesn't understand too much of what you guys are talking about um i know you say zeke and rita are free resources moving forward for fixes but is there any way to estimate what the total cost is of this today you with this particular cost if you look at the damages we're already seeing organizations get hit by ransomware um you look at the amount of time that teams have put into this to try to fix it it would not surprise me if you're looking at already hundreds of millions of dollars as far as cost around the entire planet it would not surprise me if the overall cost goes into like a billion um with something like this just because of the total surface so if you're looking at like all of those different vendors every one of those vendors dev team is now spinning up like emergency patches to get out for these tools and that's expensive for a company to do and that's right now being replicated across hundreds of companies trying to deal with this but i think the biggest concern isn't those companies that are trying to deal with this it's all of these weird kind of fringe embedded device companies that are using java that are completely unaware that this even exists or even worse yet the different products that exist that are using java under the hood that can't get patched so when we're looking at the cost on this i think that that's a great great question you know what's the cost to remediate in an organization what is the cost to patch that's a series of costs to look at but then building that cost up what's the cost to help instrument us so we can detect this better in the future looking at something like rumble looking at something like tanium looking at something like zeke and rita to help work with inventory management for critical controls one and two those are all costs every one of them so it's very very difficult to say this is what the cost is going to be because there's so many different facets of how this cost is going to hit organizations as a whole that's a great question i didn't give you an answer at all so yeah so if you're if you're a developer and all of a sudden you're seeing all this stuff happen you wrote this code you made these things it was supposed to be wonderful and then you see like all the stuff that's just and do you just throw up your hands and go like then nothing's secure why not just go home i think you could definitely do that i mean that's definitely a possibility um but but that's why i'm interested in this as a field right i mean you look at all the people we have a couple thousand people that were on this honestly most of us the reason why you're attracted to computer security is not in spite of things like this but because of things like this i've been around this a long time folks for some of you that are doing this and this is your first rodeo the next few days are going to suck and i apologize for that but you know one of the quotes i love uh from i can't remember the name of the band ajr i think it said 100 bad days make 100 good stories 100 good stories make me interesting at parties you're all developing some really good stories and the important thing is not just the story behind it but but also making sure that this doesn't repeat again so embrace it embrace the suck the other thing about this so i wanted to ask like responsible disclosure-wise how like so let's say that you're the person who let's say you're the person who discovered this vulnerability would you report this to three to 500 vendors and hope that they fix it like it's kind of interesting they just dumped it on github right maybe not understanding the ramifications but it kind of brings up maybe a gap in the responsible disclosure process where how can you possibly responsibly disclose to like this is a vulnerability to log4j and then i guess log4j would have to like just you know how many places is it being used it's just an interesting problem where you have potentially thousands of libraries that are dependent on this one library but you can't really like go out and google and find like oh it's just a list you just report it to this cert or whatever and they just you know distribute it it's just an interesting disclosure thing so the closest thing that i can think of for somebody that tried to do the right thing um was dan kaminsky so when dan kaminsky came up with his vulnerability for the dns attacks it was interesting because he reached out to a handful of different vendors and companies sorry my birds are all excited and basically told them hey this is something you need to patch immediately and it was hard and if you talked to dan about it he basically said it sucked it was not fun at all i'll be right back [Music] yeah we were talking about the cost earlier i imagine the uh third-party services like azure and aws and things like that that are running these vulnerable apps how many uh people are gonna wake up to um bills going through the roof right um for all the processes but the crypto miners hitting everything because that was like the the probably the third or fourth tweet that i saw in this whole thing was like oh confirmed crypto miner install yeah it'll be funny to see offenders comp i assume they'll have to comp it's like we were vulnerable we can't charge you for the compute resources oh no they totally well they totally well well you know other other uh articles that i saw show a bunch of botnets jumping on board as well because you know immediately that you've got remote code execution they're like oh well we need another bot out there cool um so we're going to see uh you know follow-on effect there of of and the obvious is uh distributed denial of service uh via botnet controlled entities so we'll probably see a growth uh in that activity as well so this is it's one of those that's going to go on for for a number of years the ripple effects be because of severity um you know again think solar winds i think uh you know uh ms-08067 everybody remember that one favorite of all pen testers of all time right um you know it's that kind of thing i'm a big fan of o8067 and it's zero three zero two six because i'm old hey i remember it was like my first week on the job uh first time in it and stuff and i had to respond to the i love you virus and it was like everywhere yes i'm dating myself but uh just said what about mobile devices especially android ouch um once again i i think that goes back to we honestly don't know how deep this rabbit hole is going to go um that was my takeaway over the weekend is that there's a tweet that showed up about the mars roverhead infrastructure that was using the software stack and like if this if this bug exists on like a planet you know thousands of miles away from us it's going to be it's going to be a bad one someone out there i always wanted to open a rover the longest potato or the longest pringles can in the world is now pointed at mars and someone's sending the payload going over ultra long wavelength radio or whatever so but um so yeah there's been some cool things somebody said randori has released that they've actually got uh instances where this particular attack has been used um to gain aws keys um which you know yeah it's going to be able to get just about everything um the other thing i didn't talk about but i should is a number of people are saying this is going to have a long tail on it and basically what that means is we're going to fix a bunch of stuff very quickly then all those custom apps and all those lesser known apps they're going to take much much much longer to show up i honestly josh we're still exploiting heartbleed shell shock right i seriously wouldn't be surprised if we're actually exploiting this in environments probably 10 years from now it wouldn't surprise me that much at this stage of the game wow yeah and it's it's pretty easy to fuzz too like a lot of certain exploits are hard to fuzz this one you just take a string you generate it then you just spray it out everywhere you can right you spray it against your home router you spray it against your you know thermostat you know your mars rover whatever you can find you just you and then see what happens basically see what sticks this is pretty awesome we have people that are in attendance that are smarter than i am so like jen savage just pointed out um i'm a huge fan um you know for android a lot of developers have used android logging log 4j and mubix of course is here as well so like i said be watching the feed because we have people that are wicked smart sharing information in there as well so john can we get a proof of life on that bird yeah whoever commented that your canaries at home triggered that figure no i mean you're just like sorry the birds walks right off no bird noises what happened they're in the room next to me in our freezers over there and my kids must have gone over to get some food and they turned the lights on i shut the lights off while i present and they're perfectly cool with that somebody leaves the lights on and goes upstairs and then they just start freaking out at the beginning i thought they were going to put the birds in the freezer so i'm glad i didn't go get them to calm down so no i i like my bird a lot even if they annoy even if they annoy me on webcasts every once in a while as well some interesting uh question in the chat about is this wormable like do you think oh hello you could convert this into like once it gets into the network it then scans the internal network and then moves laterally and does all that i think it's possible the exploit itself would do that right it would be the follow-on payload that would then continue to look for that vulnerability inside the network so this is just an exploit right now there's no there's no payload i mean you can put any payload you want with it but what we're talking about right now is the exploit and this exploit is trivial um it's shell-shocked level of trivial to exploit so absolutely you know i i'm actually to be honest i'm surprised we haven't seen it wormed out yet sure uh how many minutes you got i mean i bet you before the end of the webcast so we've talked about this before that you know criminals have potentially like a boardroom that they go to and they have a white board and they're like all right everybody it's time to warm this thing uh who's working on that and like oh you got it cool and you got like an agile board happening and people writing code is is that happening or do you think it's just so generally like at this situation i'm sure someone's doing it in their basement but if you look at like organized crime in nation states using it um they generally don't do worms right because worms are very flashy they show up everyone comes together to destroy the worm and the people that write the worms tend to end up in prison the best way to do it is to basically use it tactically use it in specific targeted attacks be very very very stealthy about how you're using it and then once you get to the point where you're not making any money on it then open it up to a worm so i don't think we're at that point yet so john i'm writing a new app and i'm going to trust all the user input is that a good idea is absolutely a great idea and on behalf of the entire pen testing and hacker community we say thank you send me your address and i'll send you a christmas card that's great oh you'll get my address don't worry you can oh yeah i don't get it this seems like this is the apocalypse oh no bad we're not there yet well friends didn't let friends use java anyway right so like that so people are asking why isn't java fully deprecated yet um it's still one of the top 10 used languages in the world for a long time it was number one we were talking about billions of devices running it it's it for a long time you had c c plus plus and java to write your application in and to be honest that's not even an option for most people they're going to go for java every single time um so that's why this just isn't going away anytime real soon you've got so much legacy applications and people are still writing stuff in java to this day because there's a lot of java developers out there that skill set is relatively cheap and plentiful to get in to help develop apps do you think there's nation stage right now that are bummed that this is out because they've been using it for a long time and they're like oh god so i haven't had anybody call that out yet um so the hal pomerantz command that we ran earlier i think is important um we're gonna have to wait if that was something that was happening i think we'll find out by friday because many organizations that are getting hit with this what they do is they update their signatures and then they don't just look at their current date going forward but they go through and they look at like network traffic they go back and they look through logs from their servers going backwards so they'll go through and look at their web logs to see if this particular attack was actually being utilized beforehand but it wouldn't surprise me if this has been used by nation states before it actually got here the dan kaminsky exploit is one that once we started researching it there was a couple of dod entities that were actually getting hit by that um well before it was actually released publicly when alex and i think his name was dimitri released the bgb prefix attack in 2008 there was a number of people that were like well now that explains some shenanigans where all the traffic was routed through china um for a while so you're absolutely going to see that and i expect to see that kicked out by the end of this week as people go through and they start doing a backlog analysis of their previous logging data to see if they were hit before but that's going to come from very very very advanced organizations you're going to look at it in fintech financial districts uh they're the ones that are going to be doing that and also dod i think everybody else is pretty well hosed i don't expect hospitals to do that because they can barely keep up with what they have right now yeah and i think you first jason is every monday and shouldn't people smash that is that that we were supposed to do it did i freeze yeah if you guys could subscribe if you look at this and you're like i gotta have this in my life then yeah hit the subscribe button button down below if you want to talk about every time this exploit gets used i don't know we're seeing a lot of people talk about a black hat presentation in 2016. um i've seen that rumor pop up i have not seen that presentation yet if anyone has a link if they could actually share that out we'd appreciate it but as of right now it wouldn't surprise me once again i keep beating a dead horse um that's bad turn of phrase but dan kaminsky's exploit was actually discovered about five years before he released it in the sands uh reading room white paper so yes it wouldn't surprise me at all if this is something we were all warned about and everyone kind of politely ignored it there was a couple of questions or comments in the chat about cobalt strike being involved in this uh in terms of weaponization uh and and that's kind of my read on it so just to respond to that um you know if you get a remote code execution it really doesn't matter um you can you can run any code you want after you actually exploit that so if you have a shell or command channel installed on that server and you want to deploy a cobalt strike beacon then just go ahead and do so right it's absolutely doable and it will will in fact get you that extra c2 channel command and control channel capability so and the other thing i wanted to add was a shameless plug tomorrow is the regular expressions class uh starting at nine a.m eastern i think everybody could use a little dose of regular expressions yeah oh dude you updated it to include the jd or jndi stuff right john i'm gonna do that let me tonight this straight like regular expressions you could totally regular expressions for looking through logs is like simple right for something exactly totally not going to be hard it won't be obvious at all what could go wrong yeah i think another thing that we're going to see out of this that i saw a lot with magento compromises is that we're going to see cohabitation of threat actors on systems and a fight for control of those systems so i think that'll be interesting if you're doing forensics and deferred threat hunting you're going to see some of that like i've seen with magento as far as fighting for the same space um either for crypto mining or for initial access and then that gets sold off um so it's going to be a regular battlefield out there yeah usually when you know years ago back when when i was doing offensive operations it if we found someone else was on the box which was probably 50 to 60 of the time we were just like let them be um and we would also try to like jump on top of their access but when you talk about crypto mining i could definitely see how if you're on a box as a nation-state attacker somebody throws a noisy tool on that's going to get that box looked at by forensicators yeah you're probably going to see some people fighting each other back and forth on system and then what about it like that they won't even know it because they probably don't have proper logging in the first place yeah what about the either the attackers that are patching behind them or the the rogue white hats out there that are patching things for people yeah we saw that pop up where somebody said couldn't a white hat just simply create something that goes through and patches systems for people um this was this was actually a huge uh conversation that we had back in like 2000 right um and i don't think we've ever got resolution to it so what i'm gonna say is if that's something you're thinking of doing just don't um it's something that's possible it may happen but i do know that a lot of government entities they won't look at you as a savior in this type of situation they're still going to throw a whole bunch of you know like laws at you and kick your door down so be careful just be careful sometimes people trying to do the not nachi and welchia is a good example michael brought up that one nachi was released and welchio was one um we also saw worms that were trying to patch systems just be careful just just don't do that i feel like if there's a battleground it's gonna be clients it's going to be like endpoints like that kid's mining or uh minecraft computer right that's a hot juicy target for uh someone with a crypto miner you know that's where the battleground is going to play out is on systems where no one will see it because there's no logging there's no you know would this be the best time to start my own email server right now at this point yes you should start writing it from scratch and java all right cool java makes the dependencies really easy actually you just type log4j and it picks the right version for you i think he should write it uh in in java and i think he should have a php front end it would be fantastic by the way don't take anything they're telling you seriously please don't absolutely ever ever yeah this is yeah there's people here that are writing this crap down and they're like well i did with bhi i did what god told me to do don't he's not your friend php is arguably one of the worst languages ever for input sanitization just saying absolutely all right folks so let's wrap it up i want to say thank you to everybody that attended this is a bit new we didn't use gotowebinar one of the reasons is i'm pretty sure that gotowebinar uses java and i didn't want to bring 3000 hackers to that platform because that could end poorly so um so we went to youtube and hoping in hopes that youtube had this stuff actually locked down but i want to say thanks to the cast that showed up today we greatly appreciate it and folks just be careful out there and like i said embrace the suck because it's going to be a funny story that we will tell each other over beer at conferences in five years thank you so much [Music] foreign

Channel: Black Hills Information Security

Views: 19,679

Keywords:

Id: igoDXnkYDy8

Length: 65min 1sec (3901 seconds)

Published: Mon Dec 13 2021