Are all BSDs created equally? OpenBSD vs NetBSD vs FreeBSD

[Music] let's have a big hand for a yeah bank Randall also okay yeah so as previously mentioned my tailors are all because he's created equally a survey of bsd kernel vulnerabilities start off with Who am I my name is Julia fan spindle I work for a company called ioactive I did spend the last five years in America but I recently moved back to Belgium a couple of weeks ago I I'm a computer consultant I do pen test code review I break stuff on a profit I've been going to this EC Congress since nineteen c3 it's been a while and I've spoken a couple times before I think this will be my seven or eight time and let's see how this goes right so my meditation is sort of split into three four ish pieces there's an intro which will be the next slide and then there's a little bit of data as I'll show and then there's lack of data so I want to test a couple of things and then once I've tested that and I've got some results I will present my results in my conclusions right so we'll just talk about bsd kernel vulnerabilities it's a comparison between the different current bsd flavors as for what I expect from the audience if you really expect some basic UNIX kernel ish knowledge not too deep but but some would be nice people that I think might appreciate your stalker you know if you're low level security - yes if you're USD or BSD geek and then if you're loose guy you might like this - and then generally people that are curious about OS internal security I think mine enjoyed this presentation for a move on I have to mention you know some people that I called is standing on the shoulder giants because you don't do this by yourself you've build upon work by others so there's number of people that have done interesting because eternal security over the last I know decade and a half maybe two decades some of them might even be at the Congress so Sylvia you know our lunch others that it's a really interesting work so really this this presentation was started with a post to me list years and years ago by Theo de Raadt who if you don't know who he is is the the main guy behind open BSD and I remember reading this 12 years ago and it's been sort of stuck in my head since then and it's basically quote from Thea where he goes you know if the Linux people actually cared about quality you know as the OpenBSD people do they wouldn't have they would not have had as many local root security holes as they had last year which is that year which is 2005 and Thiago's how many is it like 20 or so and so I think that was you know when he makes that sort of said Oh 20 years so I was like really you know is that true is it what's the data behind that can we look at some of this and so I basically started looking at the last almost 20 years of Linux kernel vulnerabilities and what the numbers are and so I went to CV details and they basically keep a record of his stuff and it goes back to night 99 and if you look at it you know the first three or four years it's you know in the teens and then starting 2004-2005 sort of jumps up goes up to the hundredths and then this actually does this table is from July of this year so 20 70 numbers aren't correct and so in July 2017 it was 346 Linux kernel horner abilities but as of today cuz I checked earlier it was 434 police colonel or abilities and so as you see these numbers are kind of sort of growing now CVL doesn't have the same numbers for like bsd kernel issues they have like general bsd numbers but that that's all encompassing the OS and I wanted to specific because the kernel numbers and so what I did is I sort of scrape the FreeBSD OpenBSD neppy is the numbers I sort of put them into a nice little table and if you look at the tables here for ESDS you'll see that the first couple years they're kind of on par with Linux but then the Lynx ones kind of explode and but the PSD ones will sort of more or less stay the same either single digits or low teens it doesn't really go over 17 and that's that was only as the year ago so if you look at these numbers TOS observation was pretty a pretty astute observation in fact 20 was a low estimate it turns out it was significantly more and so you see that numbers for BSD analytics over the years are not the same or not on the same level and so when I asked is our date on the equal footing like obviously there's a really large community behind a loose kernel if you look at the the BSD is it's you know that's a small group of developers or small group refuses that are using these things and compiling themselves and then so forth and so I wonder if that is if that's the reason or if there's a reason why the numbers aren't the same or if it really is as Tia said if it's code quality and obviously one of things I consider is well maybe it's the mini eyeball sting and I know there's many eyeballs thing is you know very convoluted thing but but there's some truth to that and so I was wondering if there some true to it in this case well I'm the only way to really do that is to to test it to go and see what what did what the code is really like and sure enough there's a guy that did this 15 years ago Silvio and in a blackened presentation about this 2003 14 years ago and basically he they did an audit of the be a season loose kernel and sort of drew some clues from that and one of the things that he said was he said well there really isn't much difference between the loose girl and the bsd kernels in terms of security auditing they're equally ish broken it's easy enough to find bugs of both of them and the same kind of bucks now that was 15 years ago now half things changed since then the other thing he said was that you only spent a couple isn't be as either than three months on Linux and so if you had span longer on the VSDs word that of where you've gotten more bugs would you sort of hit a limit right and then third one it's already if you look at his presentation and this is 15 years ago he mostly looked for mostly found trivial flows and and info leaks and nothing else and so the opening I had is what if I expand the number of bugs I look for what if you have to look for you know race conditions and things like that and so the data is interesting but it's kind of a little bit outdated and a bit too limited at the time I think it was perfect for today's purposes it's not quite good enough and so the only way to really figure out if those numbers line up for don't line up what the reason man is is I have to dig in myself and so I spend April May June and July of this year looking at bsd kernel code I don't know exactly how much time I spend on each one of them but I would say they're more or less equal and so basically what I did is I asked myself if I was going to look for bugs where would the bugs be where do I think they would make mistakes and so I made a list of sort of attack surfaces that I wanted to go look at and none of these are I think overly surprising obviously the the common stuff is like yeah obviously I'm gonna take a look at some sis calls and I'll take a look at the T scrappy stack and then yeah I'll take a look at some drivers and I'll look at the compat code and the trap handlers and the file systems and you know other networks and stuff and I essentially spend three years months looking at that and so what I wanted to do is I want to jump over some of the things that in each of those areas that I found and if so discussed that briefly and I have a couple of demos in here as well and then once I've run through that I'll sort of run you through my results of my conclusions so yeah so let's dive into system calls obviously this is prime attack attack surface I don't think it should be in Astraeus anybody that you know if you're looking for 2d bugs the first thing you want to do is look at system calls because that's the thing where user talks the kernel and the first thing that that you can see or can observe is that among the three big bsds there's a difference in the amount of system calls that are implemented right FreeBSD has well over 500 net BSD years almost 500 and open BSD has you know about 300 and change so right then and there there's a clear difference between attack surface between three of them right but but even sort of regardless of attack service my assumption was given that these are so obvious and so well tested there are less likely to contain security bugs and so you know I go along and I start looking and I start looking at system calls I start playing on with them and sure enough this is no breeze system call and this is a census log and you get to give you a number of bytes and basically that gets passed on eventually to malloc and I think about ancient BSD code no bezel has this if you pass an unbound length to malloc you get an instant panic and sure enough that's what happens if you do syslog and give it a really long value the thing blows up and actually have a demo of this I don't know how this is going to work on the screens let's see if I can get this to work yeah hold on yeah okay crap it's not marrying on my screen so it's kind of annoying just until to see this hold on uh okay so there's au musi and this is my little slug example and sure enough boom that caused a kernel panic okay so yeah that's the sis lock and that's the kernel panic there you go and so what's happening is basically you know uses the the syslog syscall and you pass them about value and causes a panic and Malak this is not-- musi 6.1 i haven't looked at elena once I know it's been fixed since but it's it's it's been there so via 6.0 so it's a fairly recent bug so and here's a second example this is a system call on freebsd called kld stat which basically gives you statistics about loaded kernel drivers and this thing basically creates as this stat buff from the stack fills it out and then sends it back to user land but it doesn't fill out everything in so we get uninitialized bytes in there this has been I found it a few Musial Evan but it's been in there for over just over a decade doesn't fully in the slice a structure and basically sits back to usual and and then I let me see if I can show you a demo of this too this is kld leak and then oops and then basically all these bytes are under slicer memory that's basically flashing over to screen so that's a that's basically bug number two where you get to basically leak a whole bunch of undersized cro memory so basically my previous assumption was wrong right when I said that I assumed that because there are system calls and are so well tested that it's like let's like see bugs in them well that turns out to be entirely wrong you can find bugs in system falls into B as these would in you know fairly quickly fairly easily bugs that aren't too esoteric or too too weird fairly straight straightforward bugs especially in newly added system calls but even the older stuff as you can see a 10 year old system call and FreeBSD still had a trivalent info leak so those things do occur right so now that we know there are bugs in system calls and they're fairly fine let's move on to the chippy stack this one again my assumption is it's very well tested it less like to have bugs because as you all know the bsd stack has been around for ages since the 80s at least and it's incredibly well tested and highly unlikely to find bugs in there and photos don't know every stack is the white ask things like you before a 96 near 50 feet TCP ICMP IPSec Ethernet and things like that it's been around forever and so my assumption is unlikely to have bugs in there so I start looking around and I find something you know easy if a stack which basically parses pppoe packets and then it's a little bit convoluted I guess but basically the idea is that you take it back and you start parsing it and then if it contains the exact right tag and and the right error message basically it basically does this thing where so in in the BS DS the way these fiha stack works they have this thing called em buffs and it's basically its buffer management around data that you said in and out and so there's a whole bunch of api's around it right and the idea is that if you don't use the APS correctly things go really wrong and so in this case you see this thing called M pull down and basically if M pull down fails it will free the m buff you passed it and in this case you'll see o of M bar fails there's error code we're basically there to go to done and then in done they basically free DM buff except if the thing failed DML already freed so this is even though the quote a little bit color to read it's a relatively easy bug you trigger it effect so me is the six point one but it has been there since 2004 so it's been around for over a decade it turns out when I was looking at this and I found the bug I checked all Signet B as dean it turns out and that B you guys had the same bug but they fixed it but they never released any kind of advisory for it and so clearly had ever told you guys about it and so they fixed only guys wanna fix it what I emailed him about it and so this bug was I mean funerals you know you look for this bug was relatively easy to find and so again my previous assumption was wrong bugs and G champey stack do occur with some frequency in particular newer code but also it turns out M buffs are hard very complicated and it's error-prone the api's aren't overly consistent about half of them sort of when they failed a free D I'm buff the other half when they fail to know free DM buff things like that where it turns out if you have to use em buffs it's it's not quite that easy you have to really know what you're doing and there's a number of cases that I found where that doesn't happen and all sort of circled back to M buff misuse later on when I talk about the Wi-Fi stack but anyway t3b stack turns out has bugs all right so moving on I looked at some of the drivers obviously because it turns out that there's a whole lot of drivers among the via Z's for all sorts of things really and given that UNIX has the thing where everything is a file the way you find your drivers is you go to dev and then we'll have some kind of name and then you can do file you open them and then you can do file operations on that thing right so open readwrite closed and so forth but obviously I octal is the one that you want to look at because that's generally where most of the interesting attack service will be so this is this is a dev crypto on net bsd which has this thing where they you know basically have this very very trivial integer overflow and then here you get memory corruption and it's not this is just one snippet but if you look at the actual driver it's like a octal command ba-gawk-oh commands same bug alpha command same block seven or eight times the same trivial integer overflow shows up and then you know sure enough if you trigger it so the one I showed was this uh this Sayaka but there are seven or eight others and then again let me see if I can I can demo this yeah and I think so running no I mean it'll leak remember tool in fully in a time it won't actually crash because the the memories guaranteed to be there it's just a ton this last so this is an FPS D and then crypt overflow and then boom sure the moment hit this you get memory corruption [Applause] yeah so buddy this is a classic intro flow leading the memory corruption this is definitely exploitable and then this is a similar driver once FreeBSD this is a driver called K sim which basically gives you axes it shows you kernel symbols for FreeBSD that's an optional driver you can load and they have this interesting thing where you know they do they implement opals call when they do open they basically in the driver files driver file script for specific data they basically take a copy of a pointer that's very specific to your process and then when you do end map they find that pointer and they basically use it there is an interesting issue here what expired pointers where if you if I open the device and then I pass that file scripture along to a different process and I killed original process and then that other process doesn't end map it's basically using that original pointer to the original process but the original process is dead so you have this wild pointer that's being used this is essentially an expired pointer and this is very problematic turns out because you basically yeah you're using a while Porter and I can point anywhere to read and write to and and I wanted to I can't really demonstrate to actually hit the bug I could probably optimize it I just haven't and so you'll have to do with a screenshot but this is essentially what happens if you hit the bug and I'm sure enough yeah if you look at this here uptime nine days one hour so it took nine actually hits a bug but I did hit it all right so that's it for the driver stuff and there are many many many more driver examples but I figure twos enough so compact coat is a sting in the VSDs where basically they allow the OS to sort of emulate or different OS so if you have a binary that you can file for Linux and you're running on FreeBSD and you turn on a previously loose compatibility layer and and you make sure that you have the right shirt libraries and stuff installed then your loops binary will run perfectly or more or less perfectly on your free/busy or net bsd and so they have these kind of the FDA and the FDA waivers for different operating systems and so especially net BSD has quite a few of them FreeBSD has less of them and then open beasley pretty much killed most of them but it's essentially an air freighter different OS azure older version of gos or you know so you have 64-bit and you're supporting 32-bit versions of the US all that stuff goes through the cab at layer basically that's not basically has to emulate a whole bunch of system calls and when you I found a really nice quote by tio about the combat layers where he says well you know the people that rely on the cap at layers don't care enough to maintain it and the people who work on mail systems don't care about care about layers because they don't use them and the cultures aren't aligned in the same direction and he basically says crap at leas rot very quickly and then it's absolutely true and so an example this is this is SVR for compat code on that bsd str for is ancient nobody nobody's use I think probably 20 years but it's there and they support it and this is the existing cold streams which is it's kind of like sockets but not and streams have been dead for a long time but that be easy so supports it in señora for compat and they have this thing where basically they this function and they've added sake I mean it gets called with this BND pointer and basically the content of which come is defined by you'll and and what they do is they basically you know grab a value out of it uses an offset and then basically do reference a thing and read data out of it so this is basically arbitrary you know read anything from memory out of bound read that obviously can cause a kernel crash or potentially in Foley but the really interesting thing about this code is if you scroll the way up and you look at the comments it says yes this is gross so they know their code sucks this code has been around to 1996 so that's been there for 20 years I mean there may be people in the room that are younger than this bug but yeah this is this is really crappy code this what Tia meant when he said a combat coat rots really quickly they were ordered 21 years ago they never looked back it's there probably hasn't been used since 1996 - and then so when I told him about it and they fixed it the log message basically said well we fixed a multitude of holes in SPR for stream code and then I said we should have never enabled it's by default and then it's a minefield and so when things I'd got them to do is when they fix this thing they also changed their colonel come fake and they turned to off by default which you know right so next year look that our trap handlers and I thinking about trap handles is that so trap panel is basically visiting where they handle any kind of sort of exception or fault which can be division by zero or a system collar break point or involvement access or a very very long list of things and this is essentially mostly this is sort of heart we're sort of coming to us and saying hey this kind of fall happened and you have to deal with it so I'm can be triggered by your lands some can only be triggered by heart we're still only be triggered by colonel the code to handle this and deal with justice isn't usually incredibly incredibly nasty it's it's terrifying to just read this code I can't imagine writing it and it's very architecture specific like you know the Intel ones were different from yarn which is very different from MIPS and so on and even amongst you know the Intel earlier to later there's many changes in between and so I didn't really feel like or in his code because I like my sanity and so I was like okay how about I just fuzz it well and then it comes down to okay well how do you fuzz exceptions and I'm like well I don't really know how to those specific exceptions but what if I just execute random structions surely those things will hit some kind of exception and when I say man restructures I mean like super random like I don't even know what these structures are basically what I do is I read from def few random and then I and that page I make it suitable for a couple process basically set of phones important to it let that process do its thing it will die usually after section two or three to thing is dead but you just keep doing that in a loop in a loop and loop and it generates all sorts of weird traps that the kernel now has to handle and sure enough if you do this on that freebsd you you hit a couple of bugs there's a xenograft even though i wasn't using zen there's some sense sigma bug there's all sorts of weird things that happen so this is one of them this is not a one of them and I could demo it but I don't know the thing is because it's so random you never know when when it hits but essentially this is all the code it takes I mean I have more advanced versions of this but this is really all it takes to cause it so if you just write something like this and you run it on the VSDs and you know it will hit kernel bugs anyway that's it for Travis and happy with that because I don't want to talk about draft handlers so yeah the next thing I was looking at our file systems and so the attack Siddhant fastens obviously did the easy part of the attack surface is sort of the oh oh you know you mount the USB stick and whatever and then it has to par as a filesystem and that's true I believe in my view that's attack service but it turns out in recent years there's significantly new attack servers to file systems and this comes in the form of fuse I know if you guys know refuses it's essentially usually on file systems and so what that means is all of a sudden all these UNIX VMs layers that have been around for many many years that previous years only took data from trusted drivers in kernel where the data structures were you know more or less trusted like the data in there you figured was a more or less accurate because it was given to you by kernel all the sudden all that data is handed to the VMS layer by a user line process and the assumption that the data in there is trusted is no longer valid and so if you start looking at the future invitations for the bsds the first thing I thought was like oh well as fuse it's BSD sure it's what information and they all do the same thing well turns out that isn't true turns out all three because these wrote they're entirely different fuse debilitation x' there is no co-chair at read all three of them draw entirely different my understanding is my view of from looking two codes that the net P is e1 is the most complete in terms of features the freebies e1 seems to be the one where the arguments are the most controlled and constraint in terms of the amount of validation being done and then the OpenBSD ones basically has the most minimal amount of dates or features implemented compared to the other two if you look a fuse it actually supports dioctyl but all three locations of these do not imply octal but they implement pretty much any other facet of operation read write you know read their get attributes that refuse that type of stuff and so for example this is a this is the OpenBSD get get CWD system call it gets call my sister call and basically they'll do this thing where they go through the via vessel a nurse a get attribute and then they fill out this fee ATRA structure and they assume everything into the app structure is more or less saying because it came from a kernel driver but given that you know I use fuse all the data in VA is no longer trusted it should be considered tainted and you should validate it and of course they don't so they take this massive length value from it and if acid the malloc and there's a as I mentioned before don't museum a lock when it sees a very large value called kernel panic so this right here was a bug and then if you look little refer and get Sita CWD they'll do another call to the VMS layer just uh bob ried dear and basically they get this i/o this view i was structure back and previously when you when you that kind of read their again it was it was file some data but it was headed back from I think Colonel fasten driver so the structure was more or less trusted and in this case it's really just a buffer and in a link and so the content because it now comes from Uslan process is no longer trusted and so if you look at what it starts parsing the name length of an entry in a directory it sort of just assumes that it's more or less balanced and then it uses that for a man move and that can cause another boundary so basically the problem with fuse and I would imagine you see this in Linux too but I haven't looked yet it's that your fee about yeah if you if you do if you've been talking like fuse you have to modify your VFS layer because P of s you're free of s layer has all these assumptions where it assumes the data it gets from your filesystem is valid and it's no longer true so that was a sample one note which is the but he ought to be as he says verrat similar bugs this does this particular bug the bug was there since 2006 but III think fuse is not quite at all and so this has only recently been a real issue and then obviously if you look at the actual fascism handler when you give it a file a blob at a time that you mount so this is the x2 parser for FreeBSD and this is a very very brief look to this I did not do an exhaustive search but I was like you know grab for this thing called be read and like oh well you're looking for some kind of string and the string isn't there you just panic well that's pretty bad basically I think I could just give you a melissa's x2 petition and and this thing will cause a panic so obviously the fascism parsers in the BSD s are not what they should be I suspect if you throw a simple fast and fuzzer against these things it'll blow up in all sorts of bizarre and weird ways I literally just spent five minutes on this nose like this is very very broken because clearly you're assuming valid file systems again this this bugs been there for I don't know seven or eight years or so but I suspect when you start hammering of the fast ins you'll find more bugs right so yeah networking beyond tcp/ip stack and I sort of mentioned BT Wi-Fi in order but I really only have Wi-Fi in my slides the other two I kind of threw out because I didn't get to it but when you look at the Wi-Fi attack surface there are sort of really two things you have to look at right one is the stack itself and then what are the actual Wi-Fi drivers right and ideally you won't have this thing where you have only one stack and then all the drivers were calling to it which is what to be as these did and Linux has this too but in earlier days of Wi-Fi sort of every driver came with so full stack but luckily we should have passed that and there's only there's one stack for it for all the drivers so yeah the stack is sort of it looks like the tcp/ip stack I mean it's obviously farce is different photo calls but it's all em buffs it's in and out it's stuff that gets passed but kind of back and forth the main input function is I Triple E 802 11 underscore input and once you start reading that you sort of find the entire you know 802 11 parsing stack and that function is called from all the Wi-Fi drivers and so sure enough we start reading that eventually you find this called I Tripoli 802 11 underscore eat ball underscore key underscore input and it's basically these would be fall keys and it turns out if you look at the structure for this thing it actually has two different lengths one is length and one is payload length and it's this this player function does validate length and payload lengths but then it has two distinct called an M buff pull up that is that is to say it has to actually pull up enough continues buffers to make sure there's that much length and it does this for the payload length but not for the actual the other link field and so turns out that the actual length field is used to actually read beyond the thing to read the buffer and so because it's the pull up isn't done for length but for the payload link instead if your length is bigger than your payload length then you can end up having outer bound reads and this can be triggered by remote Wi-Fi frames so again this comes back to M buff miss hit miss handling as I said previously in TCP stack part and buffs are hard this wasn't obvious 6.1 when I was honoring it there turns out the bug has been around for about nine years the drivers say the the Wi-Fi drivers this is the interesting part the Wi-Fi drivers are either so whatever there's in because these are either PCI or USB right specie is you know when you put the card in and basically has DMA access but USBs different USBs packet based right and so it comes down to question where it's like do you trust your Wi-Fi radio right what if somebody compromised the Wi-Fi radio and then from there on tries to own your OS and so for the PCI one I was like okay well if your PCI okay it's kind of game over because they can do DMA and that's I mean that's not a lot more true when you use things I do it I only mmmm you but let's just assume that if it's PCI and you're handling wrong okay fine but when it comes down to the USB drivers 3 to 2 and 11 duty protocol is essentially packet based and so your USB host really should be able to you know parse the packets correctly and not blow up if you get bizarre USB packets from your Wi-Fi radio turns out that they're currently not really doing that and this leads to very very trivial eep smashes and I have I wanted to have one example but I really gave you five examples it's like the same thing so this is one driver where they go like okay there's a length field that can be up to a page long and then okay we'll go do will create an EM buff cluster which which can be 2,000 bytes long and then we'll just do a mem copy with that length into that cluster we got ever fell didn't if that length is bigger than a cluster and obviously that means you get trivial memory corruption and then this is a different Wi-Fi driver and they say okay let's grab this link field out of the out of the USB packet and then we'll basically use that and copy into and and buff that can be up to 2 mm bytes and again very similar heap corruption bug and then this is another driver and it's a very similar thing with a mem copy with a linked and then this is another driver very similar thing where do they take a link they do an mmm copy and it causes a an EM buff he production bug and and there's more of these but those are the ones that were very trivial so this is basically wide open attack surface across all be is these across various different I mean there's an atmel there's a real tack there's a whole bunch of these Wi-Fi drivers DDI these deep this code is very trusting of the USB packet it just assumes everything is unright I think when when these are implemented nobody thought about the attack surface on this one and then so that's more or less it for the pretty attack surface I had there's a few more things I looked at and I kind of want to talk about like I didn't know how much detail about it but so two things that I showed up that that that I sort of had that that that were seemed like they were many bugs in there so one is there appear to be a certain amount of easy detectable newly Refs in the vsts I spent some time looking at it in FreeBSD I did a very quick wrap on the on that BSE blacks I suspect there's a similar amount on that bsd or basically so the way you call malloc on the VSDs basically is you pass it a flag and the flag is either basically says no weight or so I think the default behaviors that it could default behaviors that the Malik always weights and when it succeeds when it returns it always succeeds it never fails and but there's a way you could pass the flag which is no weight or can't fail and what that means is if it can't fulfill the requests would in a certain amount of time it returns and fails to request right and so what you can do is you basically grep for these flags and you see any time where there's a Malik done and return values and checked and you get an older F and the reason these things show up is because by default the the pattern is Malik and not chicken curry value and you would think okay well that's an easy pattern and no if you're right in chrome code you would never make that kind of mistake except because the the general way Malik has caused where it could never fail and so there's almost never amount for varcek except you have to do just when you're doing a new way to can't fail and it turns out there is thank you turns out there are quite a few cases where when you do Malik no waiter can fail people don't check return value and that's so I think these are pretty great people in second else's wise and I don't know why he will haven't done this yet but there's quite a few I think I brief so when I did grab him he I think I at like 15 or 20 I can't remember how long list was on the FreeBSD I don't think that was interesting sees that the the VSDs have you know the direct rendering stuff in kernel right direct branching manager correct managing drag rendering infrastructure for those who don't know these are basically the graphics drivers that are in kernel and so when X 11 runs it sort of talks to these graphics drivers in kernel and that the whole thing around it is called DRM or DRI and so this sort of because this sort of came from the open desktop people and it was initially developed separate and then a couple years ago it was moved into the Linux kernel and then what to be as easy as they kind of forked it but it's essentially more or less the same codebase and so but the busy guys kind of have to have it otherwise ex doesn't really do much anymore these days and so it's interesting when you see the struggles they have with this and if you look at the the main OpenBSD developer who's responsible for maintaining the RM dri in oatmeal's d he said well he goes all this loose codes that we were reporting it's not going to be reviewed by any real means d curl Avella pers because they refuse to read any code that's not conform to the bsd k NF standard and so you have all of these rigorous code review standard practices on open BSD accept old as the RM DRI stuff they will not touch it and so all the banners you see there in Linux kernel all that stuff is pretty much in busy and they're busy at FreeBSD as well ok so that was sort of it for running through the sort of the types of bugs I was looking forward to things I found I mean I could have spent I could have done an hour to three hours running through a bunch of bugs but you know it get pretty boring so so what was my result after I was done about three years months I about 115 bugs in total 30 bucks in FreeBSD about 25 bucks in open BSD and the lion's share was in that BSD so about 60 or so and that BSD it was a very wide spectrum of bugs pretty much everything under the Sun that you can expect right it a straight-up heap stack smashes race conditions expired pointers double fries integer issues on the floors overflow sinus bugs logic bugs I had a typo somewhere where the wrong variable was used but turns out that one existed - and then he had these weird things where the wrong thing was happening on the wrong structure because there was a typo division by zero I had some logic bucks in there I mean pretty much anything you can imagine is in there it turns out kernel code is not written by gods and they to make mistakes and they make plenty mistakes actually so I found basically bugs amongst all three PhDs amongst all of the attackers as I mentioned and within that entire spectrum but it's interesting once I sort of got a grasp for all three of them and what was where and how it was done it's interesting too that I think I can make some observations about the co quality and just looking at the bugs like just the numbers of the bugs right you can you can you can see the same thing right and so there's sort of a I think OB is the when it comes to code quality in their kernel is the the clear winner right and it comes the quota I had from the original where it says oh it's code quality that's part of it I think it's it became obvious to me that it's it's code quality but also a tax reduction it's a combination of those two that seems to be a winning formula and obz there is enormous a tax reduction if you compare it with the other bsd is there are many things they do not have they don't have loadable kernel modules they have relatively few devices they have virtually no compact code they removed their lose combat code a couple years ago they removed her entire Bluetooth stack because they thought the code sucked and they just deleted it so no bluetooth support and bsd they have significantly less system calls they have more than 200 system calls less than freebsd and they got support for a whole bunch of old architectures right and and that in combination with code quality i think is a winning formula and the quality's interesting because it really does show if you look at a code right the trivial low-hanging fruit stuff is almost entirely gone OpenBSD integer overflows sinus bugs are virtually gone you know bsd there's like the Wi-Fi driver stuff is something where I still found it because it was attack services they never thought about but anything that they know was attack surface it's highly likely you find integer overflows there or silence bugs simply because they know about them they they inform their developers about what that looks like and they have this every every commit gets quote reviewed by at least one or two people and these people know exactly what this patterns look like and it turns out when you have a process like that integer overflows simply don't occur in attack surface the other thing is that they had Furion for leaks than the other B Aziz in terms of the clear loser yeah that was open BSD I think you can see that in these bugs I mean they have 60 they are to come is less so yeah it was it was a net bsd tons of legacy code tons of compat code yeah it turns out when you have legacy attack code when you have quota it was written 96 and you haven't looked at it since yeah you're gonna have bugs they have these SEO protocols near which I don't know if anybody actually uses those these days it's really ancient code from the 80s which is written by IBM and then it was important that BSD and it's been there ever since and nobody knows what it does but it's there and a whole bunch of this really really old code the other thing is that their code seems to be less consistent which security call security code quality compared to the other PS DS unlike open BSD they have tons and tons and tons of inertia issues and sign those bugs they're required I mean just I don't want to Africa show that you know eight in your drove votes right there so so there is definitely that sort of code quality difference and I don't mean that as a disc I mean I understand that I mean building maintaining improving OS is really really hard and if you think it's easy you know try it so I understand that it's very hard to do in OS but there is a clear difference between if you if you look at one or the other you can see there's that there's a difference in code quality when you look at the attack surface and then freebsd is somewhere in between really it's hard to place it but it it's it's not it's not the code quality isn't as bad as net PSD but it's not as good as open BSD so okay so obviously when I found these bugs I I talked to the teams and I send them emails and I said hey here's a list of bugs I have you guys should probably go and find this fix this so email do you see guys and you know teal gets back to me I know about a week or so later and the first thing he does it says though I'm sorry it took me a week to get back to you I was on vacation and he says oh these bugs look good this is definitely problematic we should go and fix these things and then he says email him back and says the next two or three days you should see fixes sort of coming in our CVS and sure enough 25 bucks I report it lest of the week and they fix it all right yeah I think that's good and then a couple weeks later the obz guys basically made individual patches and fire trees and said okay if you have late if you haven't please don't be a Z and you want to fix these bugs here's the patches here's how to do it so I think that was great I think that's perfect I think that's the way it's supposed to be right sort of do you know within a week response a few days later things are fixes they are fixed in place and then like a week or two after that there's all the patches are probably available there's advisories this is exactly the way a security process should work it'll be you guys have done everything rights as far as I'm concerned so free/busy how'd that go well it sort of started off simile where I got a response within a couple days maybe a week and I got a bug back from these guys and said okay well we've seen all your bugs and we filed them in our internal bug database and this was July 14th and sure enough this is from the email and I've blacked out what hasn't been fixed yet or what I'm not quite sure of there's three narrative been fixed right so since that email not all that much has happened right it's it's five months later two advisers have been released and so obviously those two bucks been fixed and then there's a third one this arc e ed cap I saw I saw CVS commit for that so I know that was fixed all the others I it's up in the air I don't think they're fixed I I'd quickly check CVS right before I came here they look like they're still in limbo somewhere so you know this is this is where we're at with FreeBSD okay net bsd first of all when i emailed from these bugs a list of 60 bugs this got fixed overnight seriously seriously [Applause] this is obviously unbelievably impressive right I don't know how you can do that but clearly there were two developers that were like let's fix this let's fix it right now so that was that was very impressive and then as I managed to also turned off the entirety or STR for compat subsystem and if you look at the commit message said oh no the email they send me said basically and it is smeared in the commit message I said we've disabled to SBR for by default something that should have been done a long time ago so that is incredibly impressive that was late July 2017 five months ago let me let me tell you what's happened since absolutely nothing bugs are fixed in CVS there are no patches for current there are no advisories what this effectively means is if you're running a net busy today all 60 bugs are there and you can go to cv to receive yes and see exactly what the bug is so I wish they had followed up with some patches and visor ease so that kind of it started off really really really well and then they kind of dropped the ball okay so coming back to my sort of the earlier where I began my presentation this already okay are things on equal footing well I think bugs are still very easy to find in the busy curls probably about as easy as on the loose kernel even OpenBSD wasn't I mean there were certain things I didn't see no BSD but it wasn't like it was hard to find bugs in there there's certainly very good level of quality between the two oh but you know three of them it depends on the age and who wrote it and under what circumstances the most consistent set of quality I found by far was open BSD again this comes back to their rigorous review process right every every CVS chicken gets code reviewed I think that's a process that simply just works I would exception to DRI the RM stuff the OBC guys have the same shitty code everybody else's and a fairly OPC developers refused to touch it another thing I have or sort of that I think should happen so because I found it I had a couple of bugs where I was like okay this bugs don't be a see but it was fixed in that bsd or and the other thing around where i'm like oh this is an FPS de but you're busy you gotta fix it fifteen years ago right so those things from time to time did happen with some frequency and so ideally I think the maintainer zone to be as these should talk more amongst each other I understand that's obviously easier said than done because in the last 50 or 20 years they have diverged and there's different philosophies and ideas on where the bees are supposed to go and obviously you know there's big egos involved as well and so getting these guys to talk isn't always very easy and it doesn't always make sense but by a large I think there's enough commonality still between all the PhDs that it wouldn't make sense that when it comes to things that are attacks roofers and fixes it would probably make sense of these guys talked more with each other the other thing is that obviously when it comes to things you know if you look at code base alone that tells you something about your attack surface right OpenBSD sternal is about 2.8 realize of code net bsd is about 7.3 million and FP aziz and this is 10 this was 11 points here i don't know what the latest one is this is the free/busy was about 9 million right so that alone right ter tells you know bees these gonna have less bugs because they have less code right it's that easy so obviously this plays a part right you can't have a bug in code that you don't have right and the other thing is obviously sort of accidental versus planned where if I haven't gotten if I haven't been something yet then you can't have it yet right but the other thing is obviously is that the plan where's choices to make the delete code on purpose and this is something you saw in obiously where you know they chose to delete their Bluetooth stack they show us to delete their Linux compat layer and obviously it's a double it's or try do you lose functionality but you gain security and trying to find that balance but obviously you know cutting code you know that generally gives you know it cuts tax serfs and generally means you have less bugs right so yeah more conclusions basically coming back to what I originally one had which was the mini eyeball sting yeah I think that's a factor I think it really does matter right I think if you have more people looking at something war bugs are going to be found and so I think one of the big reasons why the numbers are off when you compare those tables initially I I think a large part of it is you know more people are looking at a loose girl and so they're gonna find more bugs code quality can't explain everything and right I mean you say what you want about people right loose kernel but you know there are just orders of magnitude people more people looking at that code they're just gonna find more bugs and it shows in numbers and yeah that's pretty much it [Applause] that was frightfully awesome okay I'm quite convinced you people want to ask a bunch of questions we got about nine minutes let's start with you over there one sentence one question mark thank you for your are you interested in exploiting so you had a crush who crushes and how about making a proof of concept local privilege escalation or RC remote code execution yeah given that my plan was to report all the bugs I saw really no point in writing football next points because it's wasted effort right the only to me the only way I mean if I'm gonna write an exploit I don't like I don't I don't wanna write next one for a bug I know that's gonna get killed right so I didn't see I mean some of these bugs it would have taken me weeks or months to sit down write code for and so I didn't I don't think it would have been very useful for me to write full-blown exploits I know there's a shock and awe factor to exploits and at times they can be useful given my understanding or my assumption that the people on the other end I was talking to people like Theo that are very knowledgeable in this area I didn't feel there was any need to write to write an export and so I didn't does that answer your question showing exploits usually helps to convince people in self-protection technique technologists like we have our security and yes you know I mean mitigation obviously are a good thing and and you know we should we should keep innovating as pouring new mitigations it's just that I don't think this would have contributed much to it okay thank you there's a question from the internet please this isn't to the dark side yes and thank you the dark side how would you suggest to improve cooperating between the different bsds that's a good question I mean I don't know I understand cuz I know I know tio said a few things about well you know it's not easy to talk to other people because so and so and so yeah I don't know how to you know and their today get these guys Andru and getting to talk or something obviously there's uh there are some difference between the two and they're not they there certain things obviously they that you know where there's differences they don't necessarily have to talk to each other but yeah I can't I don't really know of any specific way to try to massage them to get talked into each other oh I oh I can say is find right containers for the right subsystems and if some guys fight fixes a bug it's something where they go this is a tack surface you know reach out to the guy I'm you know the others you know the author of B is these in Vela and just cinema dropped one email I'll be like hey guys maybe you should look at this to be on that I have no no good answer really okay thank you is there another question from the internet not yet not yet how about one last question we got five more minutes anybody at the microphone woohoo I'm not really seeing anything out there okay over there on the left excuse me my question was about methodology did you use any automated tool or did you do everything by hand okay yeah well I didn't I 99% of this thing was like straight up reading code just you know I opened up in an IDE and I was reading it there were a couple times when I used some grep to look for some patterns but beyond that no it was just me reading code that was pretty much there were no other tools involved Thanks [Applause] I think we'll take one more him when more sure yeah one more here didn't you want to bow is your big chance could you describe your motivation of spending three months on that work yeah III know quarter view isn't for everyone I I think I once heard it described being you know more boring and watching paint dry I disagree with that I actually I enjoy writing reading code I I think it's fun I think there's a number of things you can learn from reading code and and thinking it's the sort of interesting sort of kick from writing like if I find a bug and like sometimes used to be the code and you're like oh like oh they screwed yourself like I notice this in the end they didn't so there there's a little bit of that but generally yeah I enjoyed it and I know some people don't but for me it was it wasn't the that March it wasn't a stretch I was I had no problem and then when I say 334 months I mean on and off because obviously I've worked in things so it's like evenings and weekends but it wasn't it wasn't hard it was difficult I was just like okay let's open up my laptop and I'll spend next three or four hours looking at you know the system call or something I didn't think it was hard I understand this now for everyone but I I I tend to enjoy code review thanks again for your work okay thank you very much am i overseeing a question somewhere no now's your chance the Internet if there's one final question from the internet why the PhDs may not be cooperating is there at least a common mission statement or any high-level thing they could at at least agree on this is a non security PS DS question I have no idea I think there are people here they're probably a much better place than I have to answer these kind of BSD questions yeah I've no idea okay if that's it thank you for letting on the final hand [Applause] [Music] [Music]

Channel: Expatriate PW

Views: 24,809

Rating: 4.9004526 out of 5

Keywords: Bitcoin, security, BSD, LINUX, solaris, exploits, holes, opsec, open source, libre, stallman, hacker con

Id: AvSPqo3_3vM

Length: 58min 58sec (3538 seconds)

Published: Tue Feb 27 2018