Kernel Recipes 2019 - CVEs are dead, long live the CVE!

hi I'm Greg this is not a talking about EPF but the obligatory thing next year I want to do USB I need to be P I know be my talking mixer seriously oh cool thing if you don't know about you said they said talk about EPF BPF allows the kernel to run user space programs you can build a user space program it's a kernel module and have the kernel loaded and run it as a user space program you have a microkernel nobody noticed thanks for Lexie it's actually really cool I more people's to take advantage of that okay um this is gonna be like a forty minute rant with an unsatisfying conclusion you've been warned um I don't normally say this but I am going to say this today this is just me I've been working with CBE since they came out in 1999 this is not the Linux Foundation the sponsor the Linux Foundation this is just me hopefully I convince them and everybody else that I'm right let's see what happens so talk about CBE CBE this comes from the website there's a link to them common vulnerability and exposures people toss this name around a lot what it really means this is what they say and this is a real quote I do not understand what that quote means the way to interrupt your meal and better security coverage all CBE is is a tag it's just an identifier it's a string that everybody can throw in their security Boulton and feel happy that's it they just feel good about it they track it they feel happy it's much better than what we used to have which was this trick something on the second paragraph and this random webpage over here that we hope will not go away and that's the problem we had we had all these problems of people we're fixing bugs and there's impossible to track them across products across devices in different world of things because we have the problem in 1999 of different libraries embedded places were buggy how do we fix them we fix them how did people know that there's a fix that people need to have but really it was the CGI plugin exploits that caused all of this there's a whole bunch of remote execution vulnerabilities and CB cgi plugins her Apache and Apache was bundled inside other products and nobody realized was because it was in there because it didn't have to be told license of Apache and all these products were vulnerable so you had all these problems that were found in these plugins nobody realized how to get that information out to everybody else CB came along the US government set it up along with some a whole bunch of other people and said here's a way to do a tag and let's track these things another good reason for CBE's zealand z libs everywhere it's all over the world it's in every product everybody uses it and it's been buggy for 15 years that's evil it's a great product but anyway so tracking when a fix happens in Z lib that actually gets into the product that you use and rely on a CB is good because the CBE just looks like this it's real simple it's a CB a year number you see a four digit number now it's a five digit number an amazing number of things broke when they move from four to five life goes on they move to 5 mm length 18 anyway it's a unique number and it sends some information behind it and that's it you can look it up online see what it kind of says I'll talk a little bit more about that later but it's just a number but it's a unique number so that means somebody has to give you this unique number somebody has you can't just make a number up you have to get it from somebody to make sure it isn't the same as somebody else's and now there's about a hundred and ten different organizations you can go to and ask for a number for and it's distributed around the world all the major countries have them lots of companies have them if you want to Linux CVE the kernel community does not give them out we actually had to a say in the kernel security document we do not care about them we do not give them out you're on your own you can go and ask my trip for them there's web forms to get them from Red Hat is one get them canonical can all the major linux dishes can do this it's just a number but it's a unique number so CDE is just a number and 10 a description it turns out you need something else behind that and that became the MVD and those people don't realize that this is happening there's a national vulnerability database it has all the CVS and it analyzes the CBS it tries to give it a score tries to say if it's good bad what is it is it important is it not important and there's tons of arguments about how they have scores so much so there's two different ways they give scores and there's a table that says version 1 or version 5 or something like that and how they give these scores and they can differ vastly depending on what you feel like what's a searchable database it is updated with updates the CVS happened over time but it's updated slowly slow to update for overall I got some stocks on that later but living not in the United States anymore the word national is interesting what does that mean this national there means the u.s. like the world series all the worlds the US right there's other countries finally came up with their own idea this because they didn't like just us getting in on this so trying to make one there's a China national vulnerability database and here are some stats the good report came out a couple years ago saying that the China database actually picks them up faster finds more things and responds faster than the US one but the bad thing about the China database is it never gets updated so an update happens to the CVE the China database never gets updated to see that and I'll talk more about updates later it's a big problem CDs so you have two big major national databases out there publicly searchable works great you're gonna look them up looks nice everybody's happy right so this I cut and pasted from a white paper written by some people I don't know if the white papers public yet it's interesting my paper about how to do security and stuff but they talk about the problems with CBS and there's a whole bunch of different interesting things I'm only gonna go over the bottom ones let's work backwards the top ones are interesting they're rejected a long time to get them for descriptions there's really if you look at the database sounds like spectra there is a problem the scoring is hilarious I'm less working these battlebot so run by the government US government this is a big problem it's a big problem for two reasons one is people don't trust governments right good thing we stressed about this government some other governments don't trust US government all my friends here working for anzi would you report something to the US government it's kind of vulnerability probably not legally allowed to I don't know if you are not but I would wonder about that I'm it's funded the funding is really really odd it's funded by the now by DHS and size' but it's really really underfunded the budget fluctuates every six months actually there are some US senators came out and did a report saying this is unacceptable you guys need to fix it wrote some big letters and I couldn't track down whatever happened to that so I think it's still in a horrible funding limbo it's very very underfunded so the ability for it to work is really not good also when you tell and you asked for a CBE usually the problem is in public yet just want a number so you can put it in your security Bolton when you publish it so you don't you want to have some trust that the people running this thing aren't going to leak right during spectrum meltdown there's a US Senate subcommittee hearing on this luckily I didn't have to go I had to testify on the phone but I'd have to go the person who did go one question really came up and was somebody from Nitra who works for CB he was asked why didn't the US government agencies know about this ahead of time you guys had a CV assigned why weren't we told about this and my truck CV said no we're gonna tell anybody and the senators were not happy with that that's not a good thing I don't know what happened I don't think I trust CVE I trust those guys are not gonna leak but you have national government agencies saying why aren't we told about this stuff ahead of time that should make people worried anyway that's it for the government talking volume or governments okay more trouble um here's the big one it's too complex Thomas spectra I counted ten patches I think was forty right spectre-1 for the beginning it was lost through the noise we had so much crap flowing around them I know you talked about that um people like tying a CBE to a single patch I'll show you an example of where a single patch got three CDs it's really difficult to say if you look at the CVE entry for Spectre meltdown it doesn't say what patch is actually fixed it not good we're still fixing spectra Gustavo's fixed Spectre for the past three years it's still going on there's another patch just one in two days ago I think for um none of these things are going so if you think oh the CVE reflects this patch great I'm secure I'm done let's go it's not true at all in fact it's a lie if you just rely on that you will be insecure and also the NVD part of that doesn't actually point to all the fixes all the time sometimes it does spectrum out that it doesn't if you look at anything to plex it just gives up and just says there was a problem these guys said they fixed it have fun not good another thing about CBE's work also i mean it's closed and open source so with the open source ID you can point to source code and sometimes they do but sometimes they don't it's a it's a grab-bag mix it's a little weird thing that's a big problem the Plex things can't be handled with a single ID like that this is what I run into all the time um a number of other people in this room initial time we have people who think they're security researchers that want to see the e for the resident and that's cool I mean I got my name on it and things like that get the patter resume say I found these security problems but the problem is we're running into more and more and more is really really stupid things you're getting asked for a CBE and most all these things are happening are not actually problems so let's look at some interesting things here's a fun number this happened this year in my code so I get to blame me for all this stuff this is what CBE 1 2 3 7 9 says that there's a memory leak a case from out of memory we're gonna leak some more memory think about that there's a memory leak up to 5 dot 1.5 that number doesn't make any sense for this another question I'll talk about that that's what it says that's all it says and here's the patch I've dug through you dig it out again did it case actually reviewed it and I agreed yeah it kind of looks ok problem if we came out like an array this is that boot time we said just returning memory we need to free some other things ok normal things you guys are technical tell me where the security foam if we are out of memory and we're returning that we're out of memory we might leak a little bit more memory what's the security problem I can't there isn't one there this is one but here's what CVE says and here's the wonderful timeline so the date on this patch was the 23rd the 27th a CV was published public so it became public no idea where it came from it came from my turn themselves so somebody went out and applied for it it doesn't say who I'm guessing the original developer here uh MVD a day later very nicely said this is a medium security issue yeah that's actually raising pretty high Fedora Red Hat in their infinite wisdom actually does really good things they actually have a requirement that all CBE's must be in the product great that's good so they grab it and they stick it in there and then they realize maybe there's the problem so there's the problem something else I'll talk to us more a minute and then finally a month later somebody said maybe this was a good idea and then that app a month later after that says a couple weeks later a 5 watt 1.3 came out I've no idea why 5.1 that three is an issue because I don't want a three came out months previous to that and all this was happening during the 5.3 issue right all right three this is during the fight or three development series no idea of a 501 C so here's them thank you Ben actually had to do more work to dispute the fact one week after I committed this because in the email through I was case and I cases like well this really doesn't quite Unwin deciphering things somebody just revert this whole thing it's a mess Ben actually proved that it really was a mess prove that the fix was actually worse than the problem if I didn't know better I would think somebody was trying to troll us but luckily you can never trip this you can never ever if the kernel cannot allocate 128 bytes of memory you are dead I don't think I think we physically cannot fail that call so this could never happen so the fact that somebody tried to make something bad happen right it's bad so here is the patch history overlaid with the CVE history tell me something okay so original path is centered on the 21st then he sent the version to based on case saying what's going on and they said version two again which really was version three because he changed something again new developer we talked about that before patched I merged it of the 24th because yeah looks like a bug fix into my development tree tree that only is in developed shows up a little connects no public use everywhere case after I commit I said probably some ways to just remove this thing three days later CB he came out no idea I was noting attention and I don't walk through these things day later NBD does something saying it's secure or it's actually medium circle score ban wakes up sends a revert I apply the reverts nothing's ever been in any public treat no ways back toward the patch nothing's been tagged anywhere the Dora dug it up put it in there 5.17 1.7 kernel 1.7 is great but what does this 1.3 number from then they realized we then reverted it Mike thankfully dropped it disputed if somebody's type of time disputed it thank you again read that what you disputed thank you it's hard it's really hard to dispute it isn't it I mean it's a it's a difficult task you have to go your way you spent more time with this I think than me my far NetApp came out and said 5.1.1 I and then 5.3 rc1 is finally released with these patches in it so a whole bunch of stuff happening in a developer tree nothing ever hit the public tree and this is a mess this is just one example we do this every month if not every week all the time all these people CVE people off running on screwing around fedoras having to track crazy things Debian is cracking these as well Debian links don't link back to the thing so I guess my tree doesn't suck it up this happens all the time this was junk total nonsense even if this was real the score of medium was nonsense it doesn't make any sense this was not a real issue disputing these issues is bad somebody was trying to pad a resume if you look this developers continue to do this watch out there's a lot of patches the problem is distros and companies have a rule all CBE patches must be expanded to their tree Fedora did that sucked it in it was wrong Chrome OS is another tool this show they have a requirement they suck everything in so they're asking you to back for patches of times for stable patches that are been disputed and been rejected they don't have a way to tell a lot of times if something is real or not because I'm not gonna take the time and refute and dispute it it takes time to do that it's a big problem we're getting junk patches merged into trees because some external database that we don't control says that it should be not good okay this was written in the white paper to the abuse by engineers circum it all right now this is why I have a disclaimer this is what's really happening some companies for their enterprise kernels have a hard time getting patches at it oh don't take a picture this is actually - this is what is happening some companies have a requirement that if you get a patch into their enterprise kernels that has to be there for a reason engineers come up and say oh I found a bug fix needs me back Corte cuz somebody did something I can't justify it I'll get a CBE and does it and we have to proof that this is happening watch this so we taste cooked and all the numbers from 2016 to the mill 2018 100,000 CVS were signed the average is a negative six state averages we asked for CDC VE days months years after it's been fixed the average fixed state is negative a hundred days the longest one is ten years I want to know about that one somebody said oh wow I really got it back for it this - like what lens for I do not know what happens Susan does do this a time Susan is actually a lot better the Red Hat they're a little more flexible but somebody does something for ten years old there's actually one out there that took us we haven't really fixed the other thing but the numbers are all over the place the standard deviation for these numbers is over a year that means people are abusing these to try and work around management problems I don't blend the engineers for doing this I would do the same thing if I were them create CVS out of nowhere Scola this is happening today the updated numbers case says he'd do them I haven't seen them yet but it hasn't gotten better in fact I think it's gotten worse the average dictate is getting longer in the past the shows that these things really don't matter they don't tell you anything right so this is what the kernel does today I'm putting into the stable trees about 22 patches a day that's 5% that's actually pretty low 5% is low the numbers of sasha and other people are running up says we should be running 10 to 15% we should be higher he's ramping up the tools to make us there that scares people these are bug fixes that are in leanness the street they shouldn't scare people these are things I will leave someone to two times a week depending on the travels or good one one of this week maybe not the best thing is these are fully tested as a unified release and I given to you for free so I've had sadden companies when they somebody says 22 a day I can't keep up with that just tell me which ones I need oh my god oh no which ones you need take them all I can't take them off like take them off why it takes too much work I'm like I'm giving them for free take them all and take them all because of this I'm fixing a known security problem once me I know there's a security issue in there once a week I'm not gonna tell you what it is I know what they are because they look like any other bug fix I'm also fixing things that really are security fixes that nobody realizes and that's happens all the time the Linux kernel security team you fix things we move on and we keep them going a bug is the bug the bug we're not gonna take the time and effort to figure out what was a bug what was the security issue or not infamously again I'll blame me in the tty layer I wrote some code a long time ago I fixed it later couple years later three years go by Red Hat says oh look I gotta go see v4 this 3-year old bug fix you did because it turns out you can route the box I wrote the original code I wrote the bug fix I had no idea it was a security issue they got a negative date on their CV they were happy they back forward it turns out rel whatever it was was vulnerable for three years to this problem anybody that actually had updated and taken a new kernel was not vulnerable and that's the thing you have to take these updates you have to take them all because again a thousand CVS over a decade very very few CVS are actually being assigned they're getting more because more people were paying attention to it and I have people come up with these graphs and puffins to say why there's somebody see bees I'm like because people want resumes but very very few happened again kernel fixes do not equal a CD that's probably see bees don't say anything about following fixes we've had spectra meltdown fixes meltdown especially our other spectra issues fixed this week fixed next week next week after that vixen lens after that never ever modified if you only cherry pick them let me say I'll just shake it you have an insecure system and I got proof of this in a minute I'll say actually the numbers so I said this in the past you're not using you are running an insecure system since this is friendly crowd that nobody's gonna take a picture of this I will say this and I have the numbers to prove it look at what red hats doing today they know that they have to cherry-pick things that were done a year ago you're in the future a year ago for their old colonel I can't think which way it goes what they're having to grab random pieces grab a CD and back toward the hell out of it if they had only updated the whole thing to all the LTS releases they would be secure it's that simple you want to know how to break a red hat box look at the tree you want how to do any of these other ones look at this tree well know how to break your phone so I thought of it lots of phone trees I'm not gonna say what company this is I think you can figure it out by reading the slides very popular phone came out March it shipped with the four 1485 colonel at that time it only out of three million lines of code book can go wrong there they're only running 3.9 million it's really weird anyway so I compared it I did this audit in May so it was obviously gonna be a little bit behind but the phone was in a it was shipping it was in my hand I actually was using it I'm still using it I'm not gonna use it anymore because I looked and they still haven't fixed their kernel so they grabbed some they cherry pick some random patches some f2 a faster my logging cleanup the best I really pays and maybe a certain chip specific faces they swore that they were only taking all the things that they hadn't had they looked through them all they read the logs and years ago like two years ago at this conference I proved that nobody reads the logs we crashed everything with the patch the Thomas wrote they said here's a problem here's the actually how you exploit it and here's the fix nobody ever taught that these guys next well-documented C B's in the changelog itself it said CBE something we rarely have them they were documented in the networking stack you can remotely crash the phone on a Wi-Fi network I hid horrible hid problems we've had over the years with thanks to fuzzing we're finally fixing them all and here's all the things they really really missed the best one is they're working right now they don't realize it I'm gonna wait to see what happens there all those fixes that they missed all those things that actually affect them the best thing worrying about people like there's so many changes I don't care about them all I just want the ones affecting me the Google pixel fountain has the script when they do a merge from an LTS tree they actually go through and in the change log they show you in the merge change point they show you what patches actually apply based on their build configuration they test it out and there's only usually a handful because of those 22 patches that happen today only a handful actually are in the stuff you actually run there they can see with it and they Auto doze they look at them all they say is their column or not that's the best way to do it descriptive three it's done every week go do that these guys were not doing that at all and I'll talk about Google again the Google security team last year did this work for me or other people they gave it to me they tracked every single thing that they told the developers that they they have a big tool that they called vomit that searches the web and grabs anything related a security issue and swabs it in there it says go fix this every single thing that they found and told the security team to fix or that was reported to them other places was already fixed in the LTS curtain every single thing the only thing that wasn't fixed out of those what 17 patches was due to out a tree code or a feature that was backward incorrectly everything was fixed before they knew it because of this Google is now requiring Android devices if they're certified to take LTS cards at a some cycle I'll take six months three months to be nice I will pull out Sony and I'll call out essential those guys update the kernels every two months it's lace LTS kernel they've been doing it for over a year maybe two years now they do a really really good job those two phones I could really say are very good pixel is getting better pixel ones are doing I think they're lagging about four or five months but the least they're they're don't philemon new pixel comes out that will do something else as well so this is a real problem these are the devices that Linux runs on 2.5 billion instances of Linux are Android phones the cloud supercomputers is a drop in the bucket it's around the air for these devices this is where Linux is the most and this is where the security matters the most they grab the LTS fixes they grab all those things it will actually be a secure device that's why Google's doing this and I really really so sad talk so see bees are broken they don't work for us right how do we fix them we can do certain things we can ignore them I like this we're doing this today well not really because some people do care about them some people do this I say we should just ignore great Thomas and I and a few other people sat down with some people from CB my trip and came up with an idea and this was floated by the other people and they said we know we know it's a problem we know this doesn't work burn this food around ask for those 22 ask for CBE for every single patch you guys do colonel I got approval I got somebody to say that they would fund it if I wanted to do it it would be a fun it would be a horrible interns job for six months they go crazy they hate me it's a horrible time has to fill out these things that actually ask your CV no maybe but we know it's broken let's not just trying to views the system might be fun to watch right that scramble but I'm not gonna do that so we're engineers we like building things make something new right you can do this we know how to solve this problem we have since 1999 we got a prior art let's figure this out what do we need to do what are the requirements first off that's me a unique identifier works great distributed we've proven that asking for a number from another person doesn't really work they've gotten better there's a hundred different people places you can ask now it needs to be distributed somehow right we able to revise it over time we should search it it's Republic all good things these things are things rise over time is important we need to have it so let's look at a patch do we do this is the patch that Ben did the reboot one so it says we revert this commit and it fixes that commit ID we say you Burt we're doing these things these commit IDs are up there from the ID 1 5 B 3 this reverse commit 8th floor you see and Mitro and CBE actually references these they say hey look and they're pointing to my testing tree which is hilarious testing doesn't can be rebased nobody ever rely on testing tree I'm the point to these good ideas luckily they stay the same these are the good idea so Mitra's already using good IDs so here's another one this is great one patch 3c bees if you look at the patch it's like fix fix fix in the kernel development community we require patches to be broken up into unique pieces one logical piece perfect we are what curse we are obsessed about this we have to make you want to break it down to the tiniest possible piece this was acceptable tool external community as one logical change to the kernel yet three different CDs I'm you get a lot by CV people anyway this is a commit it went into 419 I think or 5/3 um commit ID 5c or 7c whatever and when I backed toward this to the 419 Colonel I always put in the community commit 7 CA so you can track these things we track these things around yes slide backward this way back they all plug so anyway that's a good reason that could be about reason for this the CV I doubt my true actually asked for that I think this person won and I know this person went and asked for three individual ones they sent it to us as three actually individual bug reports is the main line that I send the acumen volatile come to you at the end so I do with John send a lot of statistics on Colonel stats everybody spells their name wrong everybody types their email address wrong that's the email address type Oh Oh make it a macro anyway I didn't apply this thing I'm sure there's a patch so but again communities we track them around the world we even up scripts there's a link to my script I say what after I back for the surface and what release given the and it says way Greg backward it's a 4/4 419 all those different releases and also showed up in the 5.3 release we have these tools we use these tools every single day we track this stuff here's another fun patch come yawn yawn some of those people if you ever see a patch from peeling the kernel look out um floppy driver turns out we were copying Dave the wrong way and the floppy driver nobody noticed for 20 years we're now actually no not 20 years actually it was only a couple years anyway he fix this when he says fixes you fix this commit so we can track that to 9 B we can see where how far back to need to track this stuff and then when we back ported to 419 it has the community of where fixes and the comment ID that and if I say where was that original 2:00 to 9:00 I said it really showed up in four it was really showed up first in 4:13 by back toward two four nine seven four four one eighty seven so I got it backwards out of one all these different ones so we are doing this today so we've been doing this for 14 years why doesn't everybody notice because nobody knows what we can do comes down to that so how do you solve the problem but nobody knows what you're doing you don't get a lawyer you get the second worst thing I know some nice for marketing people I'm actually marketing matters marketing slc bees are its marketing is showing us telling a story about what is happening in how it's being tracked around the world and how to fix it so we need a good name for this get commit ID doesn't work really well let's come up with some cool names let's get Parenthood ID okay G what's the next one when a kernel idea all right okay I the next commit ID because I don't internal is that redundancy because Linux is the kernel argue is like a new people Linux commit ID yeah okay how about girl kernel get ID it's getting better but it really is a Linux thing maybe other people do this so get kernel ID because what there's other kernels out there and you want other people to use the same thing because they use get so it isn't the kernel thing and how about this [Applause] [Music] but I do not have that kind of an ego get hash ID yeah we're getting better but what have I been calling it all this time what have we been calling it for 14 years it's really just a change ID we've been calling in this or a commit ID this is a unique identifier that is worldwide hopefully unique because there won't be any collisions it works today this is what we have let's use it let's talk about it it's a CID 12 digit hex number easiest thing the search for I wrote up I bashed together a horrible shell script and about a few minutes and said pull all the C IDs from this old kernel release boom those are all the fixes that were upstream that were already in that release those are all things that should be back ported because they all fix something they're always affixed sometimes you add a new ID I don't guess what's before nineteen before nineteen that one so I think there's just stuff Dave sent me for networking ahead of time here's another here's another Spectre again we're fixed expected things constantly I'll take them all here's the IDS put them in databases track them around the world and you want we have these you can clone them they're there let's just use these today so how do we fix them I suggest we just do two things good more them and we reran what we've been doing we need to tell people how to do this I talk to other software projects kubernetes other filed stuff and use git they're like yeah it makes total sense why don't we do that let's just unify this back port the thing you want use the good ID you can track it you can write your own scripts I mean it's a tiny little red X or use what we have today the change ID just do that [Applause] [Music] you may not you have we do not create see you for your stuff yeah well we can we can tackle with these did you track this and then next why you saying that if you're using or 1485 instead of 108 you're missing whole of fixes for networking so the question is how was the 85 release tested so we talked about why there are so many bug fixes going in the kernel it's because there's more people using it there's more people testing code paths and never happened we do have a problem of lots of bugs we're trying to fix it as fast as possible our goal is and we get this wrong occasionally not to ever regress in these stable patches everyone so what we do I think our average our average today is point zero one percent it will happen but I rely on testing so Google test the heck out of these things lots of other people test the heck out of these things lonardo has huge test frameworks working with kernel CI to make better test frameworks especially your vise I'll take a six-month delay to get to your device because you have to go through testing test it whatever you would have done for that original kernel that's you blessed and that was good enough just test it to make sure we didn't break anything again point zero one percent of the time we might fail but we're human must always have a very very good track record good track record over the past 15 years you we have the history to show that we don't regress some stable peace except on a very very time thing because the original release the spin lock was broken in the original release and nobody caught it this the spin lock was broken for years yeah and somebody tripped over it because we have a we noticed that this Steph Jenn spinning and not making progress say Godfrey stop stream it but of course you want to have it back gorgeous and it was just an event once out of in the blue moon they made eventually progress with rain drop can mean it's problem which needs to be back booted and you don't want to have that problem on your phone and I think problem when you have when you step five million phones every couple months it was actually so we run the Cisco and make sure it does we have functional test so we have a lot of tests we have more tests now than we ever have not saying we have enough we need to have more yes I would simply respond that by upgrading you are taking 0.1 percent chance of hitting a regulation and by not upgrading you are taking it one hundred percent chance of keeping its what do you want to be known vulnerabilities issues that everybody knows about and I can great breaking machine easily because it's public or do you want to take the chance of a zero zero one or zero one percent chance that one of those patches which one what's your honest if you wanna think that one fine I can crusher you mentioned that everyone following the series we're finally seeing today we've been going for 14 years there could be years ago yeah oh yeah cool one so recent networking fixes happened that were developed outside the tree developers got done and actually Tomas's been doing this for me as well for the inspector melt and stuff you commit it to your local repo now you have a stable good ID you send that bundle through email to leanness today with me we stuck it in that way and yeah that keeps the same ID and we've done that already so if you really worry about that you can do it that way and we've actually been doing this for two years so we got history doing it okay you mentioned that the way to stay safe is to always be running the latest stable or LTS then you also mentioned that you do a pretty good job at backporting these but your 10% you think you should get 15% of yeah and there's also this whole issue of a lot of fixes are sometimes I think well maybe just running mainline kernel is actually most secure because at least is getting fixes on the other hand there's so much committed to mainline it takes a little bit for bugs so I get this question asked a lot especially with the LTS feels like you're gonna maintain 4.4 for six years they're gonna maintain guns gonna be crazy and do it for 20 years hopefully not going I'm sorry why don't they just pick on four for four forever and the reason is because there are things we miss we know we miss things and there's also other things is newer kernels have more advanced security features they have better phone they have better ways of protecting it a whole class of the bugs didn't even happen so you want to use those features but how much bleeding-edge do you wanna do if you use the kernel within a year I'm happy just switch to that then we'll argue okay nuances but if you just do a year old kernel that'd be a major change in this industry today and it also newer kernels go faster Facebook has documented veyron mainly on internal network and they're getting performance increases every release so if your data center has real power and real things to do colonel that's the another big reason you want those new things it also comes up when you go back to like for whatever you take a look at a particular bug and you see how important it is to fix and you also then look up what is the pain threshold for backboard and how much risk do you have adding the regression by doing the backboard in the first place and that ratio is exponentially down yeah we also have a very well-documented some people from one big company one in some networking fixes done for today but yeah I would say the new year and then we can argue about regression so this came from the Chrome OS people who were I guess it was a noticeable regression and not a logical regret so this is this is numbers from them so they and they have this really cool dashboard they when they roll out a new stable kernels and they can see what fails and what doesn't so that they gave me that number but you're right we do fix fixes now I have scripts that pull and say whatever we fixed the fixes in there and make sure that people fixing my own fixes a lot of times though the fix actually fixes something a day later on I find out oh there's actually another bug with that same I mean syz bugs a perfect example that it buzzes up to one point crashes boom we fix that bug and then the same code battle we now crash again and we're going deeper and we go deeper fixes the fix make sure you tiger sorry I wanted to ask about like when you're doing this you know yes I wonder though if there isn't a difference between like fixing a hardware fixing a hardware block and fixing a security issue which was caused by well something which could be interesting to research so we could avoid it in the future so so this is a policy decision and I have given a whole talk here on why the kernel security team has this policy I wrote a whole white paper on it I can point you to it it was a really fun thread fun thread the cleanest and some security people about why we do this I'll send it to you but there's a reason why we do that this way and I give again I gave a whole talk here couple years ago hello thanks so I'm both holding and asking this question I already know the answer but what's your suggestion for the like poor souls that are stucked on SOC vendor on the corners can't be updated to something else like it not even LTS forget the main so this is a real problem SOC Colonels do add three million lines of code and you're running 3.5 million lines of code it's a Linux like device it's a real problem and the only way to solve that is to force the vendors to get stuff upstream and to write it in your contract so the best thing we've been able to do so far working with Google Google's got me into the door of all these vendors and they know it's a problem and nobody need to do it they're like but our customers don't seem to care so you as a customer if you make these devices have to push it back Sony in their contracts and started putting that into deficit upstream stuff has to do this they realize this is how we solved years and years ago the problem of out of tree binary Linux kernel drivers scuzzy for Ethernet it came down to the hardware vendors should built the motherboards had to put in the contract with the drive that the driver had to be up strangers and had to be able to source the only way we can solve this technological problem is with lawyers that being said I'd beat on these companies all the time media techs doing good Media techs getting a lot of stuff upstream Paul comes up streaming their two-year-old device today but it's getting there some vendors are really good with ants this I say is really good they do definitely knew it right because it saves the money so if a company has lots of money to burn like Qualcomm they willing to be on a trip so yes we were doing really good now including you from West but then so is a upstream which much everything cross rod well but still there are holders that have already we know is they never update anything like the mainline ship what is there what could you possibly there so what you if they can yes if they can update they should and you can so I took a vendor SOC tree from a major Taiwanese company in what name and they said say it's impossible to drag it forward I did it in a week it can be done it's not impossible by any means um you just have to do the work nobody wants to do the work because no he's willing to pay for the work so so we actually do these boards based on an logic and we've been like sponsoring the upstream port and it's been sort of quite the exploratory experience like started working with the bangerra working with consultancies in it but then there's really just like like don't really care about upstream right so I take your I take like stable and then I merge every two weeks and then build the kernel and then release an image but but the Android stuff right specifically the Google TVs though they may work on two-year-old kernels that an impact for two years so and then release them to vendors I mean like people who make boards and those people you know build Android TV with DRM in it and then DRM is one of the things that like drives me crazy because because you know once that key is signed and the image is signed that that thing won't get updated for three years and that's a little key box sitting in somebody's house for I agree that's a problem and they're slowly so the Android phone ecosystem is finally starting to be fixed spreading that to the auto and TV and other parts of Google is the next step I visited one of those vendors that makes Android TVs okay cool I just bought the TV and the engineers they never put it connected to the network like okay so don't connect your device to never girls but that defeats the purpose of Smart TV so then usually lokrum anyway it's a problem it's a real problem it's a known problem it has to be enforced and Google's starting to actually enforce it that's the only way to solve the problem so and this is actually comes up now licenses copyright licenses cannot dictate ethics they cannot dictate ethics that's what they're not meant for laws dictate ethics not copyright licenses so saying dictating use specific use of your code based on a license actually she tells three violated that well argue about that later that's yeah I know yeah but you shouldn't dictate use so again laser wielding sharks running Linux yes so I just wanted to end someday I a mixed question which was there the older dynamics is they are fully mainline and you sure they're on main line on that yes just like you said the newer ones are getting there hmm and if you run the vendor Colonel it's a disaster it's like 6,000 patches on top of dubious quality on top of like engine tension kernel version it's just don't use that ever I mean we've been at it we do this for every base we do this for 20 years we know how to do it right finally we're dictating penalties if you do it wrong and that's again people with lawyers can do that which is nice to see only because security matters this is the only way to get things to do I think Iran de Maio time last one and this would break a bunch of miter rules so it's more a drug than a suggestion but if you tried and have you considered getting a throw away Carol LCME just to issue the single official issued that the single officially issued Carol CVE and then just continually updated it might be fun I mean it's attacking I'm not gonna be me these people have their own problems and other things to deal with I'm not gonna be if I was younger I'd say yes somebody else is young and ambitious yeah colonel community is not let's see see na I don't want to be a CNA turns out Louis foundation is now a CNA thanks to his effort but they do not issue Linux Linux TVs [Applause]

Channel: hupstream

Views: 1,653

Rating: 5 out of 5

Keywords: linux kernel, open source

Id: HeeoTE9jLjM

Length: 57min 27sec (3447 seconds)

Published: Tue Oct 08 2019