Let's Talk To Linux Kernel Developer Greg Kroah-Hartman | Open Source Summit, 2019

[Applause] we are here at the summit in San Diego and today we had with that vector partner Greg we talked so often really not that often but still he talked about things so sometimes we do read a lot of topics right we talk about then we talk about that so I think one one want and then team and yes you are becoming kind of a security expert nowadays so I didn't want to because I had a meeting actually with Intel and I had to go to Portland so they don't let you into Portland less you have a beard right it's ladies hipster thing so now I grew it and I just haven't shaved since dirt has beard Turkish beard I'm also growing one yeah I don't know he talked to leanness maybe when you see him I'll see him in a couple of weeks I last time I saw him he did not have okay so that would be a surprise maybe then we'll also get depressed a lot of material to write about why is he growing feel super lazy and we work from home and we don't want to say so back to the serious topic is security right yeah last year you know it was a bad year for not on Linux but in general you know because a meltdown is Spectre everything happened here but as you mentioned that it was also learning lesson and until did a lot of things to fix things so we are doing now so since the big announcement thing we said these last year said there's gonna be more of these coming right there's been more coming we've had a number of these been released over the past year two weeks ago another one was released publicly that again reduced your performance by one to five percent no fun there turns out there's a lot of these new issues that are coming out people are figuring out how CPUs work internally and taking advantage of that and you can leak data an interesting thing about this stuff is if you remember when spectra Melton came out the OpenBSD guys said disable hyper threading like we were worried about areas miss stuff and it turns out OpenBSD was right so they're right for a little bit of the wrong reasons but there in that now Linux says disable hyper-threading if you're running a system that you don't trust your users you have to disable hyper-threading and that's a huge hit so on some workloads it's a giant hit on others it isn't um but I would like to give full props to the OpenBSD guys for getting that right they took that they took a leap they've chose security over speed performance and that's their mantra and they were right for this one Linux now will give you the option to do pick and choose what you want to do security over speed some cloud providers have picked I noticed I looked on my cloud provider the other day they haven't enabled hold they haven't updated their bias and enabled all the security features recently because the kernel you can see what's looking underneath on the virtual machine you can see if the host machine is actually enabled everything or not one of mine hasn't which is maybe not the best thing but it's good we at least give you the tools to or disable or enable that and see what's going on um but there's more of these coming out the interesting cool thing is about this is all the researchers are now finding these problems by reading the patents so they look through Intel's patents and they're finding security bugs based on that so if you look at all the research all the stuff that was published in the past six months that came out during a black hat and before that it's all based on public knowledge and patents so it's kind of funny it's a little interesting tidbit there so now you see researchers digging through all to find out what the next thing is gonna be in what next thing and so on so um yeah it's been a fun year for security a lot of these issues until has gotten a little bit better and they regressed am I getting a little better because it's still we keep seeing news you know the new family but it's not a you know new family of you know spectrum out it's keep popping up you know yeah I mean we knew these were becoming ups keep coming and they do keep coming because the way these chips work internally is way beyond what we ever imagined and turns out they took a lot of shortcuts in order to achieve speed and when you take shortcuts it turns out to be a side Channel and you can figure out what's going on yeah so what is the ultimate you know solution for this new architecture or the pixel hardware it's as simple as that so if you look at the last couple of things that have been announced it was only Intel chips because they only took those shortcuts AMD haven't there yet PowerPC and arm they weren't affected because they did they worked internally differently so you can redesign your hardware so these are all Hardware specific issues something you can work around in the bias and and they can do that but it comes down to they have to fix how the chips work internally and they're doing that so in a few years we'll see new chips rolled out from Intel or sooner that don't have these problems and we won't have these vulnerabilities so there is and you know kind of and in the site it's not like well it's a pipeline right so you'll see the end in sight for the stuff we found a year and a half ago maybe next year right because pipelines for Jeff companies or years long right so they can only do so much so fast so yeah but whether they keep ahead of all the issues they're coming remains to be seen there are billions of machines already in the market you know oh yeah so I mean you have to so that's why we have to fix it in the operating system right we have to ink and most all these fixes they involve flushing things so we have to when we switch between the kernel and user space or from one kernel task to another task we have to flush buffers and every time we flush buffers things are slow and that's how we fix these issues in the kernel so we have to purposefully slow things down in order to make things secure yeah so we've fixed all the problems that so are announced so far but do that we you have to take performance sets so depends on when your workload is i1 workload it doesn't affect at all other workloads it affects it hugely took 15% so beyond this no when you look at the Linux kernel just forget about Linux and in general because then this is almost everywhere Microsoft is running live so in terms of security you know new workloads are coming up what are the challenges that you're seeing there which you you know from the kernels perspective that you're like hey this is not this should not even be an issue well so there's two sides of security there's reactive security of fixing bugs that come out and then there's proactive security I'm case Cook said the best or is a Constantine with the kernel needs airbags we need crumple zones like an automobile so the kernel security team has been adding these proactive fixes for years a new feature so we have new features of the kernel every kernel release which three or four new proactive security features that cut off whole classes of attacks and that's the best way to solve these things because we can't predict where we always have buggy software but if you take our buggy software and you take away whole classes of vulnerabilities because you just can't exploit that buggy software anymore that's their true solution and we're doing that we've been doing that over years and that's the best solution there was a meeting yesterday for the past two days for the security kernel security for the proactive security stopping side um they're doing that they're adding new features and they're making it better so that's the best thing to do thunders reactive side I've been working on the reactive side and kernel adds our security to kernel adored we react to the problems that are told to us we fix them and we push out the pixels as soon as possible and it's my goal to get everybody to actually take the fixes because that's that's the other side we've been doing these fixes for 20 years and pushing them out there but then nobody ever updated their devices so I'm working with the Android vendors and now Google mandates taking these fixes I have in my talk I showed that the Google pixel team publicly analyzed or they the numbers I give is last year for 2018 they asked the the security team asked the Android team to take a whole bunch of patches that they find based on a wild or what to do for security 98% of them were all in the upstream LTS kernel released already before they had to be announced before they had to backward so you just take the kernels that I released the fixes are already there so because of that a lot of phone companies are now just taking the latest LTS release it has all the fixes they're in there before people realize that their security problem I will call out Sony is doing a really good job with that Sony actually updates a kernel all the time essentials another one pixel phones are good they update the kernel and they make sure they take all these fixes before people realize that they were even effects and then Google's mandating that they're starting to push that out and make their vendors do this more because they know that's the best way so it's to size and they'll so Google mandates a lot of these preventative issues too they put them in their kernels and they test them and they're in devices or shipping there's a whole bunch of preventative measures in the kernels tabria prevent future bugs from actually affecting anything so progress is being made in the last year yeah it's it's a constant slowly forward progress and it's doing really good so far so much so that everybody is now we're looking for hardware bugs in hard were below us right bugs in the software people are still looking for bugs in the software but there's other pressing efforts I'll talk at the Google syz blog guys are fuzzing the kernel like nobody's business that's where you throw random data added and see what happens and they've gotten the tools really good in this past week they finally got USB support in there in their tool so what they do is they look like a fake USB device and they emulate sending bad data to the kernel and then you send it just a little different way and then you disconnect it and you added a new do it and they found tons of race conditions and bugs that have been there for a long time that would just crash your machine but you could potentially create a malicious device if you have physical access and they're fixing these problems so Google's doing a lot of this work and that's all done in public which is great so we're having everybody fix it syz bots publish sizing all these bugs that are in the kernel and we're fixing and what role are you playing in this I mean you are playing you know so are these company like ok they are realizing on their own or because you have worked in past also with other companies to make them more aware of that what role are you playing yeah my I have a marketing advocating job in a way is just going convincing them I have I've done this for a number of phone manufacturers I've audited their kernels because I know we we know we're doing we've cherry picked up right things so like IATA their kernel and see what they missed and every phone manufacturer misses problems so that's part of education process of saying the process you currently have is wrong and you actually need to take the fixes and just take them all because we actually know what we're doing we're giving you 22 fixes a day why aren't you taking them and this it's good I go around I travel different places audit their kernels and show them how they need to be doing things better that's part of the Linux Foundation is it's the job is to make sure that Linux is a good vibrant ecosystem and that people are using it properly because we're giving it to them my 9 theatres and beyond beyond the mobile security anything else that you're focusing on these days or anything else your yeah the security stuff and the plotters like so bad normal development efforts patch review the mentor kernel mentorship program I help out with the review of that application process of what people submit patches the kernel is for the application process and teaching people how to get involved than with the process development processes I still work with that she runs out she work on where it runs out hopefully you can talk to her this week but she's doing a good job that I helped on the patch review part of that so yeah it's busy enough and she's also now a fellow so what happened Jim give him give us on a sword at all no I mean fellows a traditional name in companies for people who are independent and work on projects on their own and she definitely deserves that so you mentioned mentor mentoring you know and so we have talked about this earlier also the the mid dinners are like your beard is no white actually yes fresh blood we always want fresh blood right but if you look at the numbers John Corbett publishes that we're still getting a hundred to two hundred new people every release so the thing is that a lot of lot of things with your problem further that is not problem for you so Michael question but the thing is that you're right when you you guys come up with a report right but to be fair so there is so an interesting thing happened last year when Lena stepped down for a month is we talked about more about how do we what happens the bus factor right up what's it all it turns out that we always so we fill this through we always thought that I had access to write to some of these repos and things like that it turned out he didn't so we fixed that up so and then we added another person so now there's three three of us total well three including Lena second right to his repo and do releases and then on my behalf I need somebody else to help with the stable releases so shoot Sasha Levon 411 from Microsoft Microsoft kernel developer think about that is help me out with the stable kernel tree for many years and actually we're sitting down this week and trying to figure out if all my crazy scripts to release a kernel if you can do it because I think I can do it but I do it from my scripts we wanted to step through it all make sure that if I something happens to me he can do it also because he's a trusted developer it's anything for Ben Hitchens from Debian he's another stable kernel released maintainer but I take part of what he does and publish it but we need to have you know three people that can do this type of stuff in case something happens I mean I just take a vacation a week stable branch that's that's day that all the tinfoil youtubers Livia having a hard time so I mean the Linux kernel development is about not who you work for it's about individuals right trust Sasha's been a developer for but it's so funny to say that right yeah I mean actually the head of the Microsoft kernel or open source group is KY and he came from Novell and he was an ex AT&T engineer for you I mean he's a well known trusted solid engineering manager who's been involved in Linux I don't know for 20 years I mean it's crazy these but Microsoft realizes they publicly said that over fifty percent of all their as your is running Linux right so it's amazingly huge what happened to when Mica's are applied in order to be on the on the private mailing list so there's a Linux distros Bannister composed of the Linux various linux distributions that they help do coordination of vulnerabilities releases this hardware release is actually circumvent that they don't entail never followed the rules so they don't notify the linux distros but it's to like when there's an open ssh problem they all know okay next tuesday we're gonna release and well-coordinated and make sure here's the patches and share them and they work on that to join that there's a very well documented list of things you have to do and the interesting thing is in order to be part of the mailing list in order to do this you have to do work so you can't just be a consumer and say ah tell me what's going on you actually have to do work and you have to say I'm signing up for this I'm signing up for this I will do this I will respond and they and they also have public lists of what people have done in the past months so it's an interesting way to run a manual it's a very good way to run a mailing list um but there's a well-documented process for how to join and Microsoft really has a Linux distribution just like for Azure and for some devices and just like Amazon join this group a number of years ago because internally Amazon AWS systems has a Linux distribution just like were cult is the Linux distribution rehab Susan is canonical o W they're all part of this list because their Linux distribution to push the stuff out so Microsoft join so actually was a the main contact for that he's a well-known trusted developer so yeah Microsoft now as part of that he's kind of interesting I think Microsoft is now a little distributor yeah somebody joke that there might be the largest distributor with the the Linux on Windows right I was talking to Richard Brown from and he like we love it because we our tools are now accessible to 98% of market so that's a great thing it's funny I love seeing it because then people can see on the same machine at the same time how different operating systems run and take your pick for what they want to do because you need developers to be able to run on your desktop the same thing that you're running in the server that's why we uses probably always famously said that you're not gonna be able to have arm in the server station till developers can have another desktop and rely on it over there and it's the same thing with this people these windows developers are running an as you ER and you may have these tools and stuff on their desktop in order to be able to run these workloads that they're gonna do later and it's just it's good Microsoft is really good with developers and then they realize isn't it back I guess the Linux is the cancer it's fine it's funny you know I mean they're they're big companies I mean there's parts of IBM that Hayes also shows that you know you you said you know the our goal was bar dominance we dominate and we did not even talk about it yeah we won and I realize and know even Microsoft is in your bag right it's like any company it's so it's it's fine and we're welcome we welcome everybody to our cuz everybody's different workloads and we want to see in the next work for everybody yeah the problem is a lot a lot of you know people they worry that you know when these company joined they will they will destroy Linux you know but the fact is that they don't understand how open source works everybody has a funny thing is Linux everybody contributes the Linux in a very selfish manner because you want to solve the problem for you I want to be I want to I want to know what your problems are and solve it because it turns out that everybody has the same problems right no it's not so we're you're contributing back and you when you contribute you're contributing to make it easier because it's easier for everybody to maintain it and you have to maintain it on your fort you try maintain a fork it actually costs you money in time if you have money in time to waste great go off and do your own thing in vidya but if you don't you can drag it back and I mean invidious babies millions of bucks to have their own fork and their own stuff off on their own they're embedded in videos embed a group realizes they can do it and they're wonderful contributors to their current so they're embedded side is great because they realize they need to do that cell chips that's redeeming money yeah we want these we want people to contribute what everybody contributes selfishly and it helps out everybody else even Amazon's finally contributing to the kernel now which is great yeah they are there they've hired a number of kernel developers in the community they're working and I realize that it saves some time and money to contribute things upstream it's a business decision and it works out great so David what house works for Amazon yeah Oh David what house okay as a number of other people to write another college is David you may not want to talk about that but I just want to talk about the desktop Linux everybody talk about next year will you DISA does this top matter it matters to me because I've been using it for 20 years as my main work so I want to make sure it works it matters to a number of people because they actually want to be able to run their what they're gonna be making for an embedded device on their local machine okay developers so I'm talking about developers yeah that's what matters to me um matters to other people it's a Linux as a tool right if the tool there to help get you to do whatever you need to work right if you can find a better tool that lets you solve your work that's great I don't care whatever you use um Chrome OS actually is all Linux based and that's a great tool for it's like the number one leading laptop seller for the past three years Chrome OS laptops and that's Desktop Linux you know so when I succeeded there too in a way it again it was a good tool that enabled people to do what they needed to do then you don't share the same concern of returnest I missed almond Richard and I approach things from a different thing when there's the free software versus the open-source software my goal is to see that Linux exceeds hey that's my main I want to see Linux exceed and become a useful tool for everybody to use cuz then they build upon it and translate to these more successful for other people I want to make it a tool that other people can build on use richard has more lofty goals of different ideas there's a more philosophical and more thing and which is great that's his what he does I focus on the technical this way and the licensee road for GPL 2 was brilliant and it's a wonderful thing and we've used that some very successful free software is wonderful I like it I support it I license all my code under the licenses and we're scrape but um on a philosophical standpoint I don't necessarily agree with him on some things that's fine I don't have to agree I mean all kernel do I want to have 4,000 developers we don't agree on anything there's no common thing except that we want to see Linux succeed right Richard wants to contribute to Linux great he contributed the license the reason I asked the question because there's a lot of misconceptions no you can actually have a very privacy watering system built on top of yourself because you yes so everyone gave a talk of the legal summit yesterday in a way talking about how people have seen the success of the free software movement and the way that copyright is or copyleft has happened with open licenses why don't we take that same thing and use it for other things to help ensure privacy to help ensure Federation of services and sure freedom of different religions or different societies and things like that and that's great but you can't do that with a copyright license this time you can't force people's behavior based on a copyright license so maybe you need to do other things and Evans looking at what to do in other areas in order to help achieve those philosophical wonderful brilliant goals in which laudable goals that should be achieved but you can't do that with a copyright license so you need to better things I'm focusing on the technology to create a tool that people then can use to build those Xochi those things what they want to do yes you can use it for evil or good it's a neutral tool that way hopefully where you can use it for good but again people contributing I don't care what your necessary beliefs are I just want to see that you are doing something to help this tool in this project we have people from all sorts of backgrounds and all sorts of different communities that are contributing to Linux because they want to see it work for them and that's what I want to see I mean Linux is seen as one of the most successful cases of copyleft as collaborative but but sometimes people try to use Linux as kind of an example but I said he learned from the next how it made the way committee works we have a little scene over I'm sorry he said you know I don't have any which and you know I I kind of distribute vision as well people bring their own vision they can do whatever they want to do with the kernel and they're just you know to help them yeah I mean yeah we're there to be a few minutes they know the contributor will take where they want Russ so everybody who looks at Linux like when you talk happen and everybody they should also look at how the kernel committee works you know and and embrace those the work culture yeah so and that's actually gets back to the main division of why GPL - and GPL three the kernel community really had some major objection to the GPL 3 because GPL 3 was dictating on how the tools should be used and made some restrictions on how the tool can be used and and the software can be used and us the majority of us in the kernel community did not want to have those restrictions now people might think those restrictions are good and valid and and achieve the goals of free software which is wonderful and good but that might not that's not the goal of for Linux Linux is goal was to be able to for everyone to use it the only restriction is if you make a change and you we just want to see that change so let's talk about some personal questions ok so first of all now you live in neitherlands yes why oh we were living in Paris for a number of years and now we live in the in The Hague and that's a wonderful country that's a great country mine actually my daughter maybe made the observation yesterday she's like I couldn't find you in the grocery store but there's lots of tall people here yes I feel well actually my family history is a little bit of Dutch so my name it works out well that way my knowledge of the language is not very good and my family's studying it much more than I am but I think most people who speak English because whenever we used to go it's the second-largest country what 96% of the country speaks English a bit of France a struggles we would then we live there we would never be able to order peas over phone you have been in the topping so yes my French was Bistro French I could order I could read the menu I could pay and I was about it fast food snack I'm Dutch is much easier so I'm learning to read menus in Dutch but it's in English everywhere I just stay there for a while it will be there for a while these two my son finishes out school for a while but yeah we love the country it's really nice perfect and people always asked what operating system you use oh there's also a number of other kernel developers in the hague oh yeah your are so I've got yeah there's never utter core kernel developers a number right hat developers we we meet once a month on a really most beautiful hacker space I've ever seen this in The Hague it's amazing so yeah so there's a number there is an active open-source community and a number of developers universities are churning out Linux people all the time so sorry was your the question I used to live dear there is a very I mean you get the same thing it was us is big and I I mean where I live in DC there's when you're available I lived there was I believe in Germany there was a community in Belgium there was a community so it is you know everybody will find you know and a lot of people like when we're moving over from Belgium and I always used to be a Linux shirt you know so the movers there oh so you live I said you yeah I know there's a number of vans in the hague that say Linux solutions on the side now I don't know what the company does they do IT stuff but I see the Linux man everywhere yeah the community actually so it's nice being there I can take a train ride to different conferences yes it's very nice so there's not gonna be a harm Prince in the Higgs and um not for kernel people for Cloud Foundry but there's the kernel recipe is one in Paris it's a short train ride away I'll be there that's a good one and there's other ones the question second question was that what operating systems you guys use who I use I use Linux my watches Tizen so my family uses some my son's school actually mandates that they all have netbooks okay interesting so he has a MacBook my wife has a MacBook and that's about it I used arch you know I've used open SUSE anymore I use arch and my build system I think it's actually running Fedora I have a number of virtual machines running still running gen 2 debian fedora to do some testing on some user space tools but yeah I'm all my laptop's and everything switched over to arch these days one thing I mean I also use Mac forums one thing I like like I would Mac was when I discovered because I've been eyes by Mac I I talked to you also you know about it but when I discovered terminal but I was running ubuntu on my Mac there almost and but once I found terminal I were like it's almost it's Linux you know basically I have team Mac machines i SS I didn't do that and I just do everything you know OS actually is now I have a Chromebook that I play around with and you can run Linux applications when you course SSH and anything but a Chromebook is cool and that's a test Chromebook that I play around my teacher so I use my Mac as Linux that's the UI I get it's much better but I just use it like you know but I play mostly in terminal and you can do much more in that yeah well with virtual machines - I mean no windows and under OSX you can do a lot of looks why do we just arch why don't she larch I moved away from Jen to a while ago why did I choose arch and yet at the moment it had something that I needed I don't know what it was the latest development version would not and I've known the number of the arch developers over the years their idea of a constantly rolling forward moving system is the way to go I helped create a tumbleweed for openSUSE at the time a long time ago it's its distro it's neutral it's a community based it has everything I need it works really really well actually converted my cloud instances I have all over arch so I'm it works really well it's nice I mean I mean if you're an art user you'll tell that and that's a joke and I also got a very good document tutorial about it which is one of my all-time popular tutorial only only issue I have with arch is plex server okay our packages it breaks every single time but the good thing with our arch is you know when you are building you know you get all the comments from user they tell you okay this is your in a break you can fix that you know yeah the arch the wiki is amazing the devaluation is like one of the best resources out there I mean it's not about Emily if you look up any use any user space program and how to configure it and use it actually the system D arch wiki pages are one of the most amazing resources out there I mean I I said no I I was when I was pure Linux user now I am channeled so much I don't have time to bring to machines so I just use Magnus Lee but I when I was our Linux user I was in the arch user and you know I just love it because you know you keep them system update all the time and it's contrary to belief because when you make minor incremental changes the system doesn't break you know system when do you know a breakdown yeah and also main main policies of arch or philosophies as you stay as close to the upstream as possible yes and as a developer I want that because I want to see in there they're updating the kernel all the time they patch very little and babay also they're really good and actually feedback to the community because I want that testing I want to make sure the things are fixed if it is broken I learned about it quickly and I fix it and push the stuff up so that's actually a really good feedback loop and some of the reasons I needed because I was looking at I was using the latest I was modifying system D or on applying some other users based up and I wanted to latest versions of everything Fedora is great too but it was like six months behind on some things Fedora's also nice for them but the whole you have to stop your machine really updated which is I understand why they do that and whatnot and you can switch to rawhide and stuff like that Debbie's also good for other things oh cool fun fact about Debian them and the cloud the instant thing is it's not a corporate thing anymore with a huge growth of Linux in the cloud the numbers is something insane like 80% of the world's Linux servers are running either a Debian or kernel.org kernel and it's not that Red Hat and Susan's has lost market share it's just that the growth of everything else is so giant and the Carlotto base the system is like I'm JW SS colonel it's from Colonel or a few batches Azeris colonel colonel org with a few patches and they're kept up to date so and Google compute cloud same idea um so when companies only try and deal with other companies they shut out the huge majority of systems acting all community based it's amazing Debian is actually really good and there it runs the world so it's great it's a nice stable base of people work on and Debian developers are great I don't use W but those guys my server runs on fire yeah and they keep it up-to-date and they handle the security backwards I don't agree with some of the philosophies about what that what they do is they do it really really well I think we have covered a wide range of topics today and anything else you like to any message you have for the so correct thank you so much if you're talking today and as usual you know I look for to talk to you see you I guess yeah see it's another other country under continent right and please keep the beard [Applause]

Channel: TFiR

Views: 26,292

Rating: 4.9223299 out of 5

Keywords: #OpenSource, Linux

Id: sDrRvrh16ws

Length: 31min 40sec (1900 seconds)

Published: Tue Sep 03 2019