>> Hi everybody. Since we started a little bit late, Iím going
to keep this nice and short unlike our guest, Greg, who is quite tall. And--yeah. This actually--it rose out of a discussion
we were having between some Kernel folks that I know, and Bryan [INDISTINCT] and right over
here, about sort of the nature of the embedded system--what's that? >> It might not [INDISTINCT]
>> Well, no, I mean, like from outside the company, inside the company like--donít--I'm
speaking here. All right? >> Go ahead. >> Anyway, and so, you know, at the end of
this today, I want you to come out and speak. It would be fun for us and fun for you and
stuff. So, here we are, Greg. >> HARTMAN: Thanks. Is this working? All right. I'm Greg Kroah Hartman, Iím a Linux Kernel
developer. Iím a sub-system maintainer of the Kernel
for USB, driver Core, I used to do PCI, I like to see all the weird things. I'm one half of the stable Kernel maintainer
group which I'll talk about later. So here, we're going to talk about the Kernel,
the process, who's doing it, how itís being done, why you should care and why Google should
care and things like that. Please tackle--ask questions in the middle,
I like it. Make fun of me. I'm going to make fun of you. So, beat them. So, first thing. This is the most important thing, donít get
anything. Realize this. This is what the Linux Kernel do for the past
year and a half from the 2.6.20 to 2.6.25 Kernel. This is what was changed per day in the Kernel. This is a stable Kernel. This is a stable development process. This is freaking insane. It is the fastest moving so--project--so--software
project known to anybody. Largest project of its kind known to anybody
but it's also created a result thatís never been done before and we have an operating
system that supports more devices than any of our operating system ever has. We support more processors than any other
operating system ever has in the history of computing. And we're doing it in an open manner, in a
very chaotic manner. Moving at a rate, that fast; shipping something
thatís stable, that everybody, including yourself, uses everyday. We broke all the software development rules
and we're continuing to break it. If that doesnít make you guys scared, it
should. The biggest thing to that make you guys--should
make you guys scared is, if you keep your code outside the Kernel, thatís going to
pass you by. Do you think you can keep up with that kind
of rate of change, figure out what's going on with that rate of change? If you get your code into the Kernel, that
rate of change will happen to you automatically. If you donít, it will blow you by. So, get your code in the Kernel. It's the main thing I want to talk about. So, thatís it. Thatís crazy. And also, this is the first question people
ask like, "Oh, thatís jus drivers, right? Drivers are changing that fast.î No. It's the whole Kernel. I looked at it, Iíve broke it down by the
proportion of the code--size of the Kernel. The Kernel is about nine million lines of
code. The Core--the Kernelís five percent, driverís
a little over 50%, 25%. If you look at the rate of change at each
individual portion, it's exactly proportion--itís exactly proportional to the amount of code
thatís in the Kernel for that section. So, the Core of the Kernel is changing at
5% of that rate. The driver section is changing at 55% of that
rate. The architecture specific spot is going up--what
is that? I donít know, it's going at 30% of that rate. So, it's uniform. It's flat across the whole thing. So, donít think that just--that the drivers
are changing, it's the whole thing; it's everything. So, and along with that rate of change, it
breaks down to this. It's going really fast. Three point sixty nine changes an hour, 24
hours a day, seven days a week. During the 2.5, the 2.6 development cycle,
we're running about 2.5 changes an hour. And that was unheard of; no way that Iíd
ever thought we could do that. Since then, we've been going up. I donít know why. We're getting faster. We're getting faster. We're doing more rate of change and it's getting
bigger. So, this is some of itís staple. So, here's our graph. If you want to see all the--that lines of
code that are added and modified, diluted over time, I just did for the 26 [INDISTINCT]
and long. If you look--if you used that, it was a little
bit less. So, it's--itís really--itís kind of staying
around the same but at lines out, it keeps going up. Thatís interesting. Thatís not really trend worthy but it is
continuously going up. But itís trend worthy is the changes per
hour. It's continuing to go up. 2.6.25 changes per hour is amazing . Well,
that rate of change does not included moving files around. Moving files around the Kernel doesnít affect
it at all. It does not change at all. So, we're going faster, doing more rate of
change per hour per release. Everybody happy? Nobodyís questioning that? Oh, come on. All right. So, how we do this? Hereís how we do this. Extensively, it starts off at the developer
on the bottom to make a change to a file, to--they find a bug, they want to add a new
driver, do something. Then it takes that change, the path, should
they send it off to the maintainer or the driver of the file. In the Kernel, we have about 600 unique maintainers
that are listed on who owns what. You canít find what--who owns what, you can
look in the file using git and see who modified it last. Whoever touched the floppy driver last is
the maintainer. Never touch the floppy driver. So, that personís responsible for looking
at it and then they see it, they review it and then they sign off by it. They say, ìThis looks good to me. I think itís okay.î And they add their signature
called Sign Off. And Iíll talk a little bit more about that
later. So, the developer says, the sign off, itís
good. They say itís good. They send it off on to the subsystem maintainer. Subsystem maintainer like USB, PCI, BFS, security,
Core, things like that. They consolidate all the patches from all
data people and then they hold on to them. They create this little mini trees everywhere. These little mini trees are either in gits,
using quilt, somebody--I think thatís all everybody uses, some tarballs maybe. And then all that, every night gets pulled
into the Linux-next Tree. Itís from Steven Rothwell in Australia. Which--for IBM, the Aus labs. He pulls it in every night, the Linux-next
Tree merges them all together and sees if it builds or not. Sends you angry emails or you break the bill
or you break the merge, itís great. Itís awesome. Itís testing all the subsystem maintainer
stuff for all our development things. Andrew at the same is creating--is now relying
on Linux next and sucking everything in the host tree, about once a week or so. And then also, he pulls things from the Linux
Kernel Mailing List, a sort of mailing list that happen to get drop by subsystem maintainers
or whatnot, he gets--he sucks those under his tree and everybody--and then he does a
release every couple of weeks. So thatís testing, this is the developmental
stuff. So all the crazy stuff the subsystem maintainers
are holding on to, all the stuff thatís coming across the mailing list but Andrew picks it
up, gets pulled to these releases. Then when Linus says itís the merge tree--windows
open. Iíll talk more about that later. All the subsystem maintainers hit Linus. We got two weeks to hit Linus with all the
changes you want. We all go to Linus. Andrew sends a bunch of stuff to Linus after
the subsystem maintainers get there and then we go on, the cycle releases. Iíll talk more about releases later. So, nice little tree structure. So, Linus and Andrew and the next tree. Does that make sense? Nice hierarchy, right? Nice and pretty. So, Iíd try to graph this. Looking at the relationship between people
and the work--way patches flow to the tree. I think itíd be a nice pyramid or whatnot. It turned out, it isnít. I made a graph that was about 40 feet long
and 4 feet tall, of one Kernel release. It was a mess. Itís a big giant spaghetti mess. It looks like an internet routing system. And which is good because it means that, if
this person doesnít pay attention, we route the packets around them, Andrew picks it up
and then sends it to subsystem maintainer, it sends it to Linus or this person sends
it directly to there and goes around the maintainer or this person disappears for a week. So, itís very good. Itís lossy. We can handle people disappearing, handle
people come--reappearing and it works out really well. It isnít a strict hierarchical development
system which is good and it shows how we can keep up this massive rate of change so fast. Itís kind of scary. If you own a file or a driver, you need to
accept the fact that other people are going to be changing it, extensively, without you
knowing it but hopefully you should be paying attention and you just have to accept that
so other people can do that. You can always change it back if you didnít
like it. But thatís the way we get things done. Thatís how we can work so fast. Make sense? Cool. So, hereís a release cycle. So, Linus does the release based on something. We say itís good. So then, we have two weeks. We can throw anything we want at Linus. All the subsystem maintainers throw everything
we want. Extensively, itís been tested in the Linux-next
Tree, before that I have been testing the MM3 from Andrew. So, we hopefully expect that the code has
been tested, it builds, it hopefully compiles sometimes and all merges. So, thatís when the hundreds of thousands
of patches go in at once, two weeks. Linux releases an RC1 Kernel. Then the subsystem maintainers that--were
on vacation or forgot about the merge window being open, wake up and say, ìOh, no. I forgot my stuff.î They send it to Linus
again so, then goes an rc2 a couple of weeks later. Hopefully, thatís getting better. People are slowing down. Everything happens between rc1 and rc2. Then, its bug fixes. We start doing bug fixes, we track bug fixes,
regressions, we do RC1, 3, 4, 5, whatever. Until we think things are slowing down and
good enough and boom when it says, ìOkay. This oneís good. Out the door.î And then we start the whole
cycle over again. There is no stable and unstable development
trees anymore. You step odd was unstable and development
even was stable, not anymore. For the past four years, weíve been doing
this model. When we finally realize that this model really
worked good, we said, we should try and do release every two to three months, right? If you look, weíre releasing on an average
of every two and three quarter months, for the past three to four years, like Clockwork,
you can trust that a new Kernel will come out between two and three months. This is a really, really good thing. Unless the up stream just rose, figure out
in their schedules when things are going to be done within a--within a reasonable time
frame that can plan when things are going to come out. The [INDISTINCT] can figure that out, developers
can figure that out, ìI miss this merge window, whenís the next one going to come around?î
You don't have to worry that their selves are going to get drop for forever. It works really, really well. One thing that happened though is we realized
during these two to three month time, what's happens about security bugs? These are people that actually use the Kernel.org--Kernel. What happens then? Thereís bugs, security problems happen, what
are we going to do? So, we formed--two people formed the stable
tree which is me and Chris Wright who works for Red Hat and we do security updates or
bug fixes; 0.1, 0.2, 0.3, 0.4, 0.5 and we have a real strict list of rules of what we
will accept, what we won't accept. It's pretty easy. It has to be bug fixes only and security fixes. The patches have to be in the upstream tree
already, unless thereís some really, really odd thing, things like that. It must be small, obvious. We do those releases every couple of weeks,
whenever we get around to it. Sometimes we're slacking, sometimes we're
not. Security updates, we can do it usually if
we got an inbox. We can do it in couple of hours. So, we release things as soon as possible
if we can. And that's working out really, really good. The fun things is, after the--we drop them,
after 2.6.20 comes out, it might be one more 2.6.1.9 stable and then we stop. We don't have to do it. Weíll have to maintain that for forever. You can maintain it for forever if you want
to. Somebody picked up the 2.6.1.6 Kernel, decided
to maintain it on his own for a long, long time. We all thought he was crazy. He got a job out of it. So, maybe he wasnít so crazy at all. So that he could do things over time. So, it worked out really well. If you want--if anybody wants the 2.6.2.4
Kernel to maintain it for a while, let me know. Weíll give it to you. We don't care. So, that's how it works. If itís stable, releases happen, development
and releases happen and then we drop stable trees. So, we don't have to maintain them for forever,
which given our rated change previously is a good thing to do. Makes sense? Everybody happy with that? Is it working? All right. >> So, you guys don't have an open analysis
[INDISTINCT] every blogger really preferred at least [INDISTINCT]
>> HARTMAN: We don't have a notion of a preferred release anymore, no. Every release is a preferred one, the latest
one. Based on the rate of change, if you report
a bug--a bug problem on 0.2--on 0.2 Kernels ago, it's given how fast Kernelís often changing. We're going to ask you, can you reproduce
this on the current version? It's the first thing we're going to ask you. So, yeah. So, everythingís preferred. The last one is preferred. Now, note when the distros do releases, we
know what Kernels are going to release. So, sometimes those Kernels hang around a
little bit longer. Fedora and OpenSUSE are based on 2.6.2.5 right
now. Odds are that Kernelís going to hang around
for a little while because Chris Wright works Red Hat and I work for SUSE. So, weíll probably want to hold on to that
Kernel for a little while longer, make sure bug fixes get in there and it's good. But, if you want the new features and things
like that, you should go to the latest one. [INDISTINCT] Yeah. >> Why use the two-week merge window, why
not the [INDISTINCT] >> HARTMAN: Why two-week window, merge [INDISTINCT]
what? >> Why accept merges all the time? >> HARTMAN: Why Linus accepts merges all the
time? Because we want to be able to throw a bunch
of things and we wanted to shake things out and stabilize them. We want you to test and feel good that we
can--at the very end, we don't want to accept the massive bunch of USB changes that nobody
has really tested or run for the past three months--three weeks or so, you know, we want
to at least some kind of software--good engineering practices. It looks like it something, we know what weíre
doing here. Yeah. >> Thereís never a period of where thereís
no merging or [INDISTINCT] their merging over to something newer and then the RC what windows,
youíre shutting off with, is that correct? >> HARTMAN: So thereís no... >> [INDISTINCT] gaps where... >> HARTMAN: Thereís no gaps where were not
accepting any merges, yes. Well, we think of, yeah. When we--we use gits to send things to Linus. So, he pulls from the--the subsystem maintainers
trees and put--merges them in. So, that's where we merge things but it should
be bug fixes only. From RC2 on, unless it's a really odd thing. We should have bug fixes only are going in,
into the merges. I think of merging as a new feature, I'm sorry. Itís a wrong terminology. I use--you can use git to send this stuff
to Linus. Linus came here and talked about git. I use quilt. Quiltís really good too. So, yes. >> So, how do we test them? >> HARTMAN: How do we test? >> Yes. >> HARTMAN: One thing about the Kernel is,
there is no good way to test it except running it. You canít run like unit test--you can write
unit test that extensively test some of the external features, such as the system call
interface, it hasn't broken some odd things. But really it is testing--the operating system
is running on your machine. There are so many hundreds of thousands of
permutations or devices and interactions with timing and everything else that we wanted
to run on all your machines. So, we rely on you to test. We rely on the developers to test and that's
the only way we can obviously test. Like--classic example is, I had a problem
when you were--you were plugging in two USB devices at once and yanking one out, the data
was being streamed out too. You canít unit test that unless I want to
put--I have a big box of USB devices that I do test a lot, but it doesn't make for an
automated system. So, we ask you to test it yourself. We rely on the development community to test. That's all we can do. There isn't a same way to write a unit test
for an [INDISTINCT] it just doesn't work because we're relying on so many external inputs. People keep trying and that's great and we
want as many test. We want as many functional tests as we can
get. But like performance--and we performance test
also. People are trying to test it but it's a real
hard thing to do that. >> So there is no formal--letís say, we have
between--like, demos in the 2.6.25 and the last one out, [INDISTINCT] they were waiting
for someone to do something for it. Is this a formal procedure or does somebody
does [INDISTINCT] and what is something that you [INDISTINCT]
>> HARTMAN: What if--so, what was holding it up and what was the formal procedure? We do track regressions. So, like people record a regression, ìHey,
this worked on 2620 and 2.6.21 RC whatever, itís broken.î Regressions retract. So, towards the end of the release cycle we
say, have we fixed all the regressions? Have we fixed all the bugs that have been
reported? Some people are trying to track the bugs that
have been reported. If it isnít a regression and it's just as
a bug that always been there or something that new happens or for something that's been
added that's brand new and newer support [INDISTINCT]. We don't really treat that as important as
a regression from the--actually machines working. We want to make sure we never break a machine
that was working previously. We ripped whole features out because we broke
the machine that was working properly. Which is tough, ACPI and some interaction
and some low level platform code is nasty. So, we change that and it breaks whole models
of laptop, thereís the IBM series of laptops that was notorious for breaking whenever we
change anything in ACPI. Itís tough. So, we try and track those and that's what's
keeping us, ìLook, we know that thereís this one class of bugs that a bunch of people
are hitting, letís try and fix that up before we do get between RC 7 and the final release. So, we do keep track of that. So, if you have a problem, let us know. So, as a developer and as a user, if you want
to help Linus out, test these RC releases. Test them and let us know if there's problem,
you have regressions or not and they're good. I mean, all these are run on everybody's machines. You don't have to worry too much about these--these
things. I wouldnít run it on something thatís production
wise but definitely run it on your local machines--around on my laptop, all the other ones. If you want to test what's--if you want to
help us test what's going to happen next, test the Linux-next Tree, test Andrew Mortonís
tree. That shows that crazy developmental stuff
thatís still under heavy development that were trying to merge in and figure out what
is good enough to go into the next merge window. We got features in there that aren't really
good enough to go. There might be a couple releases away, things
like that. What's the question? Okay. All right. Move on. Like I said, we're doing this every two and
three quarter months. So, we're actually doing what we said we were
going to do and that's a good thing, that's a time-base release cycle; itís turned out
to be a really nice thing. It works really well. A lot of Open Source projects are kind of
switching towards it, it works. You know that all of [INDISTINCT] this feature--this
window. Two and three quarter--three months later,
itíll get in. It's a good thing to do. 9.2 million lines, we've been growing at a
rate of 10% increase in code size per year since I started tracking it back to 260. It's a linear graph. Itís just pretty much straight which is amazing. We're still growing. We're still getting bigger. That's a pretty big chunk of code. Now, not all machines run at 9.2 million lines
of code, that's funding. You write in the Core and a few drivers around
it, not working and things like that. So, that's everything. That's all the different platforms, everything
else. And in the last year and a half, this is the
number that I find the most impressive. We had 2,399 unique contributors to the Kernel
and that's amazing. That's showing we're doing really, really
well. Now, half of these contributors only contributed
one patch. Half the half contributed two, half the half
the half contributed three; well, exponential curve there. But the top of the curve is getting flatter. When we did the 2.6, the 2.5 or 2.6, the top
20 people did 80--8%--80% of the work. Now, the top 30 people did 30% of the work. Just by quantity of patches. So, we are flattening out and weíre spreading
the load out a lot more evenly. I looked--it was 200 people--the top 200 people,
do they--no. Two hundred people did 8%? Something--I was
thinking of the top 200 people that did a lot of work. I was spreading a lot better. So, that's good. I mean, here's the--the backgroundís graph. I did the crazy graph. It's on kernel.org, you can try and load it
in your browser. It's a good SVG test. It broke [INDISTINCT] machine, crashed Mackintosh,
it only--I got somebody from HP to print it out, he had printers in his barn. Big roller printers in his barn, he printed
out that. You can go through him. So, thatís our number of developers, thatís
good. And our number of individual developers, per
Kernel release is going up also. A number of companies that are actually sponsoring
developers is also going up which is a really good thing, showing more companies are getting
involved as a spread of Linux goes bigger. So, letís talk about people, quantity. We can track people like quantity, this is
just quantity not quality. I'm not going to talk about quality. So, this is the number of changes that got
into the Kernel. Top 10 people for the past year and half;
number one, Adrian Bunk, there's a lot of janitorial patches. Al Viro of BFS. Thomas Gleixner, the X86--no, i386 next 86
arch maintainer along with Ingo Molnar. David Miller networking Core. Bart, IDE. Paul Mundt is SH. Ralf Baechle is MIPS maintainer. Ingo again, Patrick McHardy is net filter
and Tejun is Lab ATA, disk drive stuff. So, two of these drive guys are top 10. Thatís quantity. Thatís a number of changes that they got
in the Kernel. Yes, [INDISTINCT] So, signed off by. Remember how I mentioned every person along
the hierarchal tree signed off on it actually reviewed it. Here's the top signed-off, this shows who
are the gate keepers and how--who's doing what. Andrew is the top. Linus is next. Now, the funny thing is, Andrew has more than
Linus, so that everything goes to Linus. Linus doesnít sign off on patches he pulls
from sub-maintainers, we're using git because he isnít looking at them. Heís trusting those maintainers to do the
thing. So, I make up a git tree for Linus to it pull
from. He says, ìGreat.î He pull from it and he
doesnít sign off on those. That's why his number is lower than Andrew's. So actual the real number patches that got
in is less than the sum of those two, itís kind of funny. So again, David Miller in networking. Jeff Grazik is in network drivers. Ingo, architecture for Intel. Me, Thomas are in Intel. Mauro does video for Linux, little webcam
stuff. He does that in his own spare time. Works for a telecom company in Brazil, awesome
stuff. Paul Mackerras is Power PC. John Linville is wireless drivers. So, you can kind of see what areas are the
code, Kernel is doing a lot of activity. So again, video for Linux is a very active
area, wireless is a very active area. Interesting. So, let's see who is funding these companies. So, we can see the people, receive the number
of changes it does. So, who's doing this work? So again, for the past year and half, Red
Hat, IBM, Novell, Intel, Oracle, Consultants, these are individuals working for consultancy,
they are being paid by other companies or working for consulting companies themselves. Academia, I'm very happy to see Academiaís
actually going up, at numbers going up. And Renesas, Renesas? I think there an SH maintainer--developer. Which is really interesting, they just pop
in the top 10. So, you might want to guess what's number
one to five? Come on. Canonical? I'll tell you where Canonical is later. Well, I'll tell you where they are now. They're about 300th, I think. Canonical had six changes in the past five
years. Canonical does not give back to the community,
that's what they do. >> Linux foundation. >> HARTMAN: Linux Foundation, no. Linux Foundation is not on the list anymore
because they only have one developer. >> They [INDISTINCT] dock book. >> HARTMAN: They log to--I'm sorry log what? Dock book? Docks has to do with others and that goes
in the Kernel. Anyway, come on. >> SUSE? >> HARTMAN: SUSE is Novell. So, Novell, SUSE same to go. HP, no. HP [INDISTINCT] all right, Iíll just go. You guess. Amateurs, number one. These are people working for themselves. We have asked them who they work for and say
I'm a hobbyist and doing something on time. And the number five is, unknown. They are either, we have no idea who they're
working for or that they wish to remain unknown. Some people say, ìIím not allowed to say
who I work for.î Hereís my [INDISTINCT] So, we track this stuff. So, between the--some of those two, itís
almost 25, almost one quarter of the number of changes that go into the Kernel. Who's being done by people that are not being
sponsored to do it. And another thing about these unknown individuals,
none of them--well, I think--I think about five of them are contributing more than 10
patches. So, there are lots and lots of people in the
tail that we donít know who theyíre working for. So, itís not a large--they're not large contributors. So--or you can look at the other way, 75%
of the development thatís on the Kernel is paid for, is sponsored. Because these companies care about what happens
to Linux. If you want to see what's going to happen
to the Kernel, what's going to work for you, you need to influence it and realize what's
going to be coming in Kernel? What's next? It's whatever you make it to be. Whatever you want it to be lots and lots of
companies are working to--are relying on Linux. You guys rely on Linux. Lots of other companies, Iíve given this
talk to a number of companies. Only one company Iíve ever given this talk
to is actually on this list. What? Google's not here. >> [INDISTINCT]
>> HARTMAN: Where is Google? Googleís 13th. If you don't count Andrew; so, without Andrew... >> We count Andrew. >> HARTMAN: You count Andrew. I count Andrew too. Andrew does great work. But--so, I was trying to think of a metric,
do you--do you do the mean of a company's developers, you know, the look of a massive
body or do you count the net quantity but in smaller companies of computer developers,
how do you rank them? I don't know how. I'm trying to figure out some fun stuff with
statistics. My wifeís in Graduate School, she happens
to be taking statistics class right now. Since she has lots of ideas, I can use her. So, yes. Without Andrew, you guys are 40th. You guys rely on Linux. This either means one or two things; that
the way Linux is going and the Kernelís going--and working, itís just fine for you. And youíll like it and youíll love it and
you're happy. That's great. Lots of people are that way but the other
thing is, do you trust these companies to develop the Kernel in a way that matters to
you. I would say no. >> Why? >> HARTMAN: You trust these guys? They don't know your business as well as you
do. You only know the loads that you do as well
as you do, so if it's going as well then fine, I'm happy and I'm not going to complain about
it. Yes, [INDISTINCT]
>> No, I mean, one thing that happened to us on the Andrew type thing is, they will
move support for--I know a trial [INDISTINCT] this is [INDISTINCT]
>> HARTMAN: Yes. >> If we invented more active earlier on,
something might be, you know, and check it. I think they mention only one thing in the
Kernel that uses this and is just not working. >> HARTMAN: Yes. >> Maybe if we were in there, we might have
been involved in this setting. They might has still been removed but... >> HARTMAN: At least you should know. I mean, if there's things you care about,
that's fine. And that's fine. If you care about--talk about this thing,
that's great. When I come is I'll talk to you and say, ìWell,
company ranking so and so is doing our work for us,î which is great too. All these companies, I know IBM or Red Hat
and Novell. We both do consulting for other companies
and do work for them. But again, their overworked companies and
there's no reason why the individual company yourself can't get involved. We make it very easy to get involved in the
Kernel element process based on our--number of people who are contributing, it's kind
of working so there's no excuse for you guys not to contribute. So, that's my reason. So, let's pick-out some and then weíll [INDISTINCT]
so, 27 people are in the list. Twenty-seven of you guys contributed a code
[INDISTINCT] and here you are. So, Andrew number 11, awesome. Great. Very happy. David, number 149th, see here, that's almost
coming. So, he is a perfect example of what I like
to say is a great person to have. He started of working at [INDISTINCT] University
doing janitorial Kernel work. He looked at the Kernel, here's a fix, I can
do, start running into it, got hired by you guys, I don't know what he does, and has contributed
now to the Core Kernel, the, BF--to the--what does he do? NM stuff, out of memory stuff, Core Kernel
guy, he's contributing pretty well, 67. That's in the top 8% of all Kernel developers
like quantity out there right now, great job. Paul, what does Paul do? I looked at CPU scheduling, that's good. Ken does what? He used TLB and other sort of stuff. Nice Core Kernel work, really good. Then there's Matt, is Matt here? Okay. Matt puts everybody else in this company to
shame. Matt contributed spelling fixes to the Kernel,
which is awesome, and I will not say--and I love him and that's a really good to have
but Matt beat out all your other Kernel guys, he's number five out of the whole company
contributing changes that fix spelling. So if you're a developer, I donít--you should
feel a little ashamed about that. I mean, I gladly take the spelling change. I hope they contribute to more in the future
but come on. All right, guys. And then I didnít--I gave up on looking at
[INDISTINCT] what you guys do. >> How does the 27 compare to the other companies? >> HARTMAN: On quantity? Number of developers? >> Number of developers. >> HARTMAN: It's pretty good. There's some companies with like 90 some,
which is amazing like that--but then there's smaller companies with like five developers
that are very high in the list too. So it's like I can't look at pure quantity
of unique contributor. I just can't figure out a number. Do I look at the mean, the contributing one? I look at the mean here, you're way down to
seven, you know. I think you guys want me to do that. So Martin, where is Martin? Martin disappeared. So the joke is when anybody ever goes to work
for Google, they enter the black hole. Never to be seen from again, Martin, you still
actually contribute to the Kernel. I know Martin, so I can make fun of him. Here's some more, two with patches, two patches,
two patches and we start getting to one. Al, thatís cool to see Al works here. He used to contribute a lot; used to and then
there's the rest. >> Just that there are so many interesting
things. >> HARTMAN: There are so many interesting--and
thatís great and thatís awesome and I have nothing against that. But if you can--if you--and I know you guys
do a lot of development in-house and have your own custom set-up and certain things
like that, which is great and fine. But if you think you can keep your changes
outside the main Kernel tree and keep up with that rate of change, I think you're going
to have a problem. It's really, really hard to do even with our
good tools that handle merging, git and things like that. I know this really doesnít show and none
of that Android stuff has landed, hopefully there's Android shuffled in. >> [INDISTINCT] know, your work is passion
able. >> HARTMAN: Okay. Four [INDISTINCT] already, I can handle it. >> Do we have a locker room? I'm hoping to get that [INDISTINCT]
>> HARTMAN: Hopefully, yeah. I've heard--I've heard that before. But yes, I trust you guys. I trust you guys will gather the stuff but,
so. >> [INDISTINCT] trust yourself this [INDISTINCT]
>> HARTMAN: Trust yourself just to maintain it. >> There's a lot of talk about how easy it
is to contribute and your mileage can vary sometimes. >> HARTMAN: Your mileage varies sometimes
and how--yes, and how easy it is. If you ever have a problem with getting things
in, talk to me, talk to Andrew, talk to the other people. We will help make this happen. I have developers that are willing to help
out with cleaning stuff up, we've take--we take code from other companies that are not
the proper coding style, we'll clean it up for you. We'll maintain and we'll get it in the Kernel. We do this kind of stuff all the time so it
shouldnít be here. So, my main point here, you guys rely on Linux,
there's no reason why you guys can't get involved. You guys should contribute and help and drive
it in a way that you feel is going to work best for you. Otherwise, it's going to work--good then I
hope--and I hope it still works, so. Everybody asks what's new. Here are some of the things that happened
in 2.6.26. It's a joke to tell you everything that happened
new. It's a lot of stuff happened, a lot, a lot
of the stuff. So any questions? That was fast. In the back, yeah. >> When you talk about the growth of the number
of lines imposed with the Linux, I think they probably have 95% of that is device drivers
orÖ >> HARTMAN: Well, maybe the talk of the code--95%
of these drivers, no. I think you missed in the beginning, 50% of
the code in the Kernels is drivers--55% of codes in the Kernels are drivers. The rate of change is proport--is exactly
proportional to the amount of code that is in the Kernel. So those 4,000 some lines added, 55% of that
is in the driver's subsystem. So 5% of that is in the core Kernel. It's proportional across the way. So the majority is drivers by a small bit,
after driver's is architecture specific stuff then networking but that's it, so it isnít
all drivers. But it's scary in a way, that means your Core
of the Kernel is changing that fast. The Core of the Kernel is changing at a rate
that blows all other operating systems away just by looking at the map or just touching
the core or the whole subsystem, think about that slurs. Yes? >> So what would you say is driving the change
to the Core Kernel? Is it because people just keep getting up
better ways of doing things or because the requirements are constantly changing? >> HARTMAN: So, what's driving the change--the
thing in the--what's driving the real change in the core? It is the fact that Linux is--I mean, Linux
always jokes--Linux is not--is not intelligent design, it's evolution which means we react
to stimuli thatís happening in the world. Just like, "No, we have to support this new
architecture which means we have to change the core to do this now." We donít over plan things. We take what's happening right now. So, yes, new features happen and we add and
we figure out how to change the core again to work. There's people who come up with, "Hey, I have
good idea that we can get this--if we rewrite these--we rewrite them--this locks to new
[INDISTINCT] instead of spin locks, we can get the RT stuff in there and get real-time
in the core. The real-time stuff shows that you can take
these crazy out of the tree patches and merge them into the Core Kernel and now we have
a real-time where Kernel of Linux--some of the people said that it can never be done,
which is pretty amazing from the [INDISTINCT] so, it's everything. Itís just responding to external stimuli. So we're not over planning, we're reacting
at the time that it needs to be done. [INDISTINCT] a lag a little bit, so some hardware
companies would come up with a new device and like, "Oh, crap, we you rewrite this." Luckily we're getting better and these hardware
companies are getting involved in the process. Look at Intel in that list. Intel's very high. They're very big and Linux because Linux runs
on all those chips they want to sell chips they make sure work's really good on their
chips. [INDISTINCT] penguin picture. Yeah. >> MALE: So why donít you guys don't we pay
more attention to consistency? >> HARTMAN: Why don't we pay more attention
to consistency... >> MALE: [INDISTINCT]
>> HARTMAN: Something's starting to change rapidly and our interface would do change
rapidly. Hopefully, we never change the user space
interface. So, we have the boot line changing for them
to do stuff. I don't know maybe they had a really good
reason to do that. Hopefully they did and that sounds like a
foolish thing to do. We hopefully never change--we change the internal
Kernel API all the time. Everybody was like, "You have to have a frozen. You have to have a state API." Then you say, "What API?" We have 8--9,000 symbols, we have 12,000 different
structures. I mean that are passed around inside of the
Kernel. You can't freeze that because it doesnít
let's you ever change anything. So inside the Kernel we change all the time,
we change rapidly, we have really good reasons for doing that. But outside the Kernel user space API's, we
should never change. I can run binaries ever built for 0.696 just
flowing today on the Kernel, 2.6 Kernel today, cisco interfaces don't change. For the most part, there's a little gray area
of some interfaces that were on the Kernel and user space like the proc file system that--end
up changing every once in a while but hopefully we catch all the--the users around that. But we're trying to document that. We're documenting that user space API now. There's documentary [INDISTINCT] ABI, that
we document that. The command line stuff--we do document that
although change was odd, I don't know why. It shouldn't have. You should complain to developers. Maybe they didn't realize the broke it. Sometimes that--like, "Oh, I didn't realize
I broke that." You know, like, "That's a book. I need to go fix it." Regressions are always going to happen and
hopefully we fix this. In the back. >> MALE: Once a upon a time, you advocated
the distribution [INDISTINCT] >> HARTMAN: Yes. >> MALE: [INDISTINCT]
>> HARTMAN: Yes. >> MALE: [INDISTINCT]
>> HARTMAN: So the question is awhile ago, I said that enterprise Kernel should just
grab the Kernel all the time for there's enterprise Kernels. While Andrew keeps complaining that the user
space that are quality going down the reporting. I think Andrew's wrong. I see every single Kernel bug that comes in
the three different districts or to Kernel.org and to two districts, gen2 and [INDISTINCT]. And over time that number reported is flat. Red Hats bugs, their bugs require for the
Kernel--is actually going down. Now, either that means we have a fixed number
of testers and people reporting problems, this can be true. But we're increasing our size of our code;
we're increasing a number of users on our code. So, obstensively, I'd like to think that we
are doing better because our code size is going in a number of bugs reported is flat. Andrew sees a lot of--a lot of bugger reports. He sees a lot of stuff and a lot of stuff
all the time. I see a lot of stuff all the time too. But if you look at our rate of change for
what's happening I would argue that we are doing better or doing good. If we break the user space API, let us know
and we'll fix it. I mean, regressions always happen. I can say that. So as far as the enterprise version, I've
said it before publicly I still think the enterprise Kernel model is wrong. I get paid by enterprise Kernel Company. I think it's wrong. Because if you look at what they do so, take
SUSE or their [INDISTINCT] SLED 10--SUSE or enterprise ten. You take 2616 Kernel froze it in time and
so, okay, here's what we're going to do. And they're fixing bugs and make sure they
pass all regression test and ship it. A year later companies come along and sort
of say, "We're going to do a service pack up date because we need to support new hardware." Problem is new hardware comes out that changes
the guts of the Kernel based on the fact that we didn't know those things are going to happen. We didn't know that ATA was going to happen. We didn't know these new devices were going
to happen. We know the wireless stack, what was going
to happen there. So you go back and you kind of shoehorn all
these stuff into there. If you look at what we have--we have I think
3000, maybe 4000 patches to 2616. It's not a 20616 Kernel anymore. It's 2616 with a bunch of bug fixes that we
know were reported to it. And that's scary because we know a lot of
bugs have been fixed between 2616 and what we have today, 26.5 that werenít reported,
that werenít fix or that we didnít know about it and we didnít [INDISTINCT] so, extensively
the Kernel, the user interface should be stable and stay the same. We donít change our version number to make
people feel good that are used to the old Unix Model, things moving very slowly. The windows model things move very slowly
that I think, "Theyíll stay with API." no, they just kind of slow API. All those Kernels--all the other operating
system, they change, they break their API, they break their driver API. I'm on the windows driver development mail
list. It's the best mailing and stuff. You just see the quotes for the Microsoft
developers. Oh, yeah, we did something change there so
we donít remember what change are we going to have to test, a lot of angry developers
all the time. So, just like users, it's just like Linux. We change our APIs all the time. They donít have any fixed API. So, for inside the Kernel, outside the Kernel
they do, hopefully. Outside of our Kernel we do. So, I still think that you should move forward. Now there's a model people that take a box,
an enterprise box, put it on it and throw in the corner and donít touch it for five
years. And then enterprise model that only gets security
fixes and backwards to that that's great but only about 20% of the market out there because
if you buy a box to shove in the corner and you're like a year from now you want to buy
a new box, the new box is new hardware, and yet this [INDISTINCT] a new hardware on that
old version of Kernel and that doesnít work, for desktop it just doesn't work. So, I still think the model isnít going to
move faster. So, inside I'm trying to argue that, [INDISTINCT]
is trying to argue that, see what works, you know, it's an interesting dilemma. So, now we're still making money so I'm happy. That's it? You guys are easy. >> MALE: So, where are you going to go next,
let take... >> HARTMAN: Where are we next? Well, what's next? Yes, well, like I say is whatever you wanted
to be. Things that I think are cool, KVM. KVM, is really, really cool. It's a running a virtual machine it's a hyper
visor, turns Linux into a hyper visor. I think it's going to win other of this and
model, two different or the VMware model. Like, what VMware does it's actually Linux
running there. So all the big embedded or big hyper visor
guys are really running Linux--cant find a watch. But so KVM School, KVM can afford the S390
main frames, port IO 64 for the arms coming and there're some other accessory on port
too. So KVM is really cool. It's a really neat technology, just new virtual
machines. So there's another way, if you have an enterprise
[INDISTINCT] did you want to run fixed, you can run on top of KVM or a virtualized machine,
you would change the hardware, you get the new drivers for your virtualized machine for
your low level stuff than your virtualized machine and you can still render old version
window just fine because at API between the two is fixed. And that's a new idea. So, you can run your applications to run your
old app--the old things, they're running on this old API especially you're C++ libraries
enterprise have a certain C++ library and it never want to change that--things like
that. I know of some big shops, Linux shops, they're
stuck using [INDISTINCT] and [INDISTINCT] three because the C++ developers do not want
to ever change the compiler. It's what those guys do as they strip out
the Kernel underneath and replace he Kernel with the latest enterprise for the latest
Kernel--old kernel and then usually above that now even knows it happens to use them,
they just know a raw metal stuff. I give [INDISTINCT]. PSP3 is coming hopefully sometime in the future
and of all without, that should be good [INDISTINCT]. Yeah. >> MALE: So, I'm wondering [INDISTINCT] has
a great [INDISTINCT] pretty dynamic thing. I was, kind of, curious, you know, like the
[INDISTINCT] on the particular subsystem they [INDISTINCT] with that and it comes to like
on particular individual. And what's your plan at certain [INDISTINCT]? >> HARTMAN: So, as the load as we... >> MALE: Or clients want to settle down. >> MALE: [INDISTINCT]. >> MALE: You know what I mean? >> HARTMAN: Yes, [INDISTINCT], resettle down. >> MALE: Settle more. >> HARTMAN: Settle more. The substance and maintenance what's our load
and as originally the--our load has gone out. I know my personal load I dropped, I donít
maintain the PCI, I mean, [INDISTINCT] more because I wasnít paying enough attention
to it. I know some of this is going to get more attention
is now taking over. So, hopefully we can or smart enough to recognize
among ourselves that when things are going wrong and we can change. We are nothing but introspective--I love talking
about this. Every Kernel, someway we always talk about
the process. On the middle is we argue about the process. As our process good or not and we changed
things all the time, we have this table tree because people like our process really isnít
covering up. Let's see how that works. The Linux next off, hey this process isnít
working some system maintainers are burning out. Let's try this. So, we're nothing if not keep trying things. So, hopefully we just keep trying things and
as soon as you're working really, really well, well better than we ever thought it would
be. We havenít hitting max limits yet so far. Every time we do a Kernel is like I was the
fastest ever. The 2625 Kernel blows away all those--those
are average number. She's a 25 as the fastest, most amazing rated
change, most developers, most individual companies and anything we've ever done. It's crazy and now, probably the next one
will be more faster. So I don't know. Hopefully--I don't know where we're going
top stop. We've already broken all the records so I
don't know. We're making up as we go. Don't feel good about Linux now. Here. Here. >> So I have a sort of strange question--I
mean, the set of layers kind of suck right now. How is it going with that? >> Set of layers of suck you, that's news. >> [INDISTINCT]
>> HARTMAN: They use more suits, then let the developers know. >> Okay. >> HARTMAN: I mean that--that's, you know,
new to me so let them know. If they don't know these problems then that's
a big thing. If you find the problem let us know that's
the best thing about our development model. You can find the person responsible for the
code and make fun of him. >> Or give me a patch. >> HARTMAN Or give me a patch. Well, that's sign off by everything. So that's really--we like it--call it the
path of blame. So you see a bug, "Hey look, all you guys
miss this, fix it." So, yes if it's using too much CPU let us
know. We'll work on it. Tashion is really good, really responsive
[INDISTINCT] really good at that. So, they're good at this stuff. All the stuff has really weird hardware. Some of the hardwares written so that it's
requiring more CPU power to do some of these things just because some of those manufacturers
are also CPU manufacturers. So, they want you to buy new CPU. I don't know. One thing about KBM it doesn't make you buy
a new CPU that's why AMD and Intel. I think itís a virtualization stuff. >> Does KBM [INDISTINCT] to [INDISTINCT]
>> HARTMAN: KBM is the--or KBM makes Linux be the hyper visor. The way it has a Kernel developer it is the
hyper visor. If you look at the way Zen model is, they
have to have to their own scheduling, their own memory management or their power management
so and then have Linux running on the side to handle the drivers because nobody wants
to write drivers. Linux is always used for drivers. Herd uses Linux for drivers. Yeah, I'll take that. So, Linux--so why not do--so Zen is having
a duplicate all the numerous stuff, all the power management never run in--on the laptop,
it will suck your power dry. So, why not just use Linux as the core, because
the core of the Linux is equal the size of the core of Zen. Actually--Linux Kernel is actually developed
by more people. Maybe they know more things. They can do more things faster so use--so
KBM models reuse Linux and then instance of the--the guest operating system runs in there's
a little driver. There's KBM driver and just runs up there
as a--it runs as a process. So, the really cool thing about that is real
time. You can now real time schedule these guest
operating systems and the real time is a real new thing. Some people can measure network latency and
latency of processes in dollars. And they can do that in Dollars and then--so,
if they can reduce that. They can make it predictable. They can throw some really, really big machines
at it and make their cost go down and make more money. So real time really matters for those guys. Real time might matter for you guys for network
latency. Other big companies would do a lot of web
services really care about the network latency they want to be able to schedule. They donít want to be waiting for another
process to come in and finish thing. They want to be able to know what's going
to happen to determine it holistically. So, anyway--so KBM can handle that. KBM can then real time schedule your guest
operating systems Zen can't do that. Zen also did their development often in the
side of the corner and never played nice with the Kernel developers. They're trying to catch up now. I don't know if they're ever going to catch
up which is sad. That's it. You guys are easy. >> Will there be a 2.7 Kernel? >> HARTMAN: Will there be a 2.7 Kernel? We said a long time ago, "Oh, if we ever do
four little page tables or if we ever replace the timers of the systems, maybe we'll need
a development branch that we're here to do this." We've done those things. The really cool thing about that development
model and the one problem of the developed models we burn engineer's time. We waste engineers. We make you break your changes up into tiny
discrete units that can be tested and obviously fit--obviously reviewed. So we make you take your, "Oh, I'm going to
rewrite the timer sub system into a lot of individuals steps over time so which you'll
never know if things go in." So, we've done these major things we thought
we're going to have to happen already and people didn't notice. Well, some of the performance guy notice four
label page tables [INDISTINCT]. [INDISTINCT] reformats some place. They fix that since then. So, I don't know. We don't need a two set. I mean, the numbers are marketing. You know, so I don't think so. Anyway I think we're running out of time. >> I just want to--let me--put the mic here. Is this mic active? Good. I just want to remind all of you that our
outgoing open source patch process is very, very low ceremony. If you got a patch right now you can ship
it out within the hour. So we're not holding you back as a company,
so... >> HARTMAN: Spelling fixes this all. >> Consider spelling fix. Anyone in this crowd can do a spelling fix
so I'll encourage you to make Linux the best spelled happening system in the world. Oh, yeah. Now, we're certainly fast when it comes to
shipping out patches. We only ask--will you look at your first couple
and that's mostly to make sure that you're not insane and not from a--not from a technology
stand point but actually from a craziness stand point, so. And after that we ended up trusting you and
that's how it works so please do it. >> Okay. Well, thank you all for having me talk. >> Not that I think that 13 is bad. I think it's pretty good for a company of
our type. >> HARTMAN: It is so. >> It's a decent place to start. >> HARTMAN: It's okay. >> But it'd be great if we were nine. >> HARTMAN: Yes. >> I could make that first slide. >> HARTMAN: Yes, everyone wants to be on that
first slide. >> Great. >> HARTMAN: Great. Well, thank you very much.
Very interesting talk, thanks for submitting!
Introducing speaker:
Someone has some sort of complex. Chill out.
Greg:
Someone has far fewer complexes.