LISA11 - Fork Yeah! The Rise and Development of illumos

I'm probably too pleased with the the title of my talk here so I'm not even going to read it aloud I'm just going to let you bask in the glory so movie alright yeah moving right along all right so illumos what what is a Lumos well of course it's an open source descendant of open Solaris which itself is a branch of Solaris Nevada which was the name of the release after scores 10 which was open but it's now closed now that's a descendant of source 2 dot X you remember was and that we changed that from 2.6 so our seven you also call it x v dot X but don't call it x four dot X because that's something complete what the hell um this is pretty complicated right and this is one of the unfortunate realities of just having a system with a long legacy and a deep history is that the history is is relevant unfortunately so to describe a Lumos and where we are today we actually need to start back at the beginning and we're not going to sort at the beginning the beginning I think we will start shortly after the end of the UNIX wars so we'll do the kind of post UNIX wars beginning of history but let's start in the early 1990s so um son ask for dot X um I assume anyone heard of some house for dot X thank you I had a very troubling moment with one of my engineers last night we were looking at this I'm talking about the presentation today he's like oh what is son us for dot X what was what was that when was that is that a a previous version of Solaris I'm like what hold on and then you I and you realize okay I'm actually I'm old that's fun I'm having one of those old moments sometimes for taxes and so thankfully I'm old but hey guess what your old do because we're old and we remember for dot X um so we did this very painful transition people remember that transition the county will here ran Solaris two one it went running sourced that's way too many hands that's a troubling number of hands no not that number of hands should never go up in any room other than like a support therapy room um - oh no did anyone run - no no - no no actually was it was never supposed to ship and I and got to very few folks 2.1 was the the the first release of the operating system that shipped but definitely did not work and fatica Lee did not work and if you were you know those of you who were around during that time remember that there was that this this huge conflict when moving from a bsd kernel to an svr for kernel and actually kind of a mutant of the two and inside Sun I don't know how many of you knew that this is going on but inside things were really compounded by the fact that the there's a source code management system that was inflicting a lot of harm called the network software environment had a very good model but a terrible terrible terrible implementation this is one of the very first spring over modify merge source control systems which was great but the thing didn't work which is very bad um and so that was that was making it even worse the engineers couldn't even they had this pile of garbage called Solaris and they couldn't even improve it because the source code management system uh Larry McCoy um from the Larry McCoy all right that's good um the I feel that Larry I don't know why I think I need to defend Larry Larry can defend himself perfectly well but I feel that that litters place in history is getting a little bit lost here um Larry played a very important role in history and that he invented this variant of NSC called NSC light on we then could actually develop the operating system NSC light ultimately began team we're team we're begat bitkeeper bitkeeper begat get so NSC light really there's a direct lineage from NSC light to get and it's kind of interesting to watch you know all these kids today discover the bring over modify merge model for the first time with git and to have to tell people that get actually do not invent these things but that's okay um it's great to actually see that so broadly now um anyway back in the day now we're talking about you know 1993 1994 on with using NSC light they could finally actually develop the operating system and get it to the point that it worked so the true old timers like Roger among them will still defend soyuz 2.3 as the finest release of the operating system only because it was the first version that worked at all on notice that the first version that worked i put that in quotes because it was actually garbage um but it was better garbage with sweaters 2.4 things slip back a bit management kind of took over the the day-to-day operations of the release once you got management making risk decisions quality almost inevitably squibs and quality dipped again with swears to four swears to five and now I assume that hands ran suarez to five so so a lot more hands go up with Suarez to five I'm with Suarez to five Soros had to get it right um there was actually a movement some people may not be aware of there was a movement inside of Sun to actually bring four dot X forward and to actually port that to SMP and actually be done with Solaris completely uh that is the kind of junk sure that Solaris was at 2.5 had to get it right forever and with that the engineers took over again in particular the that same group of engineers and that's Roger Faulkner japonic Tim Mars and Joe Kowalski Joe Kowalski sadly has since passed away on those four engineers again took over in so rs.25 and they installed bond wick as the gatekeeper this is actually a very early benevolent dictator for life model so we in software have known that this is actually the only model that ever works no one is ever been able to develop software democratically we've always had to have a dictatorship and benevolent ones are usually better um so the BD FL model existed inside of Sun as well and bond wick was I would say for life and he wasn't the dictator completely but was given enough authority to be able to back things out if they were broken engineers were making risk decisions now not management and what we ended up with was a much higher quality release in source 2 5 into 5 1 and now you had the actual first releases of the operating system that people might actually be proud of pride is something that hadn't really existed in the engineering organization for quite some period of time but now you could actually be proud of the operating system and with Soyuz 25 the engineers would never again relinquish control so management was never able to again make risk decisions because the bad old days were remembered by the engineering organization so with 2 5 we finally got it right and then we had the rise of the of the bond with youth of which I am probably the exemplars of others so what happened was in in the mid-1990s and this is horrific but true in the mid-1990s it was a foregone conclusion that eunuchs would die at the hand of Windows NT and we in the room were at our darkest hour if you remember this and I would just like to say for what history reflects that in unix's darkest hour only sons stood by it only son every other computer vendor every other computer vendor had a Windows NT strategy which was to adopt Windows NT in some important way because it was a foregone conclusion that Windows NT was going to rule the earth thank God that view is now so comical in retrospect that it's hard to believe that it was ever true it's like the Cold War you forget all that crazy we did during the Cold War it's like man cat right like walls and in World War four and all its like yeah but no no that happened that happened we were there and then this is kind of our variant of that do you guys remember even remember scaleability de goodness may 19th May 20th 1997 this is when I it's a scalability day by the way I think is that the moment at which this tips in other words up until scalability day everyone assumed that Windows NT was the future of everything Microsoft had this thing called scalability day where they're going to prove that windows was scalable because this was the one this was the one objection to Windows and they had this total sham demonstration and what was interesting is and of course we in the UNIX community knew as a sham what was interesting was that the broader world identified it as a sham at the wall street journal' identified it as a sham and the entire story became that microsoft windows actually maybe won't replace unix maybe this unix thing actually has something to it and of course the same time you had the rise of linux and the rise of x86 and fortunately thank god we did not go to this all windows route um but sun was dedicated to this vision and all UNIX vision a UNIX vision that the way to compete with Windows NT is to beat Windows NT at a time when no one else was when no other computer vendor was as a result of that we at Sun were able to attract some great OS talent myself included so I came to Sun in 1996 and well Ken great or final should be self-aggrandizing excited talent how about that interested talent but the I came to son in 1996 and we started recruiting heavily in universities for the next four or five years and we're able to bring in terrific terrific talent into the company and really motivated around a different kind of set of ideals I said this generation became known as the bond with youth and as the talent of course attracts talent eighteen was attract a tamers and we got a certain critical mass and I remember very very clearly in 1999-2000 in bond works office thank God there are so many great people here now and we've got so many people that are excited about OS innovation that you know we have got the opportunity to be to OS innovation to OS development what Xerox PARC was to computing in the 1970s of course now you could go like cue the old men from the Muppets to say like yeah you're both out of business hahahaha um yeah that's that's not funny actually um it um so yeah I guess we truly word Xerox PARC in all senses of that analogy damn it I'm supposed to sound better all right well anyway so we actually it's actually um very satisfying to me that we actually did make good on this and the reason we were able to make good on this is by 99 2000 2001 especially 2001 the the OS worked um we had fleshed it out in all the logical ways to flesh it out if you take the kind of architectural trajectory from 1990 to 1993 and you take it all the way to its conclusion you arrive at an OS that's rock solid that has got very feature full and now it's time to start some more radical ideas that to embrace more radical ideas and what had happened and we didn't really plan for this to happen it happened by accident but what had happened is so many of us had together develop the operating system for so many years that we I cut at the same time we had our own personal missions we had our own opinions about the way the system could be radical opinions that we wanted to go explore and you know it's a huge credit to son that son was the kind of company that you could go explore those opinions largely because no one was in charge um so um but the fact that no one was in charge or something that was that was great for if you were a headstrong engineer because it allowed you to go pursue these ideas and so you know speaking for for Jeffrey Banach as long as I had known Jeff he wanted to go rewrite file systems from scratch um he was all buggered up brokenness to him um he didn't understand why it couldn't be a much much simpler model one that would eliminate volume management entirely in starting in 2001 he actually started down that path with what became ZFS and we grew that into a terrific team and that ZFS of course became that vision and is is even more revolutionary I think that then Jeff conceived it to be um then for me speaking for me personally I was really sick of systems that were sucking that I couldn't understand we had these production systems that would go into the sad place and I had no idea where they were on and it was enormous ly frustrating we were having to take it we're having to modify the software load new variants of the software on there to figure out what the hell was going on if it was in the kernel that was hopeless and so on I wanted to go pursue something and had always wanted to go pursue a much more radical idea where we would actually dynamically instrument the system completely safely in production to be able to go figure out the cause of the sucking so I along with my experimented Leventhal embarked on dtrace to allow us to dynamically instrument those systems and we finished up dtrace a couple years later on and we'll describe where that ended up but the so I want to go to dtrace we wanted to go do we saw what FreeBSD had done with jails and had our own thought about about how we could actually do server consolidation and run multiple OS instances what we had seen is that the the boundary of the application had exploded in the 1990s and an app was no longer a binary but was an entire application environment and what you actually wanted was a virtual operating system not a virtual machine but a virtual operating system and that was zone zones actually born as project voir um don't name your project after after copyrighted terms this is just like a protip it's like how could you not think that like you really think we're going to call it Kevlar indefinitely like don't you realize that every time you see Kevlar it's got a TM next to it you're wonder what that is like it's called the Dow Chemical Corporation not going to allow you to so anyway we got to be like the point we're ready to ship Kevlar like oh we can't call it Kevlar it's okay you can't call it McDonald's you can't call it coke either so well anyway I'm the Welkin that's like total Sun like technology great nomenclature maybe less so marketing myself anyway the that was born as project Kevlar became Zones um and and I have to say this vision for OS virtualization was absolutely spot-on because the in with when you have a virtual OS you've got a single operating system kernel and you punch out virtual OS instances the reason that's a spot-on vision is because these guys run without performance penalty you don't have to have you don't have to have a separate operating system kernel in each of these virtual instances even though they look and feel and smell like their own machine you can reboot them you can have root on them you can if config and so on we can do that totally safely at a totally contained environment this this vision was absolutely spot-on the jail's guys got it right we wanted to take it to its logical conclusion and and set a minute much deeper into the system we did that with zones and these I know I'm not gonna go I don't go through all these technologies at length but suffice it to say that each of these a fault management architecture the service management facility where we actually elevated the the notion of from a process to an actual service and elevated that the system on fire engine where we actually were able to serve finally ten gig traffic by scaling that out across CPUs least Prive and casper dake where we're actually able to not just grant route to everybody and so on and these ideas were you know I've kind of on a spectrum of how revolutionary they were how radical they were but they were all for us major changes in the system and we were able to pursue them because we had gotten the system to the point where it worked and what we had actually I remember very vividly the meeting because he again at Sun as I've often said son had a political model like Somalia right so his son was kind of run by a feuding bands of warm that's kind of the UM so if you're wondering like why no one was in charge the sons like why is no one in charge of Somalia it's like I'm your business Somalia because and yes it's sad like you try to get a food convoy through and gets you know shot up or whatever um but you know Frank it was good to be a warlord so that's kind of a boils down to and but the because no one was in charge I actually these projects were all very very well underway before kind of higher level management actually understood what was going on um and I remember vividly the meeting where we had with our with our VP at the time and explaining all this stuff it's like wow like look what I won this is amazing like this is what you guys been up to huh it's like yeah dude like that's what we've been up to so this is what happens like every once in a while like when no one's in charge good things can happen um so he of course was very excited and to a huge credit the sons management they did actually see the value and all this stuff on and we these were all coming together in the same revolutionary version the operating system so RS 10 so where do these ideas come from and this is kind of a you know an important point is that these in this kind of the point that I've made but just to really put a sharp point on these technologies did not come from management they did not come from marketeers they did not come from from some what they not come from you know table stakes on a requirements document these technologies came from engineers from individuals and the projects very much reflect the people behind them if you know me you see me in DTrace if you know bond wick you see bond work and ZFS all over the place and Mike and Adam and Sanne means that you see all of us together did you see who we are our opinions about the way the system should be and those opinions by the way were born from the the kiln of unspeakable pain namely trying to actually make systems that work so the system's came out of experience and and frustration and rage and sadness and despair that's where these opinions came from and but they are people that did them organizations don't innovate people do you cannot take an innovative organization remove the people replace them with different and get innovation that's just not the way it works and I've had to explain this to several and senior management over the years but it this is an important point that organizations don't innovate is the people that innovate and the people will become important in a moment foreshadow it's foreshadowed by the way so that's the bold serif or okay so how about open source well on what was kind of going on is that Linux and x86 together this is this disruptive change that was thank goodness was dispelling the notion that or doing away with the notion that Windows was going to dominate with with the rise of Linux and x86 the cost of acquiring the operating system was driven to zero and this had effectively already been true anyway the OS for Sun had always been a loss leader I mean it was true that I mean I didn't run a 10k remember those the 10k you run those and if you actually looked at your at your Pio you looked at the cost of goods for for your II 10k it actually had a line item on there for the operating system and there actually was some stupid number on there um but the reality was that was just like that was just kind of funny money going on behind the scenes reality was you paid for that operating system with the box itself um so the OS for Sun had always been a loss leader we made money from hardware we met money from support services and so on so for us it actually made a lot of sense to effectively give away the operating system but we wanted to do that in a way that was was convincing effectively um and we also wanted to run all these things that we we had innovated and bended over the years to actually uh transcend product and become advances in the state-of-the-art why did Sun want to do this because we were the open systems company remember that Sun is the company that not only invented NFS but then itself ported NFS to each of its competitors platforms an entirely radical idea when it was done in the late eighties so for son it was totally natural that we would open source the system so I mean I think this is an important point because people thought that son did not want to open source the system son very much wanted to open source the system the very first executive level I would say mandate but interest a direction I would say to do this I remember being in 1990 eight so um and yes we can play the power game like oh if we'd only open-source in 1998 um we wanted to open-source the system so why didn't we what was the problem well the problem was that it's a deeply proprietary system it's got deeply proprietary roots and no one foresaw in you know 1991 that we would want to open-source the system ten years later so we signed a bunch of agreements that basically made it impossible to do so and we had all these little bits of technology and by the way it's never the awesome stuff so is the crap right so it's always like you look at like the the internationalization support in lib see like something that is not exactly like oh that code that's awesome I love that code oh the internationalization code oh you read that it's just divine but no no it's not it's garbage it's just those garbage that you've got to actually write and of course because it was garbage no one actually want to write it we've gotten a contract and gotta do it and we signed an agreement that basically prevented us from open sourcing it and you go back to all these and you know it's it is actually a tribute to two suns management that there was enough interest in this in particular with Jonathan making this a I mean Jonathan other faults aside Jonathan did give humanity the gift of Solaris by making this happen in 2004 forcing all this renegotiation some of it sadly you couldn't actually find the entity that owned it so you go do all this renegotiation and there's still some bits that you can't open source which is and we'll talk about that in just a second but it was really frustrating so um we we went through and we did all this work all this very heavy lift not technical lift but legal lift to actually be able to open-source the system and we did it and finally in June of 2005 I was actually very proud of the fact that the the first thing we open sourced was dtrace but the rest the OS and followed very shortly thereafter in June 2005 um and something that we internally were incredibly excited about because we were excited to have these innovations out more broadly we were excited to have community involvement in the development of Solaris um so that was it was great and from that point on the OS was open so that point on all the changes that went into the operating system were open sourced effective as they went in so it was a huge huge change for us and again it's very difficult to leap this proprietary chasm I mean how many examples can you think of it there are very very very few right I can think of the there's a Firefox database that used to be ingress I can think of a handful of these guys but there are very few right you're never going to see open VMs be open source even though that would actually be a great thing for HP to go do you're not you're never going to see it because it requires so much work that so much non-technical work to make happen and unfortunately even with everything we had done there were some components that we simply could not open source Lib C internationalization being among them which was very frustrating to us because it left some bits proprietary and at the time years like well we did the best we could um the other thing we had to do is we had to allow for proprietary drivers so we needed to develop a not a an open-source license that would allow that on we really liked what we saw in the Mozilla Public License um but the Mozilla Public License had some kind of bugs at it actually for lack of a better word the MS old public license in particular requires that all disputes be resolved in Santa Clara California which like that seems a bit much to kind of claim jurisdiction in a license that you think to kind of enshrine Santa Clara California is like the mecca for this particular license you can go to annual pilgrimage there to go litigate seems a little silly um so we got rid of that and we just cleaned it up effectively and got it oh and we it was approved by OSI in fact OSI was like wow this is a great license because it's a license that is a file-based copyleft license so in other words you can add proprietary works to it and create a larger work and you don't have to eat those proprietary bits can stay proprietary so it's got that BSD aspect to it that we really like I love that about BSD that that BSD is and this is like Godwin's law for for open-source is that that every presentation open-source ends in a licensing discussion so I'm I shouldn't be accelerating this by say by saying this but I love that about BSD on and with GPL I think it's more of a challenge but we couldn't do GPL because GPL left this so ill-defined we really needed to be file based copy left we thought it was all the advantages of GPL that we liked all the advantages of BSD so contrary to and there were in particular Deniz Cooper made this public claim it was very damaging and false that that we done this to be deliberately GPL incompatible that is not the case synthetic Li not the case we did this because we actually had to allow for proprietary drivers that was it was not tenable for us not to allow for proprietary drivers from partners like EMC and so on on Veritas etc so we had to actually allow for private area drivers and we felt that the cuddle was the right way to do it so we felt we did it right with the license but we left some bits proprietary and the proprietary bit stuck well we've done that we did the best we could and you know what better can you do but the proprietary bits really were a problem and they were a problem for a something of a subtle reason they were a problem because on the one hand all of the juicy bits of the operating system were open sourced but on the other hand you couldn't actually go fork the whole thing you couldn't fork me on github for opensolaris and at the time you know we didn't really think about that as being debilitating because you actually get the right to fork it you could go take all this all of the coddled stuff and go build something else on it but because of these proprietary bits like the internationalisation code like also odd OD tail like there's this random collection of like there are four utilities and five utilities that we also couldn't open source probably cuz we had some jackass contractor to write at some point and couldn't go renegotiate that contract um so the it was a sliver of the system but it's like you kind of want awed right like Li Cody you know it's like let's let's not forget OD you know help me give me run OD like you know we're know do you once a month at least exactly let's not forget OD all this kind of fancy stuff it'll forgot about although I'll deed OD there um so you know you need these guys the little workhorses so you can't not have those there were very small number of utilities that was not the problem the real problem was yet but the the like the Lib C internationalisation and then there was some drivers a small number of drivers but drivers on the last weekend open source mostly drivers from the non Sun hardware so um the reason that was a problem and uh you know to our defense we were operating in our time which was 2005 and I think we were looking at 10 it's for open source and at that time the template for open source was Apache really and we kind of modeled ourselves on Apache I think we now know that that's a dated model the Apache model I actually think is the wrong model for open source and the reason it's the wrong model for open source is it's very it's força phobic and when you're fork a phobic the problem is that lots of badness follows on from being afraid of a fork if you don't want to have Forks if everyone must be on the same piece of paper that hey guess what's next governance discussion yay and we're gonna vote soon we're gonna have we're gonna have boards and elections and it's like all this experimental democracy what the hell we don't want to do any of this stuff this is not about developing software this is about arguing with one another like I don't need Paulo I get politics in the real world I don't need it in software there's never Neverland over here I don't want to have to be doing politics over here um it but the problem is that when people don't have the power to fork you end up in these political discussions you end up in these governance discussions and for us where it wasn't even about like we didn't want you to fork but like you actually kind of couldn't you couldn't because there were these little proprietary bits and then you couldn't also because the reality was son unfortunately had a monopoly on Solaris engineering talent there were a couple of folks in the community um Robert mikowski million in the room there yo Robert there are a couple folks in the community who were able to wade in and actually make meaningful changes to the operating system but we're talking about like 10 people I mean maybe so the reality was most of us still work for son and it meant that that you didn't actually have the power to fork the operating system so the problem is and then things were made kind of even worse because the you couldn't have the power to fork and then if you want your code to be inside of what son shipped you had to assign copyright to us now that makes sense and a lot of open source projects do this corporate-sponsored open source projects do force you to assign copyright they do force you to sign copyright for some good reasons um but we'll see in a second that that copyright assignment became death and became a real problem and I think actually copyright assignment is something you actually take as a as an industry we need to take very seriously when you sign yourself away you know you sign that copyright agreement to Google or whomever else be sure you look at what you're signing you understand what you're signing because what you may find is that your contribution is not used in the spirit that you think it should be used so it is something to be very aware of and sadly we have an example of how things can go wrong so the problem with the kind of upshot of all of this is that open-source was kind of this little sock puppet over here I could do like a little open sore I can really have a little sock puppet I could talk to it how are you open-source I'm fine good have a vote on whether you should take it okay Oh favor aye aye aye yay um so um yeah you didn't think yet the performance art but you know it's um and that was just kind of sad and and I think the worst thing is that over time it just became too tempting to kind of go pull these little strings here um and in particular the strings were yanked with what we came to cut I came to call were we at the team house on a fish works came to call the open Sawyer's Missile Crisis sorry it's probably the Cuban Missile Crisis is definitely a bigger deal than this okay so I just don't want to I don't want anyone to think of making light of the Cuban surprises but we open Solaris Missile Crisis happened in the fall of 2007 so this is now we've open-sourced it in 2005 and now this is two years later fall of 2007 and son decides let's create a new open source base distribution what should we call it what should we call it I know let's call it open Solaris boy that name was right in front of us all along it's like yeah that's going to be kind of confusing because now people don't know whether open Solaris is this community that is based around a kernel or is it actually this concrete distribution I mean if a boon to of canonical decided to call a boon to Linux meaning of course is Linux but it's like no we're just going to call it Linux well that's going to create some natural confusion from the Debian folks and from Red Hat and from others right so this actually created not just confusion but people were upset about this they were very rightfully upset about this and the opensolaris governing board is like a what that hell what's going on and these kind of the the cruisers were kind of chugging into one another because we were kind of approaching this big vote at the OGB and of course is this huge government do and motions and counter motions and parliamentary procedures like this huge governance orgy I'm not going to act that one out by the way I could have liked the sock puppets go act that out but we're not going to do that um that's the nc-17 version of the opensolaris sock puppet show um it was real it was awful and actually I made the huge mistake of when I was little remembering this I want to make sure I got the dates right and so on so I went back and looked at some of the the mail trails about this like oh my god it's like you look at the number of words spend on this words from bright people that can actually write code instead of writing this and they're writing just this huge screens and so on it was just sad very very sad and a huge waste of time and it was entirely avoidable on the one hand understandable on the other because those strings were just too tempting for son to pull um and sadly on the what this ultimately ended up doing is really deflating the community because it was clear that son couldn't tolerate in independent OGB and I don't think this is necessarily a problem with son I actually think that sons intentions were as good as a companies could possibly be it's just a company has its own interests and it's very hard when that company has created this open-source community from scratch populated it effectively from scratch it's very hard not for it to think of that community as belonging to it and that's not the way the community thinks um so this is a really this is a challenge when you have when you're leaving this proprietary chasm but the community was very deflated by this and if you get the OGB composition over the years it became more and more Sun dominated and open Solaris kind of went into the doldrums for the next three years right so the open so air has kind of drifted which was unfortunate so then of course on the end of an era I'm gonna I'm gonna try to make it through this slide without crying um so son was bought by Oracle in 2009 on the acquisition finally closed in February of 2010 and Scott eulogize son very concisely um kicked butt had fun didn't cheat loved our customers changed computing forever and um you know um it makes me very proud to have worked for a company for whom that is complete accurate um we should all be so lucky as to have that be our epitaph that is all I want out of life that's it that my family that's it I want a kick butt I want to have fun I don't want to cheat I want to love my customers and I want to change computing and I think that's what all of us want um so that was son that's not a work uh uh yeah yeah set that again this is not Oracle and you know um I stayed with Oracle through the acquisition I went in actually with a what for me is now in retrospect a very surprisingly open mind but those who are talking at the time will testify that I actually had an open mind um when we went in to it the reason it was kind of silly to have an open mind is because you actually don't need to be open minded about Oracle you are wasting the openness of your mind go be open-minded about lots of other things um I mean let's face it it's work to be open minded right I mean it's like you got out constantly like discard data and be like okay I need to be open minded about this it's like no the Oracle just be closed mind that it's a lot easier and it because the thing about Oracle and this is just amazing to me is you know what makes life worth living is the fact that stereotypes aren't true in general right that it's the complexity of life and as you you know as you gives you know people as you learn about things you realize that these generalizations we have virtually to a generalization or false well except for this one as it turns out I'm what you think of Oracle is even truer than you think it is there has been no entity in human history with less complexity or nuance to it than Oracle and I gotta say as someone who has seen that complexity from my entire life it's very hard to get used to that idea it's like surely this is more complicated and it's like wow this is really simple this actually this company is very straightforward in its defense this company is about one man his alter ego and what he wants to inflict upon humanity that's it um so yeah kick but have fun not cheating love your customers changing computing forever it's more like you know ship mediocrity inflict misery lie our asses off screw our customers and make a whole shitload of money um yeah and and by the way not to put too sharp a point on this but I am and you could oh yeah yeah I'm holding back here I'm holding back oh there are things I'm not saying um but actually you know if you were to ask Oracle so you know Oracle what are you about literally what do you about why Oracle tell me about Oracle Oh make money okay yeah but like when I make money make money make money that's what we do it mate make money okay so you sit down you have that conversation if you would actually have a conversation with a bank bringing back use it down with Jamie Dimon from JP MC and you say why JP MC why Morgan Stanley why goldman sachs whether you believe it or not you will get a very societally engaged answer Oh word you know we're the lubricant of innovation and we make dreams happen and death together that's all you hear from a bank a freaking Bank you talk to Oregon's like nobody make dreams happen we make money that's what we do it's like hey you know actually funny story I just talked to the bank and the bank gave me like this really flowery answer man it's like do you want like revisit that answers like no why all right oh Jesus and you actually it's funny when I actually looked at the the philanthropic work of Larry Ellison this is the kind of things that occurred to you after you work for Oracle for a while like wait what is the nuance this guy so you know do lithography Larry Ellison literally else has been involved with the two philanthropic organizations first he made a 500 or 300 million dollar donation at Stanford oh that sounds good I'm not sure they need it but okay that sounds good three million Stanford in exchange for not admitting wrongdoing in an options backdating scandal you stay classy you stay classy okay so there's that and then and this I this is not a joke literally all other philanthropic work is to the Larry Ellis Institute for the prolonging of life namely his it's like oh my god and so he has a team of scientists Edie I'm not joking this is a non-profit this is a 501c3 is a non by a team of scientists that are dedicated to punitive Lee long life nearly hits I ain't just crazy it's crazy to me it's crazy and it's we can make more money it's like what is it about what is it about stop you're giving me an existential crisis please be about something more than this ah anyway sorry Hey it just it's it's it is crazy it's it's crazy um that it's so simple but it is that simple and and it became great and by the way and I've already fallen in this trap here do not fall into the trap of anthropomorphizing Larry Ellison because of you and you need to think of Larry Ellison the way you think of a lawnmower you don't have the motorized wheel on board a lot more just doesn't like mows the lawn like you stick your hand in there or chop it off the end you don't you don't think I go the lawn mower hates me Laura don't give a about your order can't hate you lawn mower you don't anthropomorphize the lawnmower don't fall into a trap at Oracle so in in particular with open source oh you wanted to kill open Solaris like no the lawnmower doesn't care about open Solaris the the lawnmower doesn't think about open source the lawnmower can't care about open Solaris the lawnmower can't have empathy um so it became very clear the all these kind of truisms became very clear and it became very clear that again it's other day they didn't have interest in opensolaris it's that they can't the brain doesn't work that way and so an in particular and this was what was surprising look I get it yeah they can't care ok I get it again again but what was amazing is there was actually movement inside Oracle to actually close the system wait what we need close whoa we did all of this work what are you we're going to close in there you came to that that's too stupid there's no reason to do that it doesn't cost you anything to keep it open that would just be an incredibly and I personally made these arguments inside of Oracle but again you're making the argument of the lawnmower so you know um anyway what was happening outside of Oracle is that a new thing was being born illumos and so starting in the summer of 2010 um Garrett Nomura and eggs and I along with rich Lowe Jason King and some others began the process of rewriting those little closed bits on the tail and and more importantly internationalisation supporting Lib C and so on they were either writing from scratch they were pulling them in from BSD Thank You BSD on which was it BSD served a really important facility here in terms of allowing us to get past this stuff and by early August he actually had an entire open operating system booting dubbed it a Lumos from a luminaire the Latin for illuminate kind of a pun on Sun it was made available on August 3rd 2010 and in particular the thing I loved about this announcement is the announcement was code and a demo it's like here there's where the codes available here's a demo of it working we're not going to talk about what we're going to do we're not going to have votes we're not gonna have constitutional amendments we're not gonna do rank choice voting we're deeds like there it is um and it was actually not designed to be a fork it was designed to actually be an entirely open downstream repository of open source it's actually funny to go through these slides August 3rd 2010 it slides like because there's no way Oracle would be so stupid as to close the operating system so like that's obviously going to stay open and then we're just going to be dis entirely open we're not going to be a fork when we're going to be entirely open which I actually had misgivings about because I didn't necessarily want us to have to merge what was going on upstream when we couldn't control it but that that was what it was well this is actually resolved for us and it was resolved for us in what I think is the one of the most shameful moments in the history of open-source uh and that is the this Friday the 13th memo on Friday August 13 2010 this memo was sent out on to folks internally at Sun and in particular it in this memo we will no longer distribute source code for the entirety of the open source operating system operating system in real time um this is it is absolutely shameful um and it is shameful because for so many who worked so hard to open up the system and not just for the folks inside of Sun the reason that this is shameful the way the reason that this is reprehensible is because a social contract was formed with a community and there are folks putting folks in this room that had source code that was contributed back under that copyright agreement under that copyrighted assignment and that source code was now being made proprietary that is reprehensible that is in the pool of open source and it is disgusting corporate behavior and sadly it is behavior like this forces the rest of us to need to be cynical and suspicious this is a this is a body blow for open-source um and the worst thing was not only was it shameful it was cowardly because this was never publicly announced Oracle has not publicly announced once once that they are stopping contributions to open Solaris they simply silently stopped now I get the lawnmower doesn't understand why that's not a big why that's a big deal ok I get that but you know what we're not all lawn mowers and that's disgusting it's cowardly behavior and I I have I have never been so embarrassed of my former colleagues than I was when I saw that male humiliated um and as it turns out it was a lie on its own it also says in there Oh in this manner with the following full releases of our enterprise Oh operating system we're going to open we're going to have a the source license codes can be made available yeah well source 11 will ship on November 9 2011 and I don't see the goddamn source code so now admittedly in Oracle's defense it lie to us because they didn't tell us anything right they lied to themselves now iodide actually already left Oracle at this time so the thing that unfortunate makes me the one thing that does make me I have regret about is because I didn't work at Oracle I couldn't quit over this um which kind of sucks this this happens this is one of it this is one of those Oracle conundrums you decide to leave and then you realize well go I can only pick one thing to quit over I can't kind of quit all I can't quit over the stuff that happened after I decided to quit that's kind of dishonest so anyway I would have quit over it but I couldn't a lot of other people did though and there were a lot of other people who when this happened they didn't understand I think that they knew this is a big deal there was only after days and weeks past they realize how big of a deal it was um and and how how really reprehensible this act was and what it meant for the future of the operating system inside of Oracle um my prediction to Oracle management when I was at Oracle was if you do this within 90 days you will lose the core of the ZFS team that's a guarantee that you will lose the core of the ZFS team will walk within 90 days turns out they Walker than 45 um within 90 days of doing this the entire dtrace team had left Oracle all primary veterans of ZFS have left Oracle primary engineers for zones of left Oracle networking left Oracle Oracle massive inflicted wound would love for this to come up on a Wall Street call by the way massive self-inflicted wound by Oracle when they did this major diaspora outside of Oracle bad news for Oracle good news for illumise because all these guys landed a little in illumise and it should be no surprise that again those innovations happen from individuals it should be no surprise that with those individuals outside of Oracle's walls they have continued to innovate and we've got a lot of it now this was a year ago now right this is over a year ago and the great thing is that a year later we can actually be concrete about this a year ago I would have told you this is going to happen a year ago I would have said with these people gone with for example a guess to the next slide with the key is the FS engineer is gone there will be ZFS innovations that happen outside of Oracle now year later we can actually point to those so I'm going to go through those quickly be mindful of time but you want to get through those and before I do I want to do it in the context of really a beautiful grand irony and that irony is this um there is no copyright assignment in illumise so when you contribute to a limos you're not assigning copyright oh you hold copyright yourself on your code like a lot of open source projects copier it means with you but with the the closing of Solaris um Oracle has a band in the cuddle and the only reason it can do that is because it was assigned copyright on these community contributions however it does not own copyright on these innovations in a limos beautifully Oracle can't take them back and this is open source licensing um working I have to say this is you know we prepare for doomsday scenarios in open source licensing we had a doomsday scenario and it is actually working and it's actually fun to say I have to say that the that Oracle now if Oracle wants to take them back that's great open source the system and then you can that's a beautiful beautiful thing that's that's a righteous thing uh I understand the lawnmower doesn't understand why but it's like just right just um and so in in illumos we've seen these critical innovations it's important understand these innovations will never be in Oracle suarez so don't sit around waiting for them be an Oracle Solaris they will not be there illumos has become the repo of record for these technologies so let's all go see FS first so the cords the offense EFS engineers are now in the illumos community on bond work is actually the deep stealth startup I can talk about he's doing but um and Bill Moore but Matt Aaron's our truck tort Wilson out of Leventhal but we have ski Brent and Greg are now active in the illumise community of all these folks I don't think you probably haven't met Aaron's of the co-inventor of ZFS Eric's Rock was very public about wall he didn't ZFS George Wilson assumed you probably haven't heard of George Wilson is actually the deathblow to Oracle Solaris because because George Wilson was the guy that you called at your darkest hour with a ZFS problem um George debugged countless numbers of production ZFS problems it is largely due to George and these other guys as well I don't to minimize their contributions but these guys would agree with me that it is largely due to George that ZFS is where it is which is an open source enterprise great operating system we in humanity owe a debt to George and it was great to see him not just leave Oracle but actually land in the illumos community and actually developed in illumise so um the ZFS engineers and illumos have been working with their freebsd brethren um and other folks and have formed a joint working group and they've got a the first concrete artifact of that is a versioning feature that allows us to have individual flags in ZFS and allows us to innovate ZFS in ways that are orthogonal this should speak volumes this is the first thing they did right the first thing they did is like oh crap so many of us have so many great ideas that the first thing we actually need to do is solve a way to integrate these ideas asynchronously okay so like you should take that for what it is that's that is a lot of companies making a big bet on a limos and a big bet on CFS concrete things Matt Aaron's added the ref ratio property you can understand how much you win compression egg is buying you Matt then added the ability to get estimated progress on a ZFS and receive that is a hard friggin problem and this is a problem action you think we would ever see solvency FS because it is so hard fortunately this is the great thing is matt is now at a company that really desperately needs this and because matt is that a company that needs this he's right on the coal face of it so he invented it uh matt actually invented ZFS n receive when we sent him to china and he needed ZFS n receive in order to do remote development so the lesson from this is perform human experimentation on matt and great things happen um bill PFC and jerry Jelinek both work for for joint added zones base i/o throttling which allows us to actually make progress on a very hard problem that frankly no one else has really made progress on in many many years and that is how do you actually in light a multi-tenant say that it's a cloud computing how in light of multi-tenancy demands do you actually deliver throughput on how do you minimize pathology great work that's also in illumos I'm Dan McDonald's in the room along with speak Gupta input an unmapped for STM at the scarcely target management facility allowing you to better see f s back I scuzzy luns these are features that Oracle Solaris would or should one again they can't have them um not until the open source don't do they play nice there's a the thing is this is the tip of the iceberg there's a ton more in the pipe the feature flag is becoming that I mentioned background just destroyed resumable resumable send which is going to be huge absolutely huge um in DTrace it's been great for me personally so obviously I've been very continuing very active dtrace I've been more core dtrace work in the last year than probably the previous three it's been a lot of fun I added the log-linear quantization action ll quantize umm I added KTM support we'll talk about that in just second with the vm regs variable did a bunch of little stuff added tray spam to take a variable size to upper and lower I had an L to stir to take an optional base um the added the us CT provider reaping which is a long time issue you don't if you don't know about it you don't care but if you do know you're probably weeping with joy that we actually done it um my favorite one though is Eric's rock adding the CTF aware print action yeah which is absolutely awesome I'm actually on a machine here so I can do this is a really recent these joint bits will board downstream of illumos and we've merged with with schrock's changes here so I can do a dtrace - n let's do ZFS read entry and I'm going to just print star of args 0 that's a V node pointer and with that print action and hopefully this thing will oblige by actually do it there we get it'll read there and so now you get m DB colon colon print like output on every operation which is just huge so it's a simple thing on the one hand I mean this is not the truck was able to do this in basically a day of work but it was a hard day of work and then it would had to be done by the right person and it's just huge when you want to actually go debug a kernel subsystem for the huge first time or use a whole subsystem for the first time so this was absolutely terrific terrific work and it's I think it embodies a luminous I got to say because it is useful it is innovative it is exciting and it is open and so this is really I think embodies the kind of work we've done it's a quality of life feature but it's a great quality of life feature on with zones Jared Yannick one of the primary zones engineers very active in the illumos community our focus has been on high tendency systems I can absolutely guarantee you the highest levels of tendency that are running with the the zones code base without heritage i'm including solaris in that is running on illumos running in production on a little bus we run much higher levels of tendency than anyone else in the world um and they were able to do it because of a lot of the work that Jerry has done all in the open we also added the SVC s how you use the SVC s command alright just enough of you for me to show this to you um so SVC s is a command that shows you the services in the system okay and on the system I have some zones okay all right um now I can also do a CCS - Capital Z which oh yeah oh yeah now and I got to say this is just an example of where as I aren't I implanted this I want to join horn but the N part of using did this is like oh I didn't do this before like we can go just go do this we should go do it and it's like we do this like damn this is useful and you can do stuff like - Capital Z X shows you all services in maintenance across all zones it's useful as it turns out don't wait for that in Oracle Solaris though I can also take I can do minus Z for a particular zone and now I'm just getting all the services from there I also as long as I'm over here man I got to show you this one if you use s VCS this is just one of the SMF is great but it's complicated and everyone has complicated feelings for SMF you have to almost so if you want to get the log for this thing you've got to use you an SEC s minus X and you're kind of cutting and pasting the log if you've ever used s these yes you've done this many many times it's like why didn't we just invent a goddamn oh sorry why didn't we just invent the ability to get the log so I can just do this ah ah I just feel like I passed when I did this I felt like I had passed the largest school of my life even though you know and this is like this is so stupid this is so painfully stupid um but it's really really useful and then you can do things like this so I can actually do um so - LZ is every log file every zone and now I can do this so this is getting me the log files that were touched 10 the 10 most recent log files that were touched across the entire system god I love you UNIX makes me cry UNIX philosophy oh god small tools doing well-defined things oh um so um but again it's this is a silly example but it's one that I think embodies the system that the observers - cap L + buttons E and lo cases as well that we're doing useful stuff this is stuff that we need to do what we're doing trying to do it this is not a 500-person engineering organization that's in a giant echo chamber doing what it thinks people want that are actually not what people want um and I I know I'm running shine on time here so I just want to quickly run through we did KBM on this is the kernel virtual machine allows you to run non illumos guests on top of a Lumos by taking the KVM technology from linux porting it put that in air quotes because it was very limits Pacific's way there were a lot of lift departure Lumos this was done by Robert Misaki max burning and myself at joint um and we put that out there on in August of this year we're running this extensively in production at joint on and you can now boot Linux and FreeBSD and Windows and everything else as guests as harbor virtualized guests now running at newly bare metal speed huge kudos to both the guys at Red Hat Qumran it that did KVM and of course to Intel the virtualization support actually is unbelievably good I'm crazy good so very excited to have done that work and then of course why would we do this what are the advantage of doing this Vander Savannah Lumos based KVM over the leg space gave you very simple CFS dtrace and zones ZFS when we went to KVM forum and about two-thirds of the presentations were effectively filesystem presentations and there is a filesystem crisis right now going on in linux and KVM and it is I have to say it's Clown College in terms of yeah cake out to yachts technical term cake how to and and the the butter of s and all this garbage it is absolute clown college um and any I mean there's anyway I'm done with that um the but suffice it to say huge advantage you can ZFS clone you can now ZFS clone and provision and so you can now provision and you will spend much longer booting your instance than you will actually provisioning it which is huge um d trace along dtrace up into k vm zones we actually run k vm and taught inside of his own we did that because it was we needed to use io throttling and stuff like that that was keying off the zone so we didn't do that for security and then and I swear we didn't like pay this guy to do this there was a black hat presentation at Def Con about a guy who had broken out of key mu into the the larger system because he could jailbreak out of chemo well if you jailbreak out of key mu an arcade young application you just find yourself in a zone it's like congratulations you've now burrowed out into a much more secure cell um so um and actually and each of these is an advantage for different folks so where we and when we did this I was definitely excited about it to do gave um it was a lot of work I was a great excited about it I didn't know the rest of world was gonna think it's been actually really interesting see that actually lots of other people are excited about it lots of other people see these advantages um there are a bunch of distributions that are out there open Indiana's got IDs packaging smart OS um it is a that's a joint distribution from us that features net bsd packaging we design for cloud computing a lumia ms a noon extended distribution that's coming up that's got debian packaging the important thing is that these distributions come on each other um we have we believe in the freedom to fork so don't think that there's just one choice here and by the way if like if there's a distribution that you want that's not there go invent it it's great we were excited for people to use the operating system in a new way and we view them really as growing the illumos pie much more than dividing our values are we value the freedom to fork first and foremost freedom to fork over governance code over discussion innovation over democracy innovation democracy not as it turns out compatible with one another and we believe in innovation first i'm actually a huge hat tip to Dan McDonald's in the room I love this quote from David Clarke to IETF Fame we reject kings president voting we believe in rough consensus and running code and in a Lumos instead of the devil dictated for life we have a benevolent oligarchy which is actually we always had in the operating system in that benevolent oligarchy where you have different experts for different subdomains of the system that effectively control what's going on in that system that Al Gore key has always been able to achieve very quick consensus we value utility we do not do things for their own for their own sake and if you look at the illumos hackathon out of them tile organize this you when Adam was organizing this I'm gonna like Adam you think this is going to be a big deal like you think I'll do it and we'll see how it goes and I didn't I have to say I and I've embarrassed to say this and I don't think I've ever said this to you personally so you're hearing it for the first time I didn't think this is going to work I I just like who's going to hack on an operating system it's complicated you know it's like let's go away let's go hack on a 747 800 it's like wool that's gonna you don't go do that actually um I crash if you do that but no as it turns out this was a resounding success and that D try sprint action came out of that hackathon so um boy we even it was great to see that happen very excited about that going forward Dan's already integrate some stuff that came out that so we're going to lots more hackathons in the future and I'm definitely a convert and I actually can't wait to go go to the next one and hack myself I can make it last one if you want to get involved definitely come to the illumos Boff tonight I can't make it unfortunately I've got to be back in San Francisco for tomorrow morning um but there are about Albert Lee's here at Gordon Ross is here the guys are in the room I assume Gordon's fixing a bug right now very good on the totally understandably but they um so I would say come to the Boff tonight there'll be some of the most folks there on and they can help answer your questions start playing around with the distro that suits you if it's not one that suits you make your own play around the code you can go to github or source of illumise that org if you have questions or they're the list go to IRC and if you don't know where to go you can always tweet Deirdre who I actually think is actually just a bot even though she actually works for me she does even reply to tweets 24/7 365 um Deirdre can always help you out and point you in the right direction um so with that I'd say thank you to him I just want to call out a couple thank-yous Garret d'amour conceived of this we owe a huge debt of gratitude to Garrett Hebert that he's led the communities on a great job and rich Lowe has done so much work rich is actually done I haven't told Shrock this because he's going to fly in to a rage of productivity rich Lowe has done more integrations to a Lumos than Schrock did in the previous year to actually Solaris so that's which is amazing rich is incredibly productive the openindiana guys have been huge on Josh Willesden Johnson and Sean and Brotman stalky I joined that done great work on the smartest distribution and Adam for for organizing the illumos hackathon so with that thank you very much and I'll take questions so no we are two minutes over so probably keep the questions I would say exceedingly brief but I would love to take your questions if there are any a one question yes you're Larry's attorney I represent Lawrence Ellison Oh God illumos okay game is on solaris okay it's a very important distinction yeah I mean it's actually important section yeah I I know I'm sorry mom so with the trace you can now observe this KVM VMs from the outside I guess that's right what's the state of for example things like DTrace which i think is one of the most fantastic tools on lizard or gentleman and ask our on bsd on other operating systems so what's the state of your collaboration with other Linux or I don't know OS 10 I know already includes DTrace yes it's a DTrace Linux has not included DTrace primarily for licensing / religious issues so there's not really and we there are folks reporting it to to Linux we collaborate with those guys Paul Fox in particular with the deal with the BSD folks we have have a very good relationship with them sadly the the the chief the person did all the work on that John Burrell passed away so that was a real blow to us all but so but with those folks are still working on it definitely collaborate with them the freebsd implementation is not where the illumos implementation dtrace is just because they are not able to do all the user land stuff as well they recently push that as well push some of the user bubble stuff and I mean I think they're getting there right now it's still true the most complete port of eat rice is to the Mac with whom we've got a great relationship so um you know what we love to collaborate with those guys and we'll continue to collaborate with those guys and I mean all move forward as a dtrace community and kind of separate from the limos and are your recent innovations in the trace flowing towards apple toreano right yeah I think that we caught them a bit flat-footed because dtrace was quiet for a while because and now we kind of got this big uptick and yeah I would expect them to take all that stuff they've been actually they've been extremely good about taking that so and it will all merge very cleanly and we've made it very clean how to merge it so I would expect them to take all that stuff cool thank you thank you any other questions okay thank you very much you

Channel: USENIX

Views: 200,981

Rating: 4.9504337 out of 5

Keywords: LISA '11, USENIX

Id: -zRN7XLCRhc

Length: 64min 3sec (3843 seconds)

Published: Thu Dec 15 2011

Reddit Comments

[u/tetroxid]

Don't anthropomorphise the lawnmower.

[u/remind_me_later]

At least with other companies, they give the façade of "making the world a better place". Oracle......not so much.

[u/beginner_]

It's puzzling why anyone would buy into any oracle product nowadays. I understand the need for it for legacy reasons but as far as I can see most companies are pushing to ditch anything oracle. That's the issue with wanting to make too much money, then it actually becomes worthwhile to do huge migrations to a new system.

[u/VincentxH]

I'm somehow triggered by posts like this, that I perceive as naive. To draw an analogy: the most successful open source projects are generally run by benevolent dictators. Quite often dictators are not that benevolent, and benevolence can also be in the eye of the beholder.

[u/YumiYumiYumi]

On an unrelated note, Oracle's free cloud service is surprisingly decent. I wouldn't call it great, but considering that AWS and GCP's free offer is largely a joke, Oracle's offer is surprisingly good for free. Downside is you have to log into the console once every 3 months, and Oracle being Oracle, you never know how they'll change in the future.

Still, if interested, might be worth a try, being free. Do note that they do request your credit card details, though they promise not to charge you (take that as you will): https://www.oracle.com/cloud/free/