The Container Revolution: Reflections After the First Decade

hi I'm Brian Cantrell I am the CTO of giant giant was a start-up we were recently acquired by Samsung so I actually now work for Samsung Samsung may be a company that you care a lot more about today than you did yesterday because Samsung and the division that bought us one side of Samsung is actually the mobile division and you might be surprised to learn that they actually make mobile devices phones phones that I have beautiful cameras phones that are water-resistant and they've got a very new technology they actually have a headphone jack um so I was actually kind of overwhelmed with like the DMZ yes you'd be like all right so which Samsung do I get I'm like what's no no no I mean this is the Apple how can you screw up a launch this oh never mind okay anyway so I now work for for Samsung which actually very exciting Samsung is going to be using our technology do a lot of interesting stuff and I'm presenting on on containers reflecting after the first decade which is not totally accurate I actually doctored this title a little bit to make it a little less frightening because the actual answer is if we click her note I have the wrong quicker oh there we go the actual answer is it's after the third decade I said the first decade so I seemed like old enough to be interesting without so old to be actually terrifying but I'm actually old enough to be terrifying and you just asked my kids and we are actually three decades in two three decades plus in the container so when we're containers born what's the date of birth of containers well we actually have this we actually know the date of birth containers going through the birth canal March 18 1982 do not raise your hand if you were not alive I don't want to know um suffice it to say a bunch of us were alive okay um but on march 18 1982 and they were born with this new idea in seventh edition unix um called at root and to root the idea of to root is changing the root directory so it's the root directory you you are in a subdirectory or specify a subdirectory and now this directory becomes the new root directory you are in a virtual universe a virtual file system where you can now install software thinking that it is in the root filesystem what it is in fact in a subdirectory it's a very good idea and don't know why we have it which is kind of funny um no one actually knows why builds integrated by bill joy bill joy a bit of a Space Ranger kind of in a constantly in a near-earth orbit um bill joy integrated this March 1890 a to probably some THC in the system I'm guessing memory loss issues you know it's it's the 80s it's Berkeley you'd have a contact high from doing this work but we actually don't know why he did it we think he did it to develop four to BSD he was developing forty BSD on her previous version but this is as close as we're going to get and to why we actually have true but it's an incredibly powerful thing why ever we have it it's an incredibly powerful idea and it was the birth of a very important idea and it was an idea that that people saw and they wanted to do other things with one of the things they wanted to do was CD dot dot out of a true environment which actually Italy wouldn't able you to route the box so there were some early vulnerabilities that were kind of quickly cleaned up but chroot seemed like it was a it was a good step but you could go do a lot more you could go build a much more comprehensive virtual environment in the operating system and so it's fast forward to mm with the introduction of jails from FreeBSD and this is one of these things where all you need to know is the name it's a jail okay incarceration is the model punishment is the model we are trying to punish the wayward application in this case FTP D I think you have TBD didn't have so many vulnerabilities we probably wouldn't have jails so I guess thank you FTP D for your vulnerabilities and the the FreeBSD folks invented jails to be able to take effectively untrusted software and run it on the computer as route as route in quotes it's running as root as UID 0 but it can't actually see the whole box it can only see what's in its little jail its little cell that's all that it can see and you know this is done for a bunch of reasons I think laziness is definitely among them and I say that as a compliment I say that is one lazy software engineer to another I I respect your laziness sir the the end the the kind of the laziness here is weary don't want to solve the UNIX problem of root can do everything right so UNIX has this problem that wasn't being rude you can annihilate the Box you can RM minus RF everything so what's actually we actually want to solve that problem so instead of solving that problem let's create this little virtual environment in which UID 0 is not UID 0 on the entire box and that was a jail and that was a really good idea so if true was a good idea now jails is a much fuller manifestation of it but it is a jail I mean it's not like it's not exactly a great place for an application to go I mean you actually are in jail and there were all sorts of things that an application couldn't do in jail it's like what you are in jail so of course you can't do these things things that an application wanted to go do and we kind of saw this at Sun I was at Sun for 14 years prior coming to join and we saw this at Sun and wanted to take this to an extreme and how can we go not have a jail but how can we have a full application environment and we developed we called zones so Jones develop and Sun add Sun it won't be the 2002 era a great paper on this and by the way two very good Papers written on jails and sones I actually did a papers we love on this you can see how long I'll talk when not given a countdown timer the answer is about 100 about one minute and 40 what hour and 43 minutes um so you're going to play the control Ian drinking game while watching that talk on YouTube you might end up in the hospital you might end up EMT so that it's a long presentation but it would go into these two papers that are actually incredibly interesting these are vid these are two very interesting ideas and the idea with zones in Saurus the idea with zones is we're going to run an application give it a full application environment we give it a full user space give it a full file system space and so on give it a full process space but it can only see what's in that zone now very importantly you're in a virtual operating system but you're running on the single one operating system kernel so every application that is running in a zone is running on the hardware you're actually you were getting the full performance advantage of being on the hardware full tendency advantage and so on so we thought this was a great idea and we thought we developed this fully as it turns out you know sometimes you ideas great ideas don't get traction right away this is definitely one of them some folks saw this is a great idea inside a Sun this is kind of withering on the vine fortunately we open-source the operating system which is its own Odyssey we open source the operating system and others outside of son saw the power of zones when Sun itself did not and actually one of those companies was joining so courtesy of the wayback machine here we have the joint page circa 2006 it's kind of painful for me to read this thing like a like a garbage heap from from the Paleolithic era this thing is just loaded with information for future anthropologists it's like hundreds of web 2.0 applications it's like Teddy what was web 2.0 what was web 1 that I was like nobody knows they're all marketing terms but I thought the we're lucky making Ajax on this one um yeah I mean Ajax weird uh mad so Ajax brothers you have Millennials let me just educate you for a second Ajax all in caps stands for asynchronous JavaScript and XML that doesn't make any sense of course I doesn't make any sense it's like and isn't XML itself in akram yes it was a very bad idea but it was all about Ajax was the rage back in the day back in the web 2.0 day which also made no sense moving right along as we had accelerators did you know that a joint has the largest open Solaris installation in the world like whoa two computers running open Solaris on B did you know that joint minute is a hundred and seven terabytes of data now I'm glad I heard some terabytes at least is more than you have on your phone currently but not for long um and I love this quote from this company obvious that you haven't heard of that joint accelerators less scale nothing that you haven't heard of it because they renamed themselves to Twitter Twitter was eight was a huge joint customer um Twitter was growing exponentially and then they were fought the CEO of the time likes to brag that he fired them as a customer because he was sick of dealing with him that CEO was subsequently fired well they definitely over drinks we can compare bad CEO stories um but so obvious was a customer and in the end these things were great they allowed terrific tenancy terrific performance terrific scale but something else also happened in 2006 so the an ec2 happened in 2006 at the same time I mean almost concurrently Amazon announced the limited data for ec2 the elastic the Elastic Compute cloud course very important and so joint was using containers what was ec2 using were they using tanners oh no no Oh contraire no no and this is now we need to do a sad little aside a sad tale of woe called hardware level virtualization because the containers are not the only way to virtualize a workload the way they were done at Sun and a joint entirely multi-tenant safe and so on but there is another way to virtualize a workload you can actually virtualize not the operating system but the hardware and what you present to a tenant is not a virtual OS and a virtual social system call table you present a virtual x86 microprocessor virtual DRAM virtual NIC virtual and I got this I got a prop for this a virtual one of these which you may again if you're Millennials you probably recognize this but your day is coming when that when your coworkers will not recognize one of these this is a floppy disk it's like is it floppy no it's off lobby that's a long story but it this is a three and a half inch disk on and you be egg wow what a relic and it is a relic but it's also virtualized in every in every virtual machine that you run today you have a virtual floppy disk it's like but that makes no sense you're right doesn't make any sense like why would you have that well just kind of dead code until it's exploitable that's called venom so venom actually exploited the virtual floppy disk controller in a virtual machine so this is serious stuff and it's like it doesn't feel like having a virtual computer is going to make things any faster and it definitely doesn't because every tenant now has to have their own OS kernel because it's seeing Hardware right you need an OS kernel the one on hardware and trust me I write a kernel curls don't get along with one another I mean people write kernels are control freaks and operating system kernels do not get along well with one another these are very fat applications that want to control all the resources they have and as a result and this is where you get to the real the danger of the abstraction is that because it will consume every resource that you give it it's incredibly resource inefficient from the perspective of the hypervisor that needs to run this stuff it in particular in terms of DRAM when you give an operating system DRAM it takes it the end you are giving heroin to an addict that's it up the arm it's like hey you still have that heroin I gave you two years ago it's like no what do you come on dude I'm on the fentanyl I mean I'm they the so you can't when you give D RAM to it to one of these guests it's gone from a hypervisors perspective even though PS the DRAM isn't likely being used when you spin up a VM on ec2 on Amazon how much of that DRAM do you use and if you're like I use every last bit of DRAM there's an ops person somewhere that wants to do violence to your body because it's dangerous to actually exploit every last bit of the RAM right you're running very close to the wind you're not using every bit of Ihram in fact you're probably adjusting your estimates such that you're only using half right or maybe you're using three-quarters the problem is that you're handing that estimate on to someone else who disgustin doesn't trust you and they do a little bit of a jackass nice a nice person but yeah often get these estimates wrong I'm going to multiply this again to get I'm going to actually assume that they're off by another factor and these factors get multiplied together until you have a VM that's got a ton of memory doing absolutely nothing unless it's running Java in which case it's actually garbage collecting all the time that's a different story it does not play well with others it's terrible with respect to tenancy and yet thanks easy to its de facto in the cloud so that's this is just what happened Archie was going on in kind of container land one container land which was a very lonely Island population one population largest open Solaris installation in the world on in containers at circa 2011 we actually um even though we strongly believe in containers we actually realized we need to do a hardware level virtualization as well on because we I we've got we couldn't run Linux binary so okay let's do hardware level virtualization so we can run Windows Linux and so on and boy do you thought I was a vegetarian before getting a tour of the sausage factory you I mean going into the sausage factory of hardware level virtualization and seeing just how bad and archaic it is and you can't do anything about it it's the abstractions themselves that are actually confining it's the fact that you're booting on an abstraction that dates from Boca ratone in 1979 where the IBM personal computer was first develop I mean it's unconscionable um you shouldn't have to know what real mode is real mode is a war crime and yet that every VM boots in real mode so we realize that we had to go and actually I had the debug an issue in real mode in 2011 as reported kbm Smart OS like what you and you just have to go into like you're safe antiquarian Antiques Roadshow kind of space like I'm antiquing like I'm not actually living this right now I am I am enjoy like I'm this is like steampunk i am i you know i because if you know like if you're trying to treat it's actually modern infrastructure you actually go insane why are we doing it this way it's like no no no we're antiquing we're antiquing that's a guarantee okay um it's like but but but you're trying to treat the table as load-bearing it's like yeah but i'm so and we came out of that it would actually even stronger resolve it's like okay God if I'd if I didn't believe in the container I'm leaving today nerds going into this very very very fervently believe in containers fervently believe in Oh us-based virtualization this is obviously the only answer and indeed wow this actually OS virtualization actually opens up some really interesting possibilities so it's circa 2012 2013 again the rest of the world had not discovered the the long-lost I would known as containers so we were kind of off on our lonesome at Giant and we had this great abstraction in terms of containers and we we realized that actually this could revolutionize the way we do storage because we wanted to go build a storage service Allah s3 but we do want to just build an s3 and we realized that with the power of containers we could build a storage service like s3 HP puts and gets and so on that when you wanted to compute upon it you could spin up a container where your object actually lives so you can actually perform arbitrary computation on your object without moving it and this is when it this is something that seems obvious to you now I can assure you it was not obvious to us at the time it seems obvious in retrospect but one of these ideas that is just like oh my god was that in front of us all along that is an incredibly powerful idea it is a powerful idea so over 2012 and only 2013 we built this thing called Manta which we later open-source all available its ZFS space container based internet-facing objects or so like s3 but offers that power of insights you compute which allows you to do all sorts of things without actually moving data and remember the abstraction that you're getting with a container is the operating system and by the way this operating system UNIX goddamnit UNIX is actually let's let's forget our kind of petty differences of the UNIX wars we all we are all believers in eunuchs I mean even Redmon's believers music sounds actually amazing um UNIX is actually built around the idea of ad-hoc data processing so UNIX is actually perfect for this and this this brings in again if you're playing the controlling and drinking game you want to be sure to take a drink now because we're going to do a first reading a reading from the book of Doug McElroy McElroy 3:15 on the UNIX philosophy so when when UNIX and unix-like accelerate 60s early 70s the systems that came before UNIX seemed unbelievably antiquarian truly antiquarian and UNIX actually feels modern because it was instead of having these massive monoliths it was these collections of programs that were designed to do each program itself was designed to do one thing well and you could actually string them together with this great idea called pipes and pipes were an idea by Doug McElroy Doug McElroy is kind of a UNIX founder that you haven't heard of people have heard of ken thompson dennis ritchie but it's really Doug McElroy that gives us the pipe give us the idea of UNIX and the UNIX philosophy that is say write programs that do one thing and do it well write programs to work together write programs to handle text streams because that is a universal interface if you do systems work you should have daily affirmation with the UNIX philosophy you should read this aloud as you're like greeting the Sun in the morning as assistants programmer you should be reading the UNIX philosophy on it because this isn't currently powerful still four decades later I think this is as relevant as ever and we see shadows of it everywhere so as another kind of a cytokine Dov concrete example of this is one of the greatest battles in the history of computer science Doug McElroy in one corner the inventor of pipes Unix Don Knuth the inventor of a bunch of books that are sitting on your bookshelf to look smart but you haven't actually read these two actually battling off with John Bentley author program programming Perls a book charter off the Elektra miner reddit that had this challenge read a file of tax determine the end most frequently used words and print out a sort of list of those words along with our frequencies now reading this now you're just like that's a challenge that seems like that's like not even an intern project that's like that's wait what are you talking about on this is like this is a an easy test for someone coming into ops or what have you but this was a challenge in 1986 Don Knuth solution over here Don Knuth a purpose-built algorithm in web yes that's all caps for a reason a system that you as they say haven't heard of for a reason a a Pascal like literate programming system and so he invented a totally bespoke algorithm and then as if Harrison Ford in the in Indiana Jones and the Raiders of the Lost Ark if you're a millennial gazing over right glazing over right now see Indiana Jones I know you're not seeing it because you're just trying to defy the Gen Xers and the day you boomers that you work with but do us a favor and see the goddamn movie the Harrison Ford as Indiana Jones um Doug McElroy pulls out his revolver and shoots Don Knuth with this unbelievably short UNIX pipeline that does exactly this and if you're if you're a UNIX nerves like many of us are you may be like wow like that was he really kind of put a finishing move on him with that said dollar 1q and I got a little kind of Oh flourish a little moral combat kind of flourish on because I get you should just use head for that and you wouldn't use head if it had been invented amazingly this predates head like alright only things like our tail was obvious but head was totally subtle okay anyway I'm I so so Doug McElroy slays Don Knuth and a total triumph of the UNIX philosophy and we can actually do this now in mantis this is an actual Mantha job that is finding things in my public directory v7 man but the seventh edition man page this is going to give a shout out and creating amantha job that's that M job create does and it's saying I want the map phase I want to do exactly what Doug McElroy did so that's the Doug McElroy pipeline right there in the reduce phase it uses awk god bless awk talk about another I if you have not read the awk programming language another daily affirmation for you god bless hawk I'm definitely a NOC evangelist and awkward aivalis talks beautiful British quick simple and so on and ox is actually serving as the reduce phase um and this actually gives you the same output but now it's arbitrary scale to an arbitrary amount of data so the this is the this is the power of Unix this is the power of containers this is the power of containers to revolutionize the abstractions that we're thinking about so say we we're like wow this is amazing this is amazing and Manta like boy art so we believe in zones then we do KVM and we come out being like really strong results then we do Manta and we are just on a different planet like we exist in a different time in space we feel adjoint like an alien race with some sort of advanced technology than Earthlings don't yet have but the thing is we actually want the Earthlings to have the technology goddamnit so why won't they take it on because we were like wow this is amazing but we realized people didn't get Manta they're like I don't get it like what do you don't get like you spin up a container on your data and that we actually tried to show them with us we have another utility called em login and log in allows you to log into your object right yeah you can um log in your s3 like object path boom you're in a shell a root shell in a little container sitting somewhere far far away you're in a little container with your object and only your object safely mapped into you and you can run whatever you want it's but it so you've got this little interactive shell and be like ok it's ok I'm still like my brain is blowing up like ok it's the tug ok really okay let's great well I mean what's blow up your brains like ok well like you guys are great but this is really weird man this party got real you're always strange it's like no these drugs are great it's like no I need to go home I'm serious like when's the last bus I so like don't go home don't go home uh that kind of happened over and over again and we're like when is the world going to figure this out this the world has to figure this out this can't go on forever the world is going to figure out containers and they did they did in two minutes into this talk by a past provider that as again you haven't heard up for a reason to keep cloud cloud is struggling and they have decided to pull off an amazing Hail Mary that it would all be actually grateful for an amazing Hail Mary they are failing as a business they're not going to make it we're actually going to we're going to scale all the way back and the Hail Mary is that we're going to and source our software that software is docker and that is docker Inc so if you're looking for an argument open-source software by the way an otherwise dying company that's a pretty good one um so they open-source docker and now people get it and the match is lit and at the fuse is lit and everyone gets it which is grower people begin to start getting like wow this is amazing now you might think like wow you guys must have been like totally mental frustrated by that but actually no honestly we were excited and are excited that everyone now gets to say sees the same thing that we see does the same drugs that we do and now I'm not necessarily okay we do harder drugs but the UM but seize the power container so why did the Revolution start with docker and not before that and the reason I think looking back on it is that we always emphasize aside from Manta we even with Manta we were emphasizing the operational characteristics of containers these things are rocket fast you don't have this dumbass thing slowing you down they are incredibly fast yeah great tendency and so on which people are like kind of mad I mean people do care about but the people that care about it are totally disempowered in their organizations like I love you guys but my manner doesn't care um so there's a lot of it but what docker did actually docker connected to people in a much more and and the kind of container evolution where now connects to people in a much more important way because it actually allows us as developers to think operationally in the software that we deploy we generate these giant now effectively static binaries called a container and we can now run in production the same thing that I'm running on my laptop I don't have this kind of bespoke pet this kind of zombie pet in production I am ideally running the same thing in production I'm running in my laptop this allows developers to think operationally and most importantly it allows developers to move faster software is truly truly truly truly eating the world we are all very very lucky very lucky to be in this domain at this time this is a golden age I promise you because everyone is figuring out that that 10 years ago 15 years ago software was viewed as a cost on i.t was something you out worst that be that you were and you heard Adrienne say this yesterday is absolutely right people were trying to that all they were trying to do is reduce cost reduce cost reduce cost and I think you can even argue that our arguments in favor of zones and containers came from the fact that we grew up in that area era I grew up in that era Mike book zones cost you less yeah don't care as much anymore because what I actually care about is writing software quickly so I can go compete and innovate because what we now have happening that we did not have happen for a long time or haven't ever had app is software companies are disrupting mainstream business taxi companies companies that actually predate the automobile livery predates the automobile they've had a chokehold on their markets for generations they are finally being slain by a software company by actually like five different software companies all competing with one another with all outsized multitrillion-dollar evaluations but the point is and you can argue like oh they're over this you can add a up line but like the old world is not coming back is definitely not coming back and the fact that software companies are disrupting the hotel industry software companies are now disrupting all these old industries it's got everyone's attention and containers are right at the tip of that spear they accelerate software development all in a dovetail right into micro services we heard Adrienne talk about a lot about that yesterday micro services containers these things are all related to one another they are all about allowing us to go faster so we saw this we were like yes this is great the world gets it and so we develop this thing called Triton so late 2014 or 2015 we see the energy behind docker we're extremely excited with what we want to do is actually combine the strength of zones industrial-grade strengths proven strengths of zones tonal security of zones with the the excitement the energy the enthusiasm and most importantly the accelerated software development you get with docker and we did all this thing called Triton so what we did is we took smart OS and right wait isn't that solaris like okay please don't say the s word first of all um because we've been I'm going to try to go through this without making an access powers now J with me the former owner of the that forget it um the what we did is we actually took the the Linux system call table and we implemented an existing whole table on top esparto so we were able to run Linux binaries natively on the metal in his own and then very importantly we also took the the document well-known love docker action has a remote API as it turns out and we as we were kind of exploring this we realized we could implement the docker engine in our cloud or because thanks to the remote API we can actually go implement a remote API endpoint that would look and feel and smell like docker to docker the command but would actually be an entire data center to little containers so that's what we did that's right so when you when you spin up a container with Triton you do a docker PS you see your containers across the entire data center it means the things like docker compose which are really interesting from a developer's simplicity perspective but of limited efficacy because they don't work in the cloud actually work on us because a docker machine machine is actually a data center so we were very excited about this um and again we made this very important decision to actually reimplementation saw the talk that I gave when we announced this in January of 2015 said hey we enjoy the talk at a curiosity why replace the docker daemon entirely and not implement a new execution driver for branded zones very good question a question that shows a sharp level of attention because we had the exact same question and what we realized is that when we went into the doctor daemon it was simply too specific too immature moving too quickly it was not reasonable to recast this in terms of our stack what we could do is go import that API so um to which I responded hey listen I with I said tunning I meant running running the doctor demon in the global zone is way too much risk it was much easier to do this and of course it was all open source at that point um calcitic ok great that makes sense I love congrats on nailing the implementation in the demo this shows you that Kelsey actually demos things in front of an audience um this morning when Kelsey was kind of pausing on console to see everything you like I'll come on baby come on come on because I get there right there and you're like why is he celebrating so much like should this always work it's like yes of course it always works but boom it works in the and you only understand this if you've done this under fire so but so Kelsey totally understands that the Hank the demo God's working it we're working that day thank God on and then I say hey thanks and I imagine others will also take this route I also I just kind of run back and read this tweet recently I'm like damn that was pression you know I'm not oh I'm kind of a Gatling gun of random thoughts but that one was actually pretty freshen robust api's are essential for the container ecosystems this is something I believe in fervently I know Kelsey believes and a lot of us in the room believe in this we need robust api's and we're going to talk about that as we kind of look forward here in just a second but this was to a certain degree ominous foreshadowing because the world was simpler then in early 2015 um and the Wow okay you're not supposed to have like a sad knowing chuckle I mean you chuckled as if I said in 1942 that the world was simpler in 1938 um you're not supposed to be that kind of a chuckle but it's true the world was simpler we just there was less there was less attention to it because people were just kind of figuring this out and there's been of course this explosion as everyone has realized wow this is the next thing and everyone is moving and trying to move everyone trying to move faster than than the next person next company and this has created a lot of technology and the technology is very complicated and I don't you know Camila Fournier Camille's great great great technologist CTO out of New York Camille's CMU CS worked at Goldman Sachs CMU that our Camille is nails as far as the technology goes Camille is so good as as technologists that she has the confidence to look at this stuff be like holy crap this is complicated Wow and Camille Camille is a zookeeper committer okay seriously put that in perspective like a Camille has got a high threshold for pain I mean this is a life I did this is someone who my father was an emergency medical physician this is somebody who's got a blood-alcohol level of like 25 times the legal interview like okay now I'm drunk it's like this is like wow wow wow and also not just well complicated but also crazy complicated how many characters can I fit in here say how complicated it is so wow this is complicated and the question that I was asking last year at Q Khan is are we your peak confusion in the container space and and I was kind of asking everyone this because like I was confused it's like there's so much going on I'm confused like just tell me the confusion is going to get better and people were saying no it's still accelerating in 2015 I don't know if it's too accelerating or not I kind of feel like we're maybe in that weightlessness going over the top of confusion I don't know I I don't know if it's still accelerating I know there is a lot a lot a lot a lot a lot of confusion and this is a problem as we go forward looking forward from 2015 into 2016 and beyond now there is a problem and there aren't we have got challenges in front of us and yes well we're going to collectively figure them out but sometimes people say that because they're not going to one's figuring it out we're going to be the ones figuring it out we're doing you the hard work of actually figuring this about and we got a lot of stuff to figure out so in particular there is there's a battle the battle lines are being drawn between two ways of thinking the what I call the framework approach versus the library approach the framework approach is I the framework I'm in control and I will call out to you in certain contexts and these contexts are kind of strange no specified and but that is where you will add your functionality the library approach says I will provide some libraries as a toolbox that you will pull off the shelf and use to build new things now obviously I'm a bit I got a bit of a library bias here and yes I was a j2ee naysayer back in the day feel a little bit vindicated by that one um but this is the danger of frameworks is that frameworks they ease your initial adoption but then they ultimately sacrifice flexibility because they've made so many decisions for you and they might not be decisions that you agree with or they may be decisions that you agree with but the next organization over in your company that needs to implement this that hell if they agree with it right this is I think people don't understand that when your software crosses organizational boundaries within a company you've set yourself up for failure because the organization's are obviously not going to agree with one another I mean clearly they're at war with one another um so this is a challenge with with the framework approach and all I think that some things that start off the library approach have become a lot more framework II and this is a challenge and now conversely I think things that started off as frameworks or trying to figure out a way to backpedal a little bit and become a little more like a library now I think here at hashey Kampf what we see is that library kind of idea I love the kind of the hashey ethos from my perspective is that library ethos and I love vault is just over here here's what vault does it integrates with other things but you know what it does then we got Nomad it's over here that I think is very helpful because it allows you to adopt vault without adopting Nomad or vice versa that's very important so we see examples I mean one example that is an orchestration um when you've got this kind of framework approach to orchestration you may have this kind of all-knowing framework that doesn't actually have the right fit for some applications so one of the things that we saw is like we actually really need to put the idea of orchestration service discovery into the application where it belongs not have the all-knowing framework we call this application centric orchestration um we dubbed it the autopilot pattern and go to the website and check that out and the kind of the first embodiment of this is this thing called container pilot it's all open source and let's see what container pilot is unlikely oh stuff we've done frankly you can use container pilot and not use anything else that we've built um yes it works like hash a like a chic or we've designed this to use totally on its own and people are using it on its own and which is great because we think it's really valuable we're using it a lot on because again you've got all this logic just lives in the application in the container you can totally wrap your brain around it it's a couple thousand lines ago that's it it's it's so we think it's pretty powerful and we're looking for other abstractions like that so that's definitely kind of one angle that we need to kind of collectively take um in terms of failure modes so one of the things that does drive me a little bit nuts is when people talk about containers when you and I I try to remember how long I have known Adrienne and how much I value my relationship with him but sometimes Adrienne is like Oh problems are just fixed by dwarves or they've been fixed by magical Pixies that live in the mines and it's like no Adrienne failures are actually important and just because you're not the one wearing a pager doesn't mean they're not important failures do happen do people know what this is by the way this is the the Northeast blackout in the six days so the great Northeast blackout is this is much more the kind of failure mode that we need to worry about today so no it's not the same failure mode as we had in the past but now it's a systemic failure mode the entire system can potentially fail and indeed I actually think the the most apt metaphor for container failure for this kind of micro service failure is actually this one recognize this this is Three Mile Island uh and Three Mile Island is one of the rare photos taken inside the control room during the crisis in Three Mile Island and Three Mile Island man the more you know about Three Mile Island the more you're like this is what I deal with every day I'm in Three Mile Island every single day why very complicated system so here's their micro service architecture at Three Mile Island um and so just to give you some quick education in terms of the actual failure here on so they were running auto vacuum on Postgres I'd excuse me they were they were cleaning out the the demineralized over here so that they're trying they're flushing the resins out of the demon riser they put basically too much pressure down there some water popped into an airline and that condenser pump went offline it stopped it's fine this is supposed to happen but the auxilary pumps are not in this diagram actually did not fire which is not fine that's not supposed to happen is one of the major failures here that didn't happen 14 is a third system that number seven up there is called the pilot-operated relief valve that detected what was now over pressure in the reactor right so we actually have we are not cooling the reactor fast enough and that thing pops open okay so so far so roughly good it's this we had the failure of the auxiliary pumps is very bad but here's where you get to the the most apt metaphor for ferry's we deal with because there was an when the pressure finally dropped the pilot-operated relief valve automatically closes but there was an error in the actual light on the control in that it was simply miss designed the light will go off when power has been applied to the solenoid to close it not when it's actually closed so every operator is trained to like if you feel that the reactor is is actually losing coolant make sure that the pilot operated relief valve is closed but from the panel it is it's not lit you can plain these guys and indeed the final findings of this didn't for not knowing that this thing was actually that broken in its design and this is what we deal with all the time right well you've got a monitoring where you're monitoring is actually screwed up or you're monitoring is causing the problem or you're monitoring is not seeing the problem right MA during is great monitoring is important but the problem is when we replace our basic understandings of the system with lights we we end up in this situation Elan awareness that leads to a meltdown and with it happen and obviously this is we're talking about a much more serious issue with a nuclear reactor but as more and more of the world is on software as more and more mainstream commerce is on software this is going to become more more important and preventing failures of this nature is very very important so when we talk about containers we talk on micro-service they're the GAE in production and yeah we love production it's okay stop stop this whole like romanticization of production production is war as the people in this room know and war is hell so if you're talking about like I can't wait to be in production it's like I can't wait to go to Afghanistan yeh it's like like where's your coalition of coffee please like you're not going backpacking ok this is a war zone and production is a war zone especially a distributed system micro service based distributed system because these things can fail in really really really nasty ways yes they can survive certain classes of failure but don't look at their ability to survive failure look at the kind of pathologies that they have with cascading timeouts harder reproduce problems and so on we needed beam to think not in terms of the program but the system so this is a very real problem and it's going to be a challenge for all of us going forward another challenge the Jevons paradox so Jevons was a Liva Scottish economist made a very interesting observation in the 19th century and that is as coal burning became more efficient coal use went up it's like what because people could now do more there are more things that you'll do we are seeing the exact same thing with containers absolutely because containers allow you to do more and be more efficient people are doing more and we've talked to blend your customers like I women this is more efficient right yes why is my AWS bill higher it's like ah that because it's very easy to kind of spin up all this stuff and and I either forget it or you're using it for new use that you had before so we are absolutely going to see this and the idea that we currently have now of scheduling containers in VMs scheduling your container on this sucker bad idea it cannot possibly survive into the long term especially is we're going to use computers to do more that we this is going to be explosive in terms of cost if we are doing as we are doing today which is putting containers inside of VMs how do I know this will be exciting again Kelsey gave a great talk this morning or is playing Tetris you're like no problem like we'll just play Tetris so yeah about Tetris so some of these things like you know when you lose Tetris like Kelsey did out with the ORAC I love your achill example we can lost Tetris that we you will lose Tetris because you end up with islands that have been provisioned vm's that have one and two containers on them that you actually can't spin down is it this is the challenge so we and from our perspective the ends just don't belong here we need to get rid of the floppy disk please we need to go to container native infrastructure where containers are running directly on the metal that's the only way we can achieve and unlock the tenancy that is the promise of this incredible revolution and we believe by the way that you know a lot of people think on-prem computing is dead the death of on-prem computing has been greatly exaggerated lots and lots of people have on on-prem computing doesn't necessarily mean blades and sands and enjoy what's a blade or Sam like oh god bless you let me preserve your innocence please um that is so great um I I love that and I'm not gonna I'm not going to tell you about this filth that is these legacy enterprises are carriers but that is not what on-prem computing has to mean the decision between on-prem and public cloud should be based on economics it should be rent versus buy if you've got enough compute you're going to want to buy your own computer as it turns out it's risk management what happens if I in terms of security and compliance what happens if someone gets into this data and in that admittedly has been played that to a certain degree that that is less of a concern means certainly like the public lot is safe but if your entire business as your data you that you're probably not going to trust it to anybody and then latency this beautiful is a concept that's not changing economics dominates the reason that private cloud efforts have all failed to date is because they don't realize the underlying economics they're trying to preserve these vat margins from the blade and say aneri going into private cloud we think that we can't do that we need on-prem computing should just be another AZ that's it if it's anything more than other AZ the effort itself has failed so I think we're going to see that for sure going forward so that brings us to hashtags serverless okay now fortunately I I'm going to try not to have any aneurysm on this one because I this is clearly a meeting I start obviously it's like hashtag server Lewis is like hashtag CPU lists or hash tag instruction lists like I don't execute instructions it's like all right that's great like you're not running software I'm sorry I guess I you're welcome to not execute instructions but you're not software um so it's like okay it's clearly meaningless but then I try to get to control my rage I'm like okay but on the other hand there is something real being expressed here which is the idea to kind of get off of this low level of abstraction the server and get to a higher level of abstraction to think in terms of the function that you're providing and we think this is great look at Mantha we love Mantha you love thinking about this but the virtual machine is a vestigial abstraction we cannot get to hashtags serverless without getting rid of the VM it's ridiculous to think I'm not running on a server we know you're running on a server you're running on two of them that don't even know about each other you're running on a virtual floppy disk how about hashtag floppy list let's try that one first um so I think don't say hashtag server list when please you actually mean is hashtag V endless please please please and with that thank you very much you

Channel: HashiCorp

Views: 34,312

Rating: 4.8792868 out of 5

Keywords:

Id: xXWaECk9XqM

Length: 44min 4sec (2644 seconds)

Published: Sun Sep 18 2016