micro-Linux init (PID1) in Golang

about my clinics in it and I remember a system v in it so you see how this is different this is sven who's thankfully said i don't pronounce his last name welcome spin so the title of this talk until yesterday was something to do with rancher OS and golang because i used to work on rancher OS i've been incredibly fortunate in my career i started off being a commercial c++ dev like an awful lot of people and got involved in the wiki revolution looking around most of you probably have used a wiki now and then somewhere along the lines docker came along and i started working on boot to docker which got me picked up as working for docker for a few years and then when that stopped i kind of start i did a year working on rancher OS where i continued playing with micro linux distros for containers and once we'd released that I've kind of paid it off and now I'm working at sciro so that's who I am and what unfortunate thing about this about my career has been that I've been able to follow a series of curiosities because of open source and I hope that other people get this opportunity as well so now about you guys as rather a lot of you who here has used docker okay no no who who here hasn't used docker excellent be nice to know why but yes who here has played with go in some way again again who here hasn't played with go because yeah it's it's a different group but similar sort of size and for those of you haven't played with docker have you played with containers instead like lxd or cool so it's to me one of the most fascinating things in joining the the the docker thing was that I came from spending the last 10-15 years writing Perl code and JavaScript and I just found Co incredibly easy to read and make changes in so much so that some people have started rewriting their bash scripts in go which makes me wonder why we don't have people writing their build systems in go source and then come yeah it's kind of a bit ok so this talk is kind of about container inspired micro Linux and I mean the reason I got into boot to docker was because I was prior to that mucking around with with phos wiki and twiki and I made installers for Windows and OS X despite being a Linux user on my desktop and it seemed to me that as I was writing documentation from what I was learning about docker that the biggest issue people were facing was that they couldn't actually use docker on their system because how many people actually run Linux in fact how many people here don't run Linux on their desktop or notebook and I have now joined you again in my current job I get given a Windows box it's fascinating but you know the world has moved on now we get docker for Windows we don't have to use with UNIX services for Windows anymore we get a bunch of Windows for those of you who use Windows how much fun is that having 4 different shells that don't talk to each other and don't work together I it astounds me how often I'm sitting there in one of my shells going run this command oh no that's installed over there and they yeah but anyway so container inspired I mean micro Linux is for me started out of trying to help others be able to read and play with the documentation I was writing and so I started working on boot to docker which was created by a friend of Solomon's but in playing with a micro Linux distro you realize that it's actually interesting to be able to boot quickly and obviously AWS and as you are now charging per second I get the feeling that everybody cares about boot times being as small as possible they're also simpler for users to reason about brutally so I thought until I started writing this talk at which point I'm looking at the in its of both boot to docker and Rancher OS and Linux kit and going yeah they need to be simpler and the other weird thing that I like to play with is the idea of actually being immutable and so I like to boot my hardware or virtual machine with stuff which doesn't get to change if I want to change it then I build a new one and I'm sure there are lots of other reasons why people like to think about micro Linux's oh yeah and if anybody has any ideas shout them out I will repeat them I kind of like to have the feedback of knowing what you're thinking okay so where I started boot to docker and dock at toolbox I just thought a little bit of history where Steve Marin made a micro Linux out of tiny core now tiny core is already microscopic these days it's like 15 Meg as an ISO back then it used to be I think 42 or something used to be a little bit bigger and then adding docker to it pretty much blows out the same size to where it started and then we started wrapping it in a set of tools so that you can go buta docker start and it would create your virtual machine on your box whether that was VirtualBox later qmu and so on and so therefore when you know I wrapped that in a set of installers and so on and that eventually became docker machine and docker toolbox how many of you here have played with a boot to docker before toolbox existed so the world has moved on I gave a talk about boot to docker probably in Auckland and a huge proportion of people had played with to boot the doctor at the time so it's nice to see the audience has changed and then dr. toolbox anybody have to play with that beautiful Linux audience I was going to show some of boot to doctors in it code but it is just in it has in busy boxes in it and I hope I love shell scripts and that's not very go issue so we get to move on okay rancher OS on the other hand was an experiment made by Darren and his guys at Rancher where they basically took the idea of a micro distro and said what happens if everything is a container so there's a container for DHCP there's a container for a NTP there's a container for you dev and a container for I've forgotten but even more fun there's a container image for a user docker and so that means that what the user interacts with is actually a containerized thing on top of your micro linux and that gives you fun little things like being able to change what version of docker you're running and that sort of thing so we start with that and this was after fig got acquired so it was part of the docker compose idea so they took cloud in it and added to that the idea that you can use docker compose constructs to orchestrate what is running in your operating system and to me this is his this was the idea that we would have liked to have done for boot to docker but really there were better things or different things afoot in the dock company you know dr. Mac and doc of Windows and Linux kit which I'll kind of push into next so that was the idea okay how are we doing yeah that's Tim this is the pain and because of that I of course managed to forget you start my machine as is one of the fun things where I am running a qmu virtual machines started by and created by docking machine at home I would normally shuffle this off to my VMware server and other people use it for AWS and so on yeah so here's me showing you how I would create a virtual machine using docker machine the fun things are let's see did we get there yes okay so there's our docker that the user interacts with okay maybe that's a bit big can you guys all see what it wasn't quite what I mean are we better off is that still visible okay sweet all right so we've got the docker the user docker running and we also have I'm gonna rue the day okay and this is all of the SI system services that we're running to keep branch or OS up and going and as you can see there's not very many of them one of the actually I might just start the boot to dock on as well just so much easier with my three I was gutless okay but that said we're still running quite a few things kind of scary and if we have boot to dock or running yet no we don't alright so the fun things that I liked about this is right now I could switch between any of those docker engines and I can also switch between consoles yeah and there's the downside of homebrew and so the shell that you interact with which is running in a container can be switched to whatever you're most familiar with and then you can go off and build your stuff create more composed files and use not only this development platform but also as your deployment system which I quite like meanwhile over here reason I kind of wanted to show this is Buddha docker being cut down he's actually slightly smaller not nice can I remember how to do this that there should there we go it's only a page where that's all the non colonel things that are running in boot to Dhaka and I'm not sure that I can do that right now in this colonel but we'll try there we go it's very similar yeah exactly you know and we're setting up terminals manually which is the pits not much fun okay so well there we go I could switch consoles but I don't really want to use up the Wi-Fi because this is running locally so one of the things that attracted me to the way Rancher OS did things is this is the cloud config for example that I'm using or was using to play with piehole which is a malware kind of avoidance DNS server and really I was running machine create on my VMware server branch OS with this config and bam there it was running now those of you who are ops minded unlike me would realize the fatal flaw it might not be quite obvious until I scroll down it's ran beautifully for about three months and then died because it ran at a disk space because I was not managing my logs don't do that especially when you're away on a conference yes there was many angry slack messages I was just playing with it on and this is one of those to segue reasons why I think DevOps is is a fun idea but I actually need an ops person to look over my shoulder and tell me that I'm an idiot and if anybody would like to be my ops person at home that would be great so that's the docker compose version seemingly there's this great big Syriac secret that most people don't realize is that Plowden it will also allows you shell scripts which means that firstly you can do something silly like this where my shell script if it detects that it's running on my home box will go off and use docker machine to create a virtual machine running this state and then after that can be used to format my disks to do an install and to pass a new cloud in it because the way wrench are we works is that it actually takes the cloud in it and stores the Lions disk if it gets one it's one of the things I forgot to mention is that in the previous example I'm not actually installing I'm doing this in a very booted Dockery kind of way where the operating system just runs off the ISO and I think in that case I gave it some disk but I didn't need to so when I say ran I did this disk base actually ran a memory tasty okay so when I didn't know anything about Rancho Oso I was thinking it was a cooler way to do a mutable but in actual fact it's not immutable like the operating system which is convenient for the operating system manufacturer who for a year was me it is immutable but then once the users get involved all of their customization happens at boot time which means we might have an operating system that is lovely and fastboot but it's not ready and what you want is something that will you know I want one of these now a load balancer and it's up and serving traffic so because this is golang I thought I'd do something really silly and show some code and luckily all right that's probably the most famous line of code I have ever written apparently that is still around what year is it four and a half years later and if you make a disk with boot to docker or ranch or OS or any of the other derivatives or conceptual derivatives I think photon might do it as well if it detects this is the first binary parts on a on a disk when it boots it will format this disk really handy not exactly a clever Strange's though I didn't think it at last okay so basically rancher OS goes off and does a ton of things it goes off and prepares the init Ram disk it tries to find enough information to do networking so that it can do too loud in it from a network it goes and has a look at the what's called OpenStack style config to cd-rom and then goes off and configures your operating system and it is unfortunately way worse than this looks because we keep finding little things and fixing corner cases we also had these fun little quirks where for example doing a down Holt seems to break every second or third colonel and presumably it's our fault but it's it's one of those fun pains we even do se Linux lovey okay okay yes supporting distros is hard work if you ever get the chance to make your own distro do it it's fun you'll learn a stack try not to have users yes and try really hard not to have commercial users who rely on you in production I think this is the the biggest change between food to dr. Rancho s is that food to Dhaka was clearly for developers to play with and learn Rancho s is in use in production and when boring simple you know unimportant things like meltdown and Spectre happen yeah little micro distros like Rancho s get to have people running around like headless chucks luckily now that I've left they've got more people involved I think the worst thing I did was say yes I can do this learn a lot it was fun but I yeah not doing that again because you need to keep up with everything now rancher OS has a Linux kernel in it it also uses build route even weirder as I discovered probably about two months before I stopped working on it it has to build roots in you know as things grow the complexity gets hidden especially with the containers where you know that the the second build root was actually just coming in from a container that was up there on docker hub and I had not noticed quite frankly I'm sorry yeah I think they've killed that now but yes but there are benefits and the benefits really should be boot times your security footprint should be much much smaller one of the things I have trouble feeling comfortable with is the fact that our desktop operating systems and our server operating systems for convenience reasons are essentially identical and that means while it's nice to be able to plug in USB and have my devices magically get created and all the rest of it I have a lot of trouble figuring out why do I want my server hardware to magically update at all you know maybe there's this boot window where things have to come up and be orchestrated by the kernel and you Dev and so on but after that why and obviously micro Linux is should or are slowly heading towards we bring it all up once it's there that's what's happening and it will stay that way immutable infrastructure is I think kind of really handy to be able to think about a system as this is a load balancer and when I say that that is all it does it does nothing more and yes maybe it has a sidecar that deals with you know offloading the logging but what else do I want my load balancer to be doing which unfortunately or fortunately means I'm conceptually heading towards unicorns who here likes the idea of unique owners why does anybody have a glib answer for why other than what I've just yeah true yes yes then mind you they've of course improved that in the kernel where now pending data between user space and kernel spaces is optimized but nonetheless do I need a generic operating system in my router switch or whatever yeah and and I mean yes I like open wrt so you know you've got that dichotomy of it is easier to play with something that's not a eunuch kernel but if we're deploying systems do we want to be able to play with it or do we want to know that that is running what we are what we expect it to be running and so to me it becomes easier to to reason about so one of the things that came out of dock up doing the succession from docker toolbox boot to doctrines on is that they started working on docker for Mac and dock for Windows and they built their own micro Linux distribution building thing and they went the whole hog and just said okay there's no mucking around all we do is create a file system out of containers and then we run them in containers which sounds ideal I'm dill so I have a whole pile of tabs here wonderful where did they go anybody know how to actually drive this magic yeah no you're right I'm just color blind now okay here we go this is how to define a Linux kit micro Linux distro this one is intended to run docker but at the moment there's something broken on my system I think and so it goes through figures out I want to use this kernel and I'm going to have an init set of processes on my hard so the unit ID is initially created out that innit section and then we've got a series of on boot and services containers they get run using run see and in the on boat section or used to be and then services which get run using container d and at the end you should end up with a terminal you can run docker run at I don't find this particularly readable but it is very specific and tells you exactly which images it's going to run and you can link them up so that's what you would be interacting with and as an example of their Annette a little more painful because they go off and try to do a trick to allow a docker to pivot route and so the first Annette calls another in it which then basically just goes off and calls this no I've lost my marbles so in in in many ways I think what I was trying to say with this is that Linux kid being an evolution of all of these ideas it's a net process is simpler to understand again than Rancher OS was and I think easier to understand in the scripted way and I can't say much about system D because I don't I've spent so much time not looking at system D but when I do have to play the system D system I'm not sure what's going on well I assume that people who know what they're doing with it can figure out what order things are running in by looking at the config and can debug that and I just kind of you know I I haven't tried enough so I I can't make a judgment on that you know people say they don't like system D I kind of don't like this even though it does what it does remarkably well okay and then once you run docker in qmu you can as in this image you can then go off and run docker inside this virtual machine I won't do it again because I really don't want to spend a few seconds minutes no the downside with Linux kit is that you saw from the ml file you need to not only keep up with everything so you know you need to know when there's vulnerability in Linux you need to know when there's a vulnerability in any of those containers and you need to be able to figure out how to take those goods or whatever they are and figure out what you're actually running so you need to be more and more aware of what's going on and I suppose that's actually just made more obvious by a micro Linux distro because we're putting it in your face and saying you need to choose when to upgrade whereas if you're running a Bunter on your server you're sort of outsourcing those decisions and waiting until our bun to has effects and just installing it in reality we all probably should be keeping ourselves aware of all of those vulnerabilities anyway so we know when to switch our computer off and wait for a fix at the same time so it's it's hard it's really easy to poke holes in these these linux's because they make some of the problems that we have front and center and really obvious because now you're in charge of putting these things together and I think that's also one of the things that juju or you know those Siddhi building tools can be doing if you're using them but most of us avoid it we just installed Debian as I have on this box and try to remind ourselves to apt-get update regularly enough okay I think I covered that but essentially because of the way that docker was split up we can bootstrap our operating systems and I ended up doing Rancher OS demo version using bitter linux kit and rewriting it's in it to do the same sort of thing as Linux kit and the nice thing that came out of that was let's see am i able to show this was a lovely ancient tool called boot cut that goes through and shows how long it took what parts of your boot process to run now this is probably something we should all be doing more of because as I was getting to this one this is the last iteration I did before I gave the talk there's a whole stack of things that all of our operating systems including the one I was working on are doing that you probably don't need and with the bun toe you're probably going to find the same thing and there was a talk at linux conf in prague about embedded things where the guy giving the talk whose name escapes me showed that there were three seconds of his embedded linux OS spent waiting for butter FS to decide there was no raid on the system have and I'm sure that in some of the Colonel's I've built I've been built guilty of building in butter FS as well III assume that there are ways to configure butter FS so it doesn't do that but his of course you know embedded situation he was just going why do I have butter FS remove it and suddenly his boot times went down to somewhere under a second I think and we're under a second you know that the the idea of measuring this is that you want to go from turn it on to it's doing what you need it to be doing in as little part time as possible you know the the time to get to an it is part of that and so I think in this example container d and whatever it was that I was running my tamina shell or so is ready before that last uptick so it was somewhere around the eight nine second mark which is kind of awful but a heck of a lot better than the forty or so seconds and I think it was when I started because Linux kit and rancher OS do things differently from each other clearly never do that okay all right now as so many of you put your hand up as as being I have tried go we go even more extreme there is now a really small little tool that some I think they're mostly Google developers have created so that you can create your own in it Ram FS directly from ghost source code and so what this example is doing is it's taking the gyro commands and they've rewritten a whole stack of the UNIX toolkit in go or simplified versions and it adds a shell that is slightly nicer than the the built in shell and it goes off and build something which you can then run does anybody know where I am yes okay I'm gonna let that go it doesn't take that long but the thing that I thought is the most remarkable is that they're in it process does a little bit of setting up for some of them magical talked about later and then just goes through and runs a loop starting things where am I so we mount some file system bits in and here we go you're going through this list of processes to spawn that's it so it is doing the bare minimum that you need to get a Linux system up and then starting essentially the first two the first one's legacy the second one is a user in it so that you can add stuff yourself and the third one is their basic shell and that's it and once it exits unfortunately the Linux kernel crashes so this is in many ways the simplest little thing that you could have as an init and it doesn't take long I always forget we're ready already and there we go no no this is really slow if you've ever played with clear Linux it does get to that point even faster because what they're doing is using processor isolation things in a slightly customized qmu although that might be upstream by now and it's like it's container fast the only problem is you've got to trust Intel and we've had a few things in the last couple of months which have made me re-evaluate that choice so yeah okay the fun thing about no this is this is this is the downside of go apparently they have their own flag pausing library and despite the fact that we've hated it for a long time it's still the way it is because plan nine it's probably the thing I detest the most about to go thank you alright alright I yep shown that yes okay no I think I'll just skip this and say this is how you would run up a container doing the same thing with you but route would be make build in the right place no I am NOT in the wrong place how do we get there who did this to me okay whoever organized my you know my computer they're fired me okay make build alpine okay so this is a very Dockery kind of thing you have clever I got basically what I'm doing is I'm getting the alpine container and exporting its file system into the route FS directory whacking that into a cpio which you root will then use as base so I'm going to get a new route made in it ID file system that in this case probably doesn't contain quite enough pieces of you route so I won't be able to get Network I think we'll see all right it's kind of big I'm sorry who was that enough yes it was so what I essentially have is right it is actually fetching so there we go so if you look at that list again it is mostly nothing great - the that's it that's all I'm running and it acts like an Alpine it's just I've got none of the things running which means I don't have d bus and so on for better or worse okay can I remember how to shut this down yeah cool all right and that's pretty much how I created that guy yes it is clear containers clear containers you can go docker run with whatever the parameter is to select the the run C equivalent in clear containers and it will go off and do exactly this kind of thing but then it'll also mount in file systems when you do buy in mountain or sort of clever things to work just like docker it's just a virtual machine alright so as you saw I can't remember how you rude has implemented shutdown I can't remember the flags so that was the first thing I thought I'd go off and add to my new in it so I've made a user in it that just goes through and runs date DHCP client starts a shell and when the shell exits run shutdown kind of a lot more like the way that a docker container works all right and that should be let's do this in Debian I'm sure this will work every time so here I've made my CPI oh and then I'm building in the DHT our client that comes with you root adding elvish and my you in it and that's built so now I should go make run unfortunately unpacking takes longer that's life okay so we now have an IP address and I have a shell and if I control D from this things don't quite work yeah that's just cruel very cruel yes yes yes that's right I yeah I just killed my own example Corral okay just then yes no no no no so the this compilation was when I type make build Debian it invokes you root which invokes the go compiler and does the whole thing I was going to do this with containers but things go and fall over right okay before I get to that another feature that I haven't talked about with you root because I didn't get it working in time for the talk is that they also have a source code mode where they build this init ID with the go tool chain in it and the source code to all of their commands and whatever else you link in and then when you execute those things it will build them and obviously store them so that next time you execute them they're even faster which I'm not sure what they're using it for but I'm kind of thinking instead of having my cloud in it in shell I can now have my cloud in it and go in fact that you in it that I showed you before could actually be my cloud in it again we're not immutable you know there are obviously two different use cases I think it's it's a cool example of how simple and quick and easy things are now creating this sort of thing 10 years ago was fun you know a lot of us played with Linux from scratch and you learned a lot but it's something that very few people did whereas with something like your root and possibly Linux kit you can take your small single application and run it so one of the things I'm going to do is as I said make a load balancer container with a log distribution beastie in it and that is all it will run and then I've got to figure out how to get kubernetes to not hate me okay I'm gonna segue a little bit docker machine who he has used or does use docker machine very few Wow the way the world change is cool um doc machine is stagnated a bit in the recent years and I was kind of thinking if people are interested in dock machine there are some people who are getting together and pulling in all the drivers into a github organization so that we can breathe some life back into this and one of the reasons is because mini cube and some of the other kubernetes tools actually use the dock machine drivers and so they're still being used and we need to start giving them some love because they're not part of Dockers current commercial priorities cool thank you astoundingly I think at the moment they're cheating in that they're not using a boot yep okay so you wrote is you root combining the the traditional bootloader like grub and so on and unit and the answer is no they're kind of avoiding it right now and we start qmu by saying there's your in it Rd there's you colonel go theoretically [Applause]

Channel: LinuxConfAu 2018 - Sydney, Australia

Views: 1,247

Rating: 4.4666667 out of 5

Keywords: lca, lca2018, #linux.conf.au#linux#foss#opensource, SvenDowideit

Id: WYtj7t2h5Kc

Length: 44min 27sec (2667 seconds)

Published: Tue Feb 20 2018