David Hand _ "Linux initramfs for fun, and, uh..."

hello my name is David hand my online name is Telemark which of course is spelled exactly how it sounds and this is my very first talk at a Yap tea so yeah that was I'm glad that I got that response now because yeah later I mean no okay so this is on the the Linux in a tram FS for fun and um well it's about the Linux in it rim FS four for four anyway it's about the Linux initial filesystem so EMFs is a small file system embedded in the Linux kernel image it helps the kernel to mount the root filesystem so the x86 boot process let's talk about that for just a second so you start with the you it the UEFI firmware we we used to call out the BIOS that hands off to a bootloader that - unfortunately part of this is not displaying on my presenter console that passes off to the kernel which has within it at the initial Ram filesystem and that gets you to PID 1 which is usually in it or system D or something like that so let's take them step by step so UEFI firmware here's a diagram of it don't worry I will I'll cover this you know throughout the talk you all go step by step just kidding them that's the last we talked about the firmware so that passes off to the bootloader now people have a hard time with the people have given you at UEFI a very hard time and there are some good reasons for that but most of those have to do is secure boot efi is actually really really easy so i want to just mention this real quick because it really really is easy so the important thing is you did it boots from instead of instead of with the bios where it's like this weird thing that's that should have been part of the file system but we didn't use it for the file system so we're using it for this other thing EF I looks for a very particular partition it has to have so you have to have a pretty sure a particular partition type which is GPT it looks for a very particular fat32 partition with particular flags and then it finds a file and by default that's where it's located and then it just runs that file it's just an elf executable with with it with it with extra headers so the boot loader loads the kernel and here is the kernel we will let's see we can kind of scroll through and look at you know the net no we're not going to do that so why is that not pardon me okay so that hands off to the colonel the colonel has inside of it it via knit rhyme FS let's pause on that for a second first what does the anit Rama fest need to do on the other end so it passes off to PID 1 PID 1 is not part of the colonel right it's just a normal user space program it needs to have everything like it needs to have present everything it's going to need in order to be able to run now what does PID need in order to be able to run basically everything at first at least it needs it needs a a full system available to it right then now it doesn't need for example your home directories right in fact it's part of its job to load your the the partition on which your home directories live but it needs more or less the full file system before it can start so the net Rama festa job is to provide that full file system so again what's the inner m FS it is a small file system embedded in the Linux kernel image that helps the kernel to Malthe root filesystem so let's talk about that part so why would you what where is the root filesystem usually located right in the normal case there right it's just that's and in that case you probably don't need an amethyst you just need to pass the partition name to your kernel so but what if it's more complicated right so you could use NFS this is pixie ok pixie is a system by which let's see how if I can do this so you you can sort the the firm with the firmware consults a particular kind of DHCP daemon the DHCP returns an address it tells you where you could you should actually load the kernel image from and it does that but it can only one load one file right you don't need an amethyst for this but you're gonna need it in it Ram FS for the next step because pixie has only handed you one file now Vienna tram FS is contained within that file so that's pretty pretty darn handy and and from then you're going to want to mount whatever you're going to want to mount but it's a boot from pixie for example you're gonna need an Internet RAM F s probably a custom one so here's another reason so I like messing with my laptop and I had a laptop with three hard drives which is a little bit weird so the the first hard drive SDA was a fairly fast SSD SDB and SDC were larger spinning rust drives right and one it was my first Mac I'm sorry is my first Linux computer after having a had a Mac and I was what jealous of the full disk encryption so I wanted to you know I wanted that so what I did was I put Luc's which is the the encrypted filesystem thing for Linux and it actually doesn't expose a filesystem it exposes a partition again right so on top of that I sat LVM and then on top of that a whole bunch of weird partitions some of which spanned the SDB and SCC some was striping for some of them not oh no I just messed up your mess around with it a logical was probably the probably not quite the right word for it so again what's in it Ram FS it is a small file system embedded in the Linux kernel image that helps the kernel to mount the root filesystem let's talk about the embedding in the Linux kernel image so so who would like to watch me a compile a kernel No so how does it get into the kernel loan by being compiled in so this is make menu config I have gone into general setup and I don't know halfway down we have initial RAM file system and ran RAM disk support so if I if I if I select that and then I go to ram if a in a tram ifs source files this is it and that's really friendly the help well I guess that clears it right up yeah yeah you know the CPA i/o archive with a CPI or suffix for God's sake anyway so basically though it comes down to these two configure config variables config block Devon a tardy and config in an RMF s source now for just a second let's talk about a synonym of RAM FS that's in it Rd it's kind of not a synonym so in it Rd used to be well yes so so in there in it Rd used to be these files that sat in your boot partition and the kernel would load them now I say used to but that's that's actually my boot directory right now okay it's still there and for some reason when we split off the interim FS into its own file which you can do it's called an init Rd people people tend to call it and in it and in it Rd that said I mention that it's not quite the same thing because if you end up digging into the documentation you should know that init Rd has a slightly different API under the hood so okay the the the secret about the in it Ram FS is there's always an Aram FS so the weather there whether or not you compile one in whether or not you turn on that one variable there's always in a net Ram FS it may be null but it's still present so okay Nara MF s is a small file system embedded in the Linux kernel image that helps the kernel to mount the root filesystem so in what way is a small file system embedded in the kernel image now I don't just mean the kernel runs it that's true from all four alpha well for most file systems and I don't mean that like I don't know I mean it's so but what did you mean is that it's actually in the kernel image it's in the kernel file unless you split it out but even then it loads in it loads it into Ram in kernel space in the kernel image so let's talk about what that is so in an it Rama fess it is a ram disk no the RAM disk is an old thing it's there so available but Ram disks was the RAM disk was a way of carving off a a block device out of RAM and then what you could do things with it like you could do to block devices for example you could write a file system to it right but with a with a ram disk you would create the device and then you would like make FS something right you would actually have to you would have to format it as a disk so Ram disk is cool for what it does but that's not what we use so there's this other cap thing called RAM FS Rama FS is really really slick so it is not a block device it is I mean it's a file system and that's cool also because you don't actually have to format it it is already a file system it's also kind of neat because it live is like it it's built out of the thing that caches files when you read when you read them and write them in Linux right it automatically caches the writes and you know the reads it might they might save those for you right in the buffers so Ram FS is actually just that only there's no underlying file system okay temp FS tempeh fest is basically Rama fest but you can put limits on it and you can like let users do things with it also okay you can you know you can have a you can make it so the users come out with their own little time professors so rou defense route FS is a tempeh fess or amethyst that exists even if you haven't installed and like even if you haven't done anything else it is there before you've mounted your own route there's already root of s in fact after you've mounted your own route route if s is still there you just mount over it so in it ram FS oh yes in amethyst then well there's always a route if s also right so okay in amethyst sits on top of it so I said that there were these two variables and one of them you just turn on this other one can they config in it Ram FS source so you can give it directly a cpio file so let's talk about that CPI oh it's not tar it is completely something else and it's a bit weird I don't know well I can tell you why it's it was chosen over tar but I wasn't convinced but anyway it was so apparently the tar format is actually even uglier than cpio or at least according to the kernel developers so you can give it a CPI Oh file you can give it to subject eree you can or you can give it more than one directory or you can give it a specification file so if you give it a directory you are of course I mean you're you're obviously trying to well me back up the inner NFS is also kind of a full complete system in particular it needs dev right it needs dev TTY otherwise where is it going to open anything it needs let's see all sorts of other all sorts of other special files now you are obviously compiling your kernel as a non root user right okay well you should be and if you do it's hard to have a rectory with special files in it that you've not that you've created as a non root user right in fact the special word for hard is impossible I think so instead you can use what's called a specification file so that's what I prefer to do now the thing is you can also have a mixture well you can have a mixture of either of the either the last two and they're processed in order for example one thing you can do is just have a raw directory with all the files we need it okay and then have a specification file that applies the the special files and whatever what's it called owners right the ownership of all of those files and whatever so I actually prefer to use a specification file entirely so this is my specification file and we are this should have a 10 20 minute talk so so so this is my specification my specifications file I apologize that it's a little bit hard to read so you can see that we we it's just like what kind of file where's it gonna be whatever what are its permissions and who owns it there are a bunch of them because I have lots of stuff ignore the microcode thing so we you start over when you're starting you don't have Devon all you have to say I need Devon all right you don't have def zero you def men which is unfortunate so you don't have deaths standard out unfortunately so now what goes in the net ram FS right what is it supposed to do first it needs to mount all of the file systems that are necessary for PID one to run right that is its one core responsibility incidentally it can do all sorts of other things while you're doing that you could have I don't know Tetris loading while you download your Tetris playing why you download your your your root filesystem from 4chan to I don't know you know right you know over I don't know you know HTTP but not HTTP right just to make sure you know so but you so you can do anything you want is the netra methis is a full user space okay now what is in that user space in it okay now this is not the same in it as before oh I'm sorry pardon me this is not the same in it as later so this is not the in it that is that is PID one on your system though this is PID one okay at this particular moment and it can it can be in any format that the kernel can run importantly you see this shebang line surveying bin busybox right the kernel is what interprets that shebang line right it's not for example the shell right so you make sure that you have an interpreter in your in your NFS and then you have a script that uses that interpreter right and you can move on move on from there so this is this is the init that I made it it's a little bit weird but let's jump to the end so do you see this is X which I apologize that it's small you see this is the exec switch root target in it okay you may remember what exec is perhaps the exact built-in and perl exec does not so much start start another program has become that another program right so this PID one is going to be your new PID one it's just it's going to be different code but this that line there is that is the line that does that so switch route let's talk about that for a second switch routes a bit weird I said that you mount a different route on top of this one right cool where does this one go and what do I do in the meantime but I don't have like I'm kind of between routes or so fortunately you don't have to care switch I mean you can switch route however it does that right so it accepts a a new part so it expects something else too already mounted at Target okay and then once that happens it expects something relative to target mind you like relative to target as the new route it expects something to run so that's the init okay so that's that would be for example system D okay in all likelihood so um and I apologize whips mmm this is my last slide because I yeah but um but there's a lot to talk about in it so okay so in here the main thing that you need to do so I this is actually the in it that I used to do that weird what illogical volume management right so here I am actually decrypting well that's actually okay I it was a little bit weirder than that so you don't hear my name rock a so I had a weird USB key it had a it had the decryption key for the rest of the system it was itself encrypted I had to type in the password what with the key in then I had to remove the key it would decrypt the the rest of the system and then boot I'm not so much paranoid as just bored so my and my yak is so very smooth so so this is how it handles the USB key then this is how it handles the the decryption key on the USB key it forced me to I it actually forced me to remove it so because I wanted to make sure that like I don't know yeah then it's another vector for attack and then now that's decrypted everything it well okay no I'm sorry this is the decryption activates LVM oh we should talk about hibernation in a second actively so VM and then mounts everything right we're gonna skip that and then well and then we move a couple of things for example you know we've we've had to mount dev right we you needed dev right but we are going to also need that Devon the new system so we just move it right so there's certain things like that at that point we have a full root filesystem right we have root we have user right we have everything this just indeed needs in order to start let's talk about a couple of the weirder things so I'm sorry just a second so restoring from hibernation if you want to restore from hibernation you need an init Ram FS that said if you're running Red Hat Gen 2 whatever they are I'm sorry Ubuntu or whatever they provide you in a tram FS and it has it expects to be able to restore you from hibernation this is what you do if you're writing your own you basically just find you you find the major and minor number of the swap partition on which the hibernation and the hibernation image is stored and you pass it to a file system rescue it's actually it's kind of odd it's a it's it's this weird thing where you're expecting it to be harder than it was so if you don't be like if you pass it something nonsense it just doesn't do anything and then you proceed on you're in it Ram FS and right if it does actually have a hibernation image on it this stops executing right because you're now just loading something completely else into RAM with it with its own state and it just it just starts from there so it's actually pretty slick that is the basics of this init script you see though like I said that it's not a binary right youyou kind of expected something so low level to be how these that you have to write write it and see and you know and you certainly can but you can just use a you can just use an interpreted language I next year maybe I'll show you an in amethyst init script written in Perl 6 you know I don't know there's the in principle there's no reason why you can't actually I should pause for a second and say and go back here oops pardon me so you may object if you know anything about in it REM FS you may object when I say that you could use Perl 6 right because how are you going to statically linked Perl 6 are you going to make one monolithic Perl 6 binary how can you use shared libraries in an Internet in an immigrant s right whenever anybody talks about an 8 Rama fast they talk about you know statically linked you know we don't have to we can't deal with libraries it's false right it is more convenient if you have a statically linked single binary but only because you you only have to include few like it requires fewer lines in this line or in this file so in fact here are the dynamically linked libraries that everything else requires there is a fairly simple command LD tree that you can use to to find out what the libraries are that are required by your binary and you just include them simply so yeah I don't know I apologize my my discount is perhaps a bit more awkward than my my start but this is pretty much it is anybody have any questions [Applause] so I think the question is isn't an IMF s required just a boot from USB No so right right right so if your if your file system is local right and that includes USB then it is not necessary to have an in rem FS technically there is still one and what I believe what it what it has in it is let's find the first the first partition and of the first Drive and just try it right if you don't want to do that you pass along the partition and the drive as a boot parameter right now if you're booting from USB in all likelihood what you've done is you've told your boot loader to look for the EFI boot image on USB ok in especially if you have a more modern computer that that does not have the old weird BIOS way of doing things as a fallback okay some computers will try one and try the other ok if you are trying to boot and you don't have the the old the old way of working then basically your USB needs to be partitioned with with GPT it needs to have a fat32 filesystem with those with those particular boot Flags enabled and then you have to have something in that 'men aimed that in that directory right but you but that on its own does not require an idiot Ranma fess yes I do not believe there is one that is or that is to say I'm sure that there is one but it's probably something like you know it has to do with like buffer overflow on a 64-bit inch or something right I had the NIT Rama fest the main problem that I've had with size on my ram FS is that I was silly and I made my boot partition too small and I wanted to keep too many backup kernels right so as a practical matter it was just you know I couldn't store them all so I was trying to shrink my in it rhyme is a bit that said do you run Linux so you notice how as you're booting it's very rare that you actually see a bunch of boots scrolling messages go by right that's because there is a program called I'll come back to that I'll try to read that but there is a program that provides you know a splash image right that like I say Anna FS can be arbitrarily complicated you can do absolutely anything right you could have Firefox running right because you have all the libraries right you just make sure that you put all the libraries in your FS right you could have Firefox and it's like here you browse Facebook while I I don't know find an image off of you you see P you know right so you see what I'm saying so there is no particular size or complexity limit that's that's given to you by outside of for you're in it Ram FS that's an excellent example thank you yeah Libby let me repeat that for the for the record so yes so if for example your scuzzy drivers are not compiled directly into your into your kernel and one reason for that could be because they're binaries right and therefore it's illegal or you know a violation of copyright violation of the GPL - it's a good signal it says it's compiled them directly into the kernel right so they have to be shared like what does it modules but there's a there's another word besides before module anyway but but you know separate separate modules that are not cloud directly into the kernel loadable there and yet they have to be loaded into the kernel in order for you to mount your weird scuzzy system with its proprietary drivers then you need to put that module in near an it Ram FS and actually in my example that was a huge lacunae so thank you for pointing that out because usually actually in an interim FS is most frequently used for loading kernel modules so in particular it has to be there to load kernel modules for things that are going to be required to boot another example is sometimes even you could theoretically have you have your USB support on a kernel module in which case you wouldn't need needin it Ram FS - booed off USB though hopefully you're in it Rama fest isn't on the USB let's see in fact actually my the example in which I was mounting off of you know LVM on top of luke's is actually a special case of of the case that was just given right so this is a situation in which the you know the what is it the serial ata drivers on which my actual drives depend those are built compiled into the kernel but the data that they contain is this weird format well it's you know it's it's it's you know statistically random noise right and tell you until you load the particular program that can they can read it and then on top of that you have yet another thing right so that's actually an analogous thing to needing to load your schedule divers so okay right so well actually I should make something clear as I said and I tried to make this diagram reflect this the EFI boot partition is not part of the encrypted space right that has to be a normal unencrypted fat32 partition sitting in a GPT partition table on a normal spinning disk or you know or whatever it doesn't have to be spinning that you know normal a normal Drive right so to answer your question about how to how to restore it so in fact this is something I had to do the the computer was essentially destroyed by being in check luggage and being thrown by anyway that was bad idea but um so what did I do right so I pulled the I pulled the drive out I went into the efi system partition got got my kernel image right my inator FS was actually in the kernel rather than a separate file right that would have been a little bit easier it turns out that using a very strange like cpio output command that I had to look up you can extract the inner manifest from the kernel at which point I have the decryption key for the for my external USB key right so then I decrypted that got the decryption key off of that for the for the rest of the systems and then by God copied those in plaintext somewhere right so that I could actually you know restore without having to do this every time buddy yes sure well I yeah I'll do that without actually using a slide so the slide was no helpful anyway so CPI oh well actually I guess which cpio thing do you mean why cpio Rev and tar it is it's it is it is it's older Sagan yes sorry so what is CPI oh and they're rare so what a CPI oh and why it was it used rather than tar for this format the answer is that it's older I don't know a lot of details about it but it is considered to be more regular than tar apparently tar is actually not a format as such it is a command that reads what it reads right so yeah go ahead you have you have it you have a way of answering this everybody's fine the end would sort of do everyone the CBI over liked our recipe directory very glowing it takes illusive wireless let's interview and then ranks without when they did you see format uses more modern porosity solve this with targeted backup place files or symbols which yes that is right thank you very much so yes the the so there there there there are a few important things in what mr. Lombardo said but the the part that if it is perhaps the most important is that tar can't really tore up a device special file right and cpio has a way of representing that it should be a particular device special file with particular major and minor number and particular permissions and all of this so that is of course essential for the in it Ram FS if that is it then I'm going to get out of here and drink something [Applause] you

Channel: Conference in the Cloud! A Perl and Raku Conf

Views: 11,455

Keywords: YAPCNA, Perl

Id: KQjRnuwb7is

Length: 36min 52sec (2212 seconds)

Published: Sun Jul 01 2018