Code Tour: Serenity OS startup (from boot to GUI)

well hello friends today I thought I would give a little overview of what happens when serenity boots and sort of go through step by step the boot process and how the kernel starts up and sort of what happens initially until we get to the GUI because it's something that a lot of people have requested that I talk about and it's something that it's probably and I always like to think that the system is very approachable and understandable but it's probably much easier to follow along if I just get you started so this one is for all of you curious people who reached out to ask about this so I don't have a plan so so I never make plans so we'll just uh would just boot up once just to see what it looks like so you can see there's all this spam happening here right in the terminal as we boot up so the very first thing all of this gunk this is just the emulator that I'm running in QT mu is just o telling us that there's this been a CPU reset and blah blah blah and then here the blue stuff is basically the blue stuff is from the kernel k printf so if you call the K printf function in the kernel mm-hmm then the first thing it does it cost color on and then when it leaves it calls color off and this basically just prints out a little escape sequence that makes the output blue if viewed on the terminal and this is just to help me sort of distinguish but this is a message that comes from k / def but anyways it's not super important but the very first thing that happens you can see here is that it figures out what year month and date is but instead of looking at these messages let's look at the code so the very first thing that happens when I run the run script' is what it does it actually runs a queue EMU command and it's this particular one here and as all these different settings and stuff but the important part is that we're using a feature of cue EMU where you pass in a kernel and in disk image so ki-moon has a - kernel option and basically if your kernel has a so-called multi boot header which is a boot protocol header that is used by the grub boot loader then queue EMU is able to boot that kernel without the presence of a boot loader and the reason that I use this is because it makes it makes building and running really really fast because I don't have to go through the extra step of creating a disk image and installing a boot loader and installing serenity in that boot loader instead I can just tell QT move to act like a boot loader and it boots up like really really fast right so oh and we're up and running now I actually I also we also have another script here we have built image chemo and then we also have build image grub if you wanted to use the grub bootloader you can use this to create an image that you can then write on to physical disk if you want to boot serenity on real hardware for instance you can use this I wanted a hell it just helps you get a disk image with grub on so I mentioned the multi boot header so multi boot is a protocol that is defined I think originally by the grub bootloader and I'm not sure if it's very well supported by other boot owners but if we look for multi boot protocol multi boot specification yeah it's it's a new thing and we're using this version of it 0.6 and it it takes care of a bunch of things for us that we don't have to deal with and the most important things are use it to set the screen resolution and we also use it to figure out what kind of memory the system ask because figuring out what kind of memory a PC has is a tedious task and the bootloader this loader protocol already knows how to do that so we just say hey if you if you already figured out what kind of memory there is why don't you just pass us that information anyway and then the video mode actually let me show you the boot header so it's in kernel boot boot no oh it was moved it's an arch I 3d6 boot boot s there it is and here is the multi boot header that I was talking about and basically before or rather it's down here these are just macros more or less or definitions so here's the header and we tell it that we want the memory information and we wanted to set the video mode and this allows us to not have to worry about setting up the graphical video mode so down here we specified what video move you wanted to boot in and we're asking for a 32-bit color and this this just makes things easy going forward we're gonna have to implement a bit more stuff like to get better video drivers but right now I've been using this and it just slaps you into that mode the big downside of this is that if you want to be in a different mode you gotta edit you gotta edit these headers you know if you want to be in 1024 by 768 you got to go here and edit this so this is not super convenient and there are BIOS calls that you can do where you asked graphics BIOS to set a different visa mode but in order to call those BIOS operations you have to you cannot be in 33 but protected mode which the kernel runs in so you would have to either run in virtual 8086 or you have to swish out the real mode called a reason interrupt and then switch back to protected mode so that's a big mess but it's something that is being worked on casually and we'll see how that goes anyways so this is basically information for the bootloader so the bootloader will locate with this header by like looking through the kernel that we pass it and it finds these things and it says oh you want this video mode fine oh you want to get the memory info fine I will populate these will get populated with memory information from the bootloader and then the bootloader will basically you send us here this is where we start start execution so the first thing we do is we just set up interrupt every clear the interrupt in Direction flags and we set up in initial kernel stack which I think we predefined some spacer 32k here yeah just an initial stack and then we push the multi boot the pointer to the multi boot data structure onto the stack and that allows us to wait why am I doing this way I'm pushing it but then also putting it in this okay maybe I don't need to do this actually but I'm not here to edit code I'm just here to talk about how it works so this is the important part we populate this global pointer to multi boot info pointer we put the multi boot information in there and then we call init and then if in it whatever return we would print a little exit message here Arnold exited and we print that and then halt the machine but we never actually returned the minute button it is in C++ so if we go to and it's CPP it's this monstrously large function or file rather actually it's not that monstrous it's just a couple of hundred lines Wow I guess it feels monsters to me because I don't know why so here is where we enter into C++ land for the first time so we go from this boot dot s and we jump to in it a chinos and the first thing we do is we check if we have a serial debug command line option set to the kernel because if so we want to start streaming all of the kernel output to the serial port because obviously like if you connected a serial port for debugging purposes you want everything to come on there and you don't want to miss anything so that's the very first thing we do and then after that we call SSE in it which is basically just sets up the CPU so that it allows use of SIMD instructions with the intel's SSE instructions and the reason that I do this is because the ports of GCC that we build for serenity it ends up using SSE instructions so if we don't do this then GCC just seg faults and it will be possible to you could you could run serenity without SSE support and if you wanted to but since every machine that I've tried to use it on so far has had as a Z anyway it just seems silly not to to turn it on and yeah so after we do that the second thing is that we initialize came a lock and K malloc is a well it was supposed to be if you look at the very first comment here it was supposed to be a really really really quick and dirty malloc and free implementation just to get the kernel going and I wanted to get rid of this and replace it with something nicer before I ever what I was working on but then I forgot about that I forgot about this comment and then I just left a comment there because now it's kind of funny but yes okay Malik is a quick and dirty first fit allocator with a static pre-allocated buffer it allocates one megabyte total of memory and then it has a bitmap where track it's divided into chunks and each chunk is eight bytes and yeah you can see it's one megabyte at the whole pool each chunk is eight bytes and then we have the allocation map which is one bit per chunk and it just tells you if that chunk is used or not used and then came out Locke will search the alloc map for like if you want to allocate 32 bytes for instance that would be four chunks so then it will look thicker and search through outlook map linear scan it until it finds for sequential bits that are all zeros that means that that course wants to an address where 32 bytes are available so that's how it works and yeah that's basically that and then I mean there's there's a little more to it than that and it has a you micro optimizations that make it go a little bit faster than maybe it should but it's it's definitely a very crappy allocator and the size limit is constantly causing trouble because it's very easy to boom or like get an out of memory panic and the kernel because we exhaust this 1 megabyte if came out with memory then there's also this concept of K malloc eternal which is a separate allocation function and K malloc eternal is specifically for allocating memory that you're never gonna free so it is an extremely simple pump pointer allocator that has the next eternal pointer and it will just whenever you allocate with camel hawk eternal it will just give you this pointer and then increment that pointer by the size that you've just allocated and we have a total of two megabytes available in the eternal memory space and I think it starts now it comes before a kin malloc range anyways so that's what that's for so those are the two came out like alligators they're the old ones and then recently I added a new alligator called a slab alligator which that basically it's used for specific things at specific sizes so it's an alligator that when you call slab a lock if your size is one of these and your size has to be less than 48 bytes or equal to less than or equal to 48 bytes and then we'll find you one of these slabs so slab allocation is all about having a lot a large cache of same size allocations that you can very easily reuse because then we don't have to look and find we don't have to look through like this big bitmap of the available space to see if we can find a fit somewhere because the slab allocator is essentially just a free list of slabs until you have allocated everything from it so this is very useful for some of our memory management objects because we allocate a ton of them especially when you're doing something like running GCC it ends up very aggressively allocating stuff and they end up coming out of these especially Slovak iterate that's the 8 byte per slab we have we were served three hundred three hundred and eighty four kilobytes for eight bytes laps and it's very very busy alligator and this this was just a way to avoid having to go to came out look for these things because this is all of one rather than event so it's it's much much faster than que Malik if you know that you need a specific size that we can accommodate anyways after we initialize the two alligators and then the next thing we do is we initialize the Kasem's and Kasem's is the symbols of the kernel and those are stored in a file actually it's probably something that could be done differently and I realize now that that was just initializing we're not loading them we're gonna go we're gonna load them later so hold on we're just clearing out these things apparently I don't know why this is done in a function I think I probably added this before this was before startup time constructors were possible under Karl and before the kernel VSS was initialized actually start up time constructors are not possible on the kernel yet but the BSS agent is initialized to zero and what that means is that if these are like static file or like Global's basically and previously these would just have garbage data in them because we didn't zero out the memory segments that contained these things but now that we do that it's actually not necessary to have a function that's called on boot that sets these two zeros although this one is not set to a zero so maybe there's still some value in having it but they're the absolutely right thing to do here is to implement startup time constructors because then we could well this shouldn't actually need construction because this should just be regular data segment stuff okay anyways this is probably totally unnecessary and could be just we could just initialize them up there but that's what it does it just initializes the stuff related to kernel symbols won't get to the kernel symbols in a moment okay and then we create the K prime object and that's just a convenient object that allows you to check if a particular option is present on the colonel command line so the colonel command-line can contain stuff like serial the bug which hole causes to run to send all the output to the serial port and text about which will cause us to start the kernel in text mode and we won't start the Windows Server but instead instead we'll just start a shell on the console so this is just forgetting to debugging the console where I eat quickly was added earlier today actually bye bye they have also known as supercomputer seven and anyways so yeah that's just what that's for and then here we just say if this thing is set it it's kind of weird I should probably just use has this should be like this instead anyway I'm not gonna edit now okay I got it gotta focus on what I'm trying to do okay so after we've basically parsed the kernel command-line then we will create the VFS and a VFS is the virtual file system and it is it's like the mother object of everything file system related so the VFS is responsible for the root directory and everything that has to do with files where you just have a file name and you want to go to you wanna you want to like turn a file name into something tangible like a file or a device or something you have to go through the VFS because the VFS knows which file systems are mounted where it knows which device numbers correspond to which physical devices stuff like that like the VFS knows all that so that's what we instantiate it here and then debug log device that's just a silly little neat one log that i said i don't have to talk about that one and then we instantiate the console which is very very old abstraction that it's very outdated but it's an it was an early development abstraction where I think I wanted to have some kind of objects that's responsible for receiving kernel messages so that you could say now I'm on the Virtual Console number one then it is the console and the kernel output goes there and if I switch terminals then the kernel output would go somewhere else and you could potentially set it up so the kernel messages go to a log file or whatever but that's what that was about and it's still there and I forgot about it and then comes a lot of device initialization so the first thing we do is to real-time clock and that's actually the first thing that we saw there in the debug output let me boot it's this right here what year month and day it is so the very first thing that prints apparently prints anything for me is the RTC the real-time clock and it's right here we compute what time is it now and then we initialize the programmable interrupt controller and then we set up a global descriptor table and interrupt descriptor table this is just setting up like the x86 CPU stuff so that like when a certain irq or interrupts happens we have a callback that gets invoked and is able to deal with that and this just sets up all those mappings and stuff basically and then now actually come to devices so initialize the keyboard the ps2 mouse the sandblaster 16 they have no very important device then a bunch of serial ports and then we make for virtual consoles and you can actually if you're running in text mode which you know most people never runs already in text mode where you actually you can and you can switch between the consoles with alt one and all to about three and so on and it's pretty neat and then here of course we say starting surrounding the operating system after that we initialize the memory manager and the memory manager is basically responsible for all of the paging stuff and it it it's where you call to when you want to do something like I need a physical page of memory or I want to map this virtual memory to this physical memory stuff like that the memory manager knows all about that and when we initialize actually that's where we take in that memory map that we talked about that we get from the bootloader so what we call initialize paging we do a whole bunch of stuff like this is where we protect the very first page so that if you dereference a null then we'll actually crash so important because I remember early on you referencing Nova would not crash in the kernel or wouldn't crash anything and make some things a bit more difficult to debug it's good if no the references crash but basically here we will iterate this memory map that we got through the multi boot header and we will will find the contiguous regions of memory and then turn those into metadata for our physical memory allocator so we create these physical region objects that then they know like how much physical memory is available and where is it and then that's what we use later on if some program or if the colonel wants to allocate like a physical page of memory anyways that's what the memory manager initialized is all about and then it sets up paging and stuff so that virtual memory works and let's go backwards ok and then we try to initialize the apec now is very early like unfinished code so I just kind of eagerly overly eagerly merged a pull request from someone who was working on epic stuff because it didn't break anything and I like where it's going but it was probably a little too trigger-happy and merging it because it was a work in progress and anyways it's it's interesting so eventually that's this is basically where we'll go down the path of being able to run multiple CPUs and in doing doing things like SMP and then here we initialize the timer the pit the programmable programmable interrupt um and essentially we're just telling it hey give us an interrupt every one millisecond so that's the ticks per second that's what we pass to it as the reload value basically I mean we compute the reload value based on this we want 1,000 ticks per second which means that we will get what is it i RQ zero yeah Q zero a thousand times per second and that's what drives the multitasking in the kernel so after we initialize the primer we will walk the entire PCI bus and just dump out all of the devices that we find because this is just here so that we in the blue blog we get like a dump of all the devices because then we can go and look those up and if it doesn't boot then we can figure out well okay so what are the devices here like that that can usually help you figure out why it wouldn't boot if you're struggling on some new hardware it's alright helpful information and then here is just setting up if it's text mode then we do nothing but if we are setting up a graphical mode then we make a frame your device and the this device is what ends up in /dev /fb which is a device that the Windows server will later memory map and render the screen image into so if we have information from the multi boot header about a frame buffer that was the thing I was or that's something that gets populated if the bootloader is successful in setting a mode then it will be these values here and we have the address of the frame buffer and physical memory so we can just instantiate this thing so MB Vijay device stands for multi boot vga device or I guess multi boot video graphics array device but if we don't have multi boot info we will try to create a V X V J device and that's a box emulator card which is the same card used in the Box emulator VirtualBox and @qv mu so they're all compatible and that's the that's the frame buffer driver that you that were using here let me think yeah we're using V X V J so you can see that we have a frame buffer at this physical address here and yeah then after that we try to set up some network adapters so we always make a loopback one it's for like local host communication talking to yourself and then we try to detect if there's an e 1000 we try to detect if there's an RTO Realtek 81 39 compatible card but auto detect these may fail right so they only succeed if we actually find a card so actually if I run here in QE mo if config will tell us that we only found in a 1000 or an e 1k there was no real tech card and then after that we make the praça fess and that's big class of its own and then we make their pts FS as well for pseudo terminals and that BTS s is kind of interesting it it's oh the mount command me to fix this it's something missing there but basically it creates these files and the pts and this is actually the tty landmark and it just automatically populates this so that if you have more terminals like this is now pts one and you can see that there that the file is always there and it's just a device file that has the correct device major mine in a number so that you can get to the tty somehow this is because when you close this the suitor terminal goes away so we don't want the device file to exist after that and we don't want to hard-code them in the data file system because then we would have to like decide on the limit on how many pseudo term else it could be so I started have a special file system that just generates these files so you can make as many pseudo terminals as you would like in theory things and then after that after we do that then we initialize the process which is now we're getting to the multitasking stuff here so this basically we make a we make the processes list which is just an linked list of all the processes in the system and then I don't know why this is here but this is this is just for storing the hostname of the machine somewhere and then we create signal trampolines and signal trampolines there are these little these little pieces of glue code that we use when when is when a signal is dispatched to a process then we need a little bit of glue code to to like run the signal handler and like a return execution back to normal afterwards and that's that code is identical in all processes so we just create it once and then we use it everywhere and that's this is where we create that and then we also initialize thread which I guess all that does is initialize the scheduler and this is where we create the very first processing system which is called the kernel process and the Caryl process is it's the process that will run when nothing else wants to run so it's always pit zero and it only ever runs if there's nothing to do and it's essentially the idle task if those four entity kernel and that's the very first thing that we will do is create that process and let's see so thread initialize actually just call scheduler initialize but I'm not going to edit now and then the first thing we do after this is ready then now it's time to go to the init stage - I call this I don't know why the but the name stuck and stage 2 is basically we go into multitasking mode so we create a new kernel process which means it's a process that runs with full superuser privileges it can access all the memory everything and it's just a continuation of this function that we're in basically it's just jumps to run it in a separate process and then we also create two more kernel processes we create the sync D which just you can see here it sleeps one second and then tries to synchronize any pending writes to disk I'm not sure if maybe it should sleep a bit longer that's what it does so it's just a process that will wake up every second and see is there something to write out the disk if so I'll just do that and then we also have finalizer process which is a very special process that went thread or process dies in the system then it doesn't run its own destructors and it doesn't do its own teardown is that it just goes into a zombie state and then finalizar kernel task when it gets to run it will go and run the all the destructors and do all the teardown on behalf of dead processes so this is just a way to have like a cleanroom environment when you are tearing down a dead process because you don't want to be like in a potentially weird context when you want to do teardown so we just we just let the finalizer do it do that originally I was doing teardown in a scheduler code but I found that it was much nicer to keep the scheduler very simple and not do stuff like process teardown in there and so just put that in a separate task so after we create these three initial things then we call scheduler pick next and we enable interrupts and then we go into the idle loop so pick next that just means pick the next thread a chidren and then we set interrupt enable I mean which means that the timer interrupts will come very soon within one millisecond and and do a task switch and then I had a loop that's just this we just call halt in a loop basically with a little optimization to to to reduce latency but that's it's not terribly important but then I guess we could look at the second stage of in it so in its stage two it's just basically just continues initial initial izing more devices and actually we initialize the system call interface so system calls and trying to go through int a B to X and there's if you look in sis call that H there's a huge list of different syscalls that you can make all of these you do them through the in 282 interface and in assembly this is call looks like this so it's in 82 and then you pass the argument the function number in a X and then the arguments up to three arguments in the X ax and B X respectively and if you want to pass more information than that then you have to pass them as a pointer to a struct and here are some examples of that like the Select syscall for instance has more than three arguments so we pass them in a struct instead of passing them directly yeah and then in its stage two we just initialize more devices like f0 that full that random that PTY multiplexer and then we figure out what's the route device that we should mount the filesystem on like watch what filesystem should be mounted on slash and that's we get that from a kernel command line parameter so I think we're not using a command line at all here supposed to say append yeah this is the command line Reese and action we say hello on the command line so just defaults to dev HDA and then there's a bunch of code here that looks through it takes the string and then looks at the different partition partition tables and tries to figure out well what's the actual root device that you want and once it has some idea of what device you really want then we see if we can find an ext2 file system on it and if that doesn't work then we just hang the machine because right now we have to have xD - root filesystem and if we manage to instantiate an ext 2fs object then we say the FS mount root with this file system and this after this point root directory is this file system right here and it's not possible to unmount root it will stay forever and then here actually is finally where we load the Kasem's so that's the kernel map and we find that by asking to BFS to open the slash kernel map file and kernel map is produced by the build here and it's just a sorted list of all the symbols in the kernel with their super loose plus names D mangled and this allows us to do black traces so you can see like if we exit a process then you can see here like what was the stack trace and everything that happens in the kernel like right here these are kernel stack frames they come from the kernel dot map and we load that whole thing and start up and store all of the kernel symbols in a big table so that we can look them up quickly for generating these stack traces it's very handy to have properly proper named stack traces like this very helpful for debugging and then here is Jesse's code for for the floppy disk drive drivers and we never actually tested this with a real floppy drive but I hope he has I hope you've tested it on a real thing Jesse sometimes I wonder why we have floppy support but it is it is cute we'll see we'll see what happens with that code I know I don't mind having hardware support as someone's using it it's totally fine but it's not wise using it and a little bit skeptical anyways and then here is where we yeah so if we're booting in text mode like I mentioned earlier then we will just create a shell but if we're booting in outside of text mode it's a regular graphical mode that we boot this program call the system server and we say that the first grab at terminal is graphical that just means that basically what that means is that we will let the Windows Server deal with incoming keyboard and mouse events and stuff like that and we don't try to hog them and act like there's a text console so at this point we just launched the system server and we also launched this other thing called the network task and the network task is a kernel task that is responsible for incoming network packets and that's this whole thing of its own but basically it's a kernel process that runs a loop that just waits for incoming packets from the network and then processes them and puts them in the right queues and stuff so that they can be picked up by the different other networking code in the kernel because of sending something out to network you can just do that when you want to send usually but when it comes to receiving it's you you it's nice to have all that in a single place so that's what we do there we just have a separate task for that but ultimately though like we should probably let the network task and do more sending and retrying and stuff like that I just didn't it's not super sophisticated yet needs to work a lot of these things need work and then after we launch the network tasks we call exit because this exits them in its stage to process because it's now done and it has left system server running instead and if we look in the system server program now this is a regular user space program it's in bin system server and it's sort of like the init program and the first thing we will do is just run now - a which will go through them let's see where this F is tab find it and finds these and they will mount these file systems one by one according to this file so just make sure that bak file system is mounted on /proc we have the pts where it's appropriate and we actually have a temp FS in slash temp temp FS is a special file system that does not write temporary files out to disk instead we just allocate memory for each file and store it in memory so it's like a ram disk and then after we mount everything we start a single x console just in case in case you would need it and then we start a lookup server which is the DNS resolution server and the windows server audio server for sound we start the task bar which is still a little thing on the bottom here this is a dashboard we start this first terminal and this launcher and then we go into and basically an infinite loop where we sleep a lot so just this is just so the system server doesn't go away and that's it that brings us to the state that we are in now we have booted up these programs have been started and we are ready to use the system and yeah I guess that was that's that that's how we boot I hope I managed to cover most of the important parts obviously if you have questions you're more than welcome to ask but I definitely encourage everyone who's interested in this to look at the code mess with it play with it take a deep dive in any part that you're interested in I'm happy to answer questions about that and happy to discuss how it works how it should we work and stuff like that if you want to talk in depth you can find me an IRC in the certain areas channel on freenode but github issues all those things you can always communicate that way too well yeah so it's a lot of people asked about this I hope that you found this informative I realized I was probably going pretty fast but I just wanted to cover as much as possible and get all the way from from boots to here so yeah if you made it this far then I thank you for watching and I really hope that you saw something interesting here and yeah thank you for checking it out and I will see you next time happy hacking bye

Channel: Andreas Kling

Views: 18,513

Keywords: os hacking, os development, c++, c++ development, kernel development, make your own operating system, make your own os, make your own kernel, operating system development, operating system programming, c++ programming, serenity os, serenity, serenity operating system, hobby os, hobby operating system, programming stream, os programming, c++ kernel, osdev, alternative operating systems, operating systems, linux, unix, bsd

Id: NpcGMuI7hxk

Length: 45min 16sec (2716 seconds)

Published: Wed Oct 30 2019

Reddit Comments

[u/SerenityOS]

Hello friends!

I made this video tour to show how my OS goes from the boot loader to the GUI, and what happens along the way. Most of it is stepping through the C++ kernel init sequence, explaining things along the way.

Perhaps you might find it interesting :)

[u/lightcast04]

Thanks a bunch! I'm going to watch this now.