Linux Kernel Programming 01: Compile and Boot

hello world my name is victor engelmann and in this video series i want to show you around in the linux kernel i want to really dive into the subject i want to go through the source code and show you everything and explain what it's doing and yeah let's just dive right in so when you want to start kernel development then of course you need the source code of the kernel that's not surprising so i go to kernel.org and i go to browse for one of these kernels kernel.org has been the main website of the linux kernel since forever so i copy this link up here and now i'm going to use a program called git i guess you know that program already it's a very popular version control system if you don't have it on manjaro i'm using manjaro here on manjaro i would install it with sudo pacman s git i have already installed it but okay yeah on other distributions the command might be different like sudo apt install git or maybe ur pmi it depends on your distribution i guess you know how to install software on your machine if you are ambitious enough to start kernel development okay so as i said i've copied this link up here and now i make a directory and now i will use that version control system git to clone this git repository so i will make a local copy of it this repository is over two gigabytes large so it's going to take a while okay so now we have the source code and well if you want to become a kernel developer then you probably are interested in how to edit the source code you know i personally like the text editor kate for programming because of its multi-document support but yeah for something this big i think we need something more sophisticated you know a real full-fledged integrated development environment an ide and yeah on linux i like to use kdevelop for that so this is how k develop looks when you start it then you go to project import project and you navigate to the directory where you've downloaded the linux source code select the make file continue finish now it takes a while you see the progress down here okay now you see yeah this is a file hierarchy you have the different subdirectories here and yet for this video i want to show you the file main.c that's in the init subdirectory okay um yeah when you import the project like this then you get a lot of error messages here because it's looking for header files and it doesn't know where to look for these headers so you go to the settings language support the first tab here is for the include directories then you click here then you click here on this selection button select the directory then you select this include directory inside the linux source code directory then don't forget to press this little plus so that it is actually added to this list okay now it takes a while you see it's at three press up you see it's at three percent uh importing that stuff but uh okay uh what i wanted to show you in this file is down here around line 800 at the moment here we have the function start kernel and yeah this is the starting point of the kernel so when the kernel is booted this is where it starts running okay so yeah if you want to start looking into the into the source code i think this is a good starting point okay so now let's say you have downloaded the source code maybe you've edited it maybe you've only looked at it and uh yeah the next thing you want to do of course is you want to compile it and then run it so how do you compile the source code so you go back to kernel.org and go to documentation down here and you need to scroll down a bit here's a section introduction to kernel development and in there you have minimal requirements to compile the kernel okay and there you have this list of programs that you need to have so um on a manjaro system i would again say sudo pacman-s and then list the these packages that i want to install again on different distributions the command may be different like upped install or urpmi or whatever so i would install gcc llvm is optional make bingo tilts flex bison util dash linux if you are going through the linux from scratch book or the videos that i'm putting up about linux from scratch some of these packages might look familiar to you okay util linux k-mode e2 e2 fs prox 4x2 file systems j f s utils riser fs prox xfs procs squashfs tools bt rfs pcmcia utils quota tools ppp nfs utils proc ps o profile udef grub mce lock ip tables open ssl lib crypto calculator and yeah things is only needed for the kernel documentation so i'm gonna skip that okay it tells me that i basically have all those things installed except a few things well pcmci a utils um i don't know who still has pcmcia that's pretty old technology and i was able to build the kernel without that so i'm gonna ignore that the same thing for ppp or profile mce lock and clip crypto i've managed to build without these but from an older version of this list of things that we need i have also installed the packages pa hole cpio xmltow and then python swings and python syncs underscore rtd underscore theme i think it's a good thing to also have pearl tar and x v and x z yeah oh i must have put well i have all these already installed so i'm just gonna say no here okay so now i have all the programs that i need to compile the source and yeah so that brings us to the topic of the dot config file so when you compile the linux kernel the dot config file is a super super central thing so the linux kernel has over 11 000 different options and yeah the config file stores all of your selections for these options and yeah let's go into the linux directory so you see there is no dot config file yet because well the linux kernel doesn't tell you which options you should take so yeah there is no dot config file at this point and but without a config file you cannot build the kernel so yeah so you need a dot config file and how do you get one now one way you could get a config file is by saying make config but that would basically just give you a command line tool that asks you about each and every of the 11 000 options and you would have to manually select one and you would have to manually set each and every of these 11 000 options and i think that's not a good idea so i think make config yeah you shouldn't do that you can say make death config that would give you the default config where every of these variables gets their default value so something that the developer of each of these options thought would make sense but well the developer of a feature doesn't necessarily know your hardware or your distribution and the selection of drivers and programs of your distribution so default configs might not be the best way to go here i think the best option is to take the the config file from your distribution because your distribution knows to at least to some extent the programs that are available to you it knows at least to some extent which drivers are available to you so i think it's really the best idea to take the config file from your distribution and there is actually a quite easy way to get that because yeah when you build the kernel um the compilation really puts the entire dot config file into the kernel as a string resource and you can just ask the kernel hey give me the config file you can do that by saying that cut proc config so that's the configuration of manjaro so that's manjarous configuration and i could just save that as dot config or if you don't have that cut you can just copy and this is gzipped so you have to unzip it now it's named just config which is not the correct name so i'll rename config.config ok now when you already have a config then you can edit the config with make menu config and this gives you this text-based tool to edit all your options like here you can if you want it to be 64-bit or 32-bit power management stuff i won't touch that don't want to save it no you can also do make x config which will give you another configuration tool but it's not text based it's yeah it's it's a graphical tool but it's very similar so i think menu config is the way most people go okay you see you have again such a such a tree of options here you would have again the option to make it a 32 or 64-bit kernel this card changes but i want to point out one thing um i have downloaded the kernel source as you can see up here for 512 rc7 okay and my config file is of course for the currently running kernel which which kernel is currently running where with you name dash a i can see the kernel that's currently running is named 5110-1-manjaro so this is a config file for 5.11 but i'm about to to compile 512 and some options might have been added since 5.11 and for these options this config file doesn't have any entries so if i tried to build the kernel with this config file i would probably get errors and i've actually tried it i would get errors so what you would want to do is update this old config file to the new kernel version and you can do that with make old config if you do make old config it's similar to make config in that it will ask you about certain options but it will only ask you about the new options that have been added since the release of the of the kernel that this config file was for so my 511 had no support for acpi firmware performance data table that has been added between 5 11 and 512 and now i can just hit return and that will select the the default value for for this option the same that you would get in make devconfig and i can do that for all the new options enter return return return return can hold down the return key if you want okay and now my dot config file is updated for 512. so now i would be able to compile using the command make but well i have one problem it depends what you want to do i want to keep this video relatively simple so i don't want to go into the initial rundrive magic stuff so with the manjaro config i have a problem so i need to go to the option config underscore x4 underscore fs hit return and you see here i have the option m m means that the support for the x4 file system is going to be built as a kernel module now the thing about that is if i try to boot this kernel i cannot load a kernel module from from a hard drive that has a file system that i don't have support for you know so the module would be on an x4 file system so i would need x4 support to load the kernel module from an x4 drive so manjaro would solve that by putting the kernel module for x4 into the initial run drive but as i said i want to avoid going into initial run drive stuff so i will change this m to a y and then the support for the x4 file system is uh is gonna be built into the kernel itself and not as a module that needs to be loaded at runtime from an ext4 formatted drive okay now we have everything prepared we can just start make to compile the kernel or i will say make j8 because with nproc you can see i have eight processor cars and that means that i can run eight processors in parallel so if i say make dash j8 eight uh source files will be compiled in parallel and that will just speed up things okay let's start this and yeah this will take quite a while but uh yeah while it's running i want to mention this book here by greg crowe hartman one of the main maintainers of the kernel linux kernel in a nutshell it was about programming the linux kernel but it's not this book is about yeah configuring it and building it and booting it but not about programming and it's a pretty old book so it's from i think 2007. and yeah 2007 is quite some time ago i'm recording this video in 2021 so this book is 14 years old and yeah it it talks about linux version 2.6 which uh i mean that was a legendary kernel it's been the main kernel for i think seven and a half years so when i switched when i switched to linux in 2005 it was a pretty new kernel but it's been the main kernel for the longest time and it was the first kernel that supported several things that are standard these days like the dev directory um so again it's an old book but i'm i'm happy to have it because it gives you some insight into how the kernel developers think and it also has some really good references about the different the different command line parameters that you can pass to the kernel through your grub configuration um it has several hints about how to which how how to set your options it's a really useful book i think so i'm really happy to have it so yeah okay so i'm gonna let this run for a while now um last time i ran it it took about two and a half hours so yeah okay compiling has finished that was a bit quicker than i expected i hope that's not a bad sign okay i want to mention a few things if your compilation fails or one of the next steps then you are probably just missing some package and then i would advise you to just google for your distribution name and something like compile kernel or something um and then you will usually find some information about compiling kernels on your distribution if they have anything that's not completely standard um and if you have any problems do not try to just say make again because uh yeah your whole tree is in a broken state at that point so you should first have make mr proper and that just resets your entire tree back to the initial state but make mr proper also deletes your config file so you should backup your config file before you do a makemister proper okay so the next thing that we need to do is make modules install and this writes to system locations so we will need sudo make modules install okay so we've installed the modules um one thing i want to point out you can see here 5120rc7 that dash manjaro that's the name of the kernel that's how the kernel says this is my name let's say you want to make your own linux distribution then you might not want to well have some other distributions name as the identifier of your kernel then you could go to the config file and go to the variable local version where is it config local version um and yes this is where that identifier is controlled so that's what you would have to change if you want to make your own distribution you might also want to change the default host name okay good so we've installed the modules if you're interested that they are now in lib modules and then and then the kernel version so that's where they are now then we do a sudo make install okay this has now copied the file to to the boot directory so this is now here this vm linux file yeah the other kernels here have more expressive names so i think it's a good idea to change that name also and you can actually do this automatically you can you can use the command make with the special target kernel version inside your source directory of course that will then print out the the version number for that kernel so then we can use that here and for this oops for this processor suffix we can take you name dash m so you name dash good so uh next thing we need to do so the kernel is now in slash boot but to really boot the kernel we need to inform the bootloader about it i mean the file is there but the bootloader would not list it yet so we have this grub cfg and there this is what grub uses to make its list of choices when you reboot so we could edit that configuration directly but yeah next time you install some updates on your machine also maybe a new kernel then the operating system might just say oh i will do an update grub now and if you have this update grub will just overwrite your grub cfg so if you have written your entry directly into boot grub grub cfg um yeah then it will just be gone and you will not be able to put your kernel anymore so a way to do this in a stable way that will survive an update grub you can edit etsy grub d 40 custom so in grub d you have several scripts which are always run by update grub so they are used to to generate the new grub cfg so this script in particular will just print everything after the first few lines so what so everything that you put down here will be put into the new grub cfg so what would we put here um i mean we've taken the config from the distribution so i think it just makes sense to also look at the at the grub cfg from the distribution okay up here we have several entries for manjaro linux i will just copy one of them and paste it here okay so let's say i'm making my own distribution now algorithm and linux with kernel 5 12 r67 get rid of all this stuff okay okay so these options up here are not really necessary so i will just comment them out and i will also comment out this init rd here but let me emphasize one more time i would not be able to turn off the initial run drive here if i had not built the support for x4 file systems into the kernel so if you don't have an initial run drive then the file as a file system type of your root drive must be built into the kernel otherwise the kernel will want to take the support for that file system type from the initial ram drive and that will just not work and then you will get a kernel panic and yeah that's not gonna work okay so then we only have these two lines here left i'll remove the quiet and yeah to avoid any problems with hibernation i've also removed that part okay okay what does this all mean this search here tells grub so this is an information for grub itself this search tells grub that the hard drive with this uuid is our root hard drive so this is where grub will then look into the directory boot and for the 5vm linux 512 0 rc7 you might also see something like set root equals hd 0 and then partition 5 of an ms-dos partition but okay so uh this here is more robust if you um if you change yeah if you unplug your hard drive and plug it into some other port on your main board then this set route equals hd0 ms-5 or whatever would not work anymore because well hd 0 might not be the correct identification anymore but this search will still work because this really looks at the you at the unique identifiers of the hard drives and selects the correct hard drive by this you might also see something here for the root uuid equals and then also that or maybe maybe a different uuid here but i've had problems in the past with a uid here so def nvm e0 n1 p5 that's if i look into at cm tab up here my root drive is def nvm e0 and one partition five okay and yeah as i told you it's formatted x4 so that's how you know what file system you need okay this here means read write so that you can write to the hard drive afterwards if you have a dedicated boot partition then you might not have to look in in the subdirectory boot because if that is mounted just if if that drive is just mounted at slash boot then the kernel would be in the in the root directory of that partition right so so this location here is relative to that drive okay if you have a dedicated drive for that then on that drive you wouldn't have the subdirectory then this would be the root directory of that drive okay now um before i um boot this i will put a little error here so i want this file name to be wrong so that i can show you a little bit around in in grub so i'll save this then sudo update grub okay unknown device type nvme zero and one that's always unnerving to me but still it still works okay so i will now reboot i'm sorry but i have to re-record this reboot so because the autofocus of my camera just completely ruined the recording and that's why i have put this yellow little piece of paper here that's to help my cameras autofocus so okay let's reboot again okay so this is grub i guess you've seen that already before and down here i have my algorithm and linux but as i told you i've mistyped the file name so if i try to run it it will just say file not found not surprisingly if the fire name is wrong so if this was my only kernel and this was my only grub cfg then i would have broken my computer at this point right because this entry doesn't work and if that's my only entry then i cannot get into the operating system to fix the problem right but that's not true because down here you see you can press e to edit the boot options or c to go to a grub command line so if i press e then i can simply edit the grub cfg entry that i had made before and i can just fix the file name here and then i could press f10 and then it would boot but i will press escape instead because i also want to go into the grub command line with a c and yeah here you can even boot kernels without even a a working grub cfg you can just use the same commands from the grub cfg that you've seen in the grub cfg like in smart insert module g3 io for example is one module that you could load or you could set the root to this is a american keyboard layout and i have a drum keyboard so it's a bit so um i said set root to hd and i even have tab completion so it now tells me hd0 or hd1 well let's try hd 0 comma tab completion okay well partition one two three and four um yeah this is not the uuid that we've seen before and i'm pretty sure that my boot disk was not partition 4 so hd 0 was wrong apparently so let's try hd1 okay what do we have here ntfs ntfs xt that looks good partition 5. we had partition 5 before so that sounds good so let's say gpt 5 here ok and now let's this is still on can you still see that um in the line down there i'll just uh press escape and command line again so set root equals hd one comma g p t five okay and now i can say linux slash boot slash v m limits minus 5 12 0 dash 7 and i can again use tab completion so that's nice i need to tell my linux kernel which root drive to use that was def n v m e zero n one p five and we want to mount that with read right okay that looks good and now i can just say boot and we're booting our system so there we are back in the graphical environment okay so we have successfully booted our self-compiled kernel with your name dash a we can check okay 512 0 rc7 so nothing is faked here on my other machine where i have an nvidia graphics card i have had problems with self-compiled kernels because well nvidia doesn't necessarily play well with self-compiled kernels normally when you have a new kernel you would recompile some of the kernel modules with dkms but yeah nvidia doesn't um integrate into that very well and manjaro has its own way of dealing with nvidia graphics cards so i was also too afraid to mess with that to install the official nvidia driver so yeah but i was still able i mean i tried to boot the kernel but the screen stayed black and but with ctrl alt f2 i was able to go into a text mode log in and log in there and yeah that really just proved that the problem wasn't the kernel itself it was a driver problem you know so um so if you have any problems uh it just keep in mind it might not be the the kernel itself that's a problem okay i think this is a good point to finish this video if you liked this video like it share it subscribe and please consider turning off your ad blocker while watching these videos that would help me a lot next time i want to go into kernel modules and drivers i think that's going to be interesting so definitely don't miss that so see you next time and goodbye world

Channel: Write your own Operating System

Views: 17,832

Keywords: linux, kernel, programming, grub, source code

Id: WiZ05pnHZqM

Length: 51min 57sec (3117 seconds)

Published: Sat Apr 17 2021