OS hacking: Updating the SerenityOS Python port

okay welcome so what we are gonna do today is um updating the serenity us python port because there's been a mana bug fix update for python a couple of days ago and i like to keep the port and serenity up to date so let's see how that works today usually what i do first is just running the existing version so i have built that earlier just to make sure that it still compiles and we're not running into unrelated issues so let's see if that works booting up serenity this is the uh x86 64 version so it's a bit slower still than 32-bit uh there we go python as you see 3.10.0 which was the initial 3x10 release from i think a month ago or two um but there has been an update so let's see if we can find that thank you federer python releases so let's just grab the downloads yes december 6th so what we need first of all is um let's have a look at the existing port so as you might know um in pause there's a whole bunch of them already and here's python so we have this package.sh script which every port has which is basically where you define the version and source files and build process and then you run that and and then it builds and install support for you and um as you can see we saw a version here which is a different script just with a bunch of definitions and the reason for that being that we also have a tool chain python what's it called i think build python yes no [Music] this one so that's a separate script that basically just builds python from source on the host uh the reason being that you need the same version of python so um on the host for cross compiling it so in my fedora box um i just have uh three or ten as well um the miner version doesn't need to match so that's fine uh but uh so 3.9 wouldn't work um so let's bump that up and grab a new hash so we're using ssh256 here which i guess i have to make myself because they only provide md5 and then we use the tar.x set so i'll just copy that um let's grab the link copy copy copy link yes and then i just where you got that with download and i can make actually let's make the font bigger here okay so now we got it downloaded and then we need uh 256 some of the pattern uh download and then i can just really remove that and it will be downloaded again later so uh that should be fine so what i usually do next is look in the patches we still have a couple of patches so uh updating the path up here and this doesn't really matter and what might happen is that the patch doesn't apply cleanly anymore and then we have to tweak that but one step at a time so instead of doing that manually let's just control f like that in the patches and replace that wrong way like that should be fine place on yes and then you can see it's just updated everywhere and let's try to install that now so i just run the uh from the right directory so i'll see the into that and just running the package.sh script basically which should install the port and uh yes that didn't work so what we got here is um it's trying to apply patches uh right so um which version yeah okay that's right um so it's trying to apply patches patching file that applied applied so this one failed config.zip so um apparently they updated that uh let's also get rid of some old files here so um just rft python three attended zero and then also the except for that because we don't need it anymore okay so let's have a look at the failing patch um which is config that's up which is autoconf so as you can see uh it's a really small patch actually so i'm not quite sure what change there um but the way i usually do that is um let's just get rid of that source and so i enter that manually so now i basically have the unpatched python source code and then i look inside and so i need a config.zap which is here so i make myself a copy of that it's a text file yes uh yeah fine right and then let's look at the patch again so basically what we are adding is just these two lines so what i now do is in um the file that i want to edit oh okay they removed that apparently so i'm just trying to find that um [Music] same place basically but these all seem to be gone um so we need to find out where to insert that uh serenity line okay oh so it looks like they removed the dashes fine that works for us um so what where where do we go fix it so what we need is just these two lines i'm just gonna add that manually uh just copy that apparently there's an os definition now um which i don't know what it does but we'll see soon enough so just like that basically and then the next step is to generate a new patch from that so i use the patch utility for that um unified and then the [Music] uh i am in the right one okay the original file which is config.original and then config.sub so now basically comparing the original and our edited file all right um i'm using the wrong tool it's uh of course not patch it's uh diff so you can actually see here i did this for the initial 3.10 part already so um let's just tweak that and then not set up that pi but config. and it's here as well and i don't want to write that into the patch directly i just want to see it right okay and then put that in patches um [Music] fix autograph but we don't want to overwrite we just want to append i should use the right tool like that okay so now at the end of the file we should see this which is our updated patch basically so replace that and then we should be good to go and let's check what else failed um so this failed rejected it just got an offset just got an offset okay so um this is also the autoconf patch uh one for configure the ac and one for the configure script so i guess let's redo those as well um [Music] it's a bit more complicated so we got one change another change um yet another change and then a fourth change so um that should be fine okay so same procedure basically uh we type the original file configure that ac i make myself a copy original and then i edit this file so i just look again for um [Music] the right place where to put it uh so i just put this one before linux android um so let's copy that and then rename it um so we got serenity and done like that cool and then next change is down here just adding another serenity entry um it's always tricky because sometimes it's serenity os sometimes it's serenity so here this is part of the tool chain and i think this is what python uses as the platform name internally so need to get this right okay and then next one is down here host cpu so we can just look for that oh it's right there okay so we want to add serenity host cpu equal to cpu so um like that i guess and just double check yeah okay and then some linker flags i guess we just add yes linux gnu and linux can use our nsus so like that and that should be all i think uh yes and the next patch is plane configure so i guess uh we don't really need to change configure the ac i guess um because the configure script is generated from that but i still like to um like have a proper patch that updates all of them in case you want to regenerate configure as well um so let's do the same here configure configure that original and then tweak that i guess i can let's see autoconf okay so this is again our change let's just look for that down here we got serenity and the system name was just around the os like that and then down here i think it looks like that yep okay and then again her cpu and the link are flags um so i'm sure there's a proper way to do this like um i mean in theory we still got a working patch but the offsets are wrong um i never figured out how to do that properly so this is how i do it but if you know how to do it properly please let me know and i would be happy to learn that like that okay and then lastly the linker changes i mean this is not really an issue because it's uh only like once every two or three months that i do this uh basically whenever there's a new python update i use sometimes i do on the same day for example for the 3 or 10 release i was watching the live stream where they released it and doing the update at the same time okay so now we've got configure the ac and add that to the patch and the same for configure like that so now we grab our patch and so you see it added a bunch of stuff which is our new patch and so we can remove all the old ones up until here and what i like to do is just clean up the file name here and that should be all so you see it updated all the markers um just because they added some new stuff somewhere and so all the offsets change the actual patch didn't change that's good and i think that's it so let's try this again um right let's remove that just to get a clean um source directory and it won't download the table again because we already got it let's see if it works uh reverse patch fix oh hmm so what did not work here stand another patch that's failing it's patching posix module.h pipot.h all of these patching file okay so we got not just one client oh oh it's this one and the path is different here i don't know why uh so let's just change that quickly i should just say that one and then maybe it works this time no okay so um let's have a look at the source file again so basically just doing that whole process again um looking in the python source it's in lib http client.pi so lib http client.pi um that's a lot of warnings i don't know what changed so i'm just gonna do it manually make myself copy and then look at the patch so it's removing this line which hopefully still exists yes it does okay that's easy then just take this from the patch um all of this oh that's already here so all right okay it's removing this line and adding the try did they add that upstream i think so this is this is our patch um let's remove it again just to be sure let's close all of these service antar that source again it's possible that the patch that we have was still in development and we just copied it and now it's um an upstream so we might not need it anymore ah this is not supported because it's possible that someone took that patch and upstreamed it which would be really nice because then can just delete the patch and are done basically um let's see so in python the source code lib http client yeah it matches perfectly so uh someone took the patch and uh upstreamed it so we can just delete it perfect uh let's don't do safe delete that's fine yes and then we remove it from the readme so we have like a little readme explaining each patch um just so that when you go back six months uh you still can see why the patch was needed and um if it's not needed anymore you can just remove it uh which is nicer than just having a bunch of patches and no one knowing what they even do because obviously you can't add comments in the patch itself um so yeah let's try this again so you see oh it's still failing oh now it's set up that pie okay so apparently this is a big change even though it's just a patch release um but let's go through that so set up the pyramid okay [Music] this one yeah okay it's failing as well so i'm just going to do the same reapply all the changes make a new patch where is it the route setup that pisces let's reject those again i'm sure this is not actually the proper way to do it but it's worked for me so far so i'm gonna roll with it let's find that so what the setup that pi changes us basically um just adding uh include paths because of the way we have the root file system organized it's not finding them initially and so when it's trying to build uh c extensions for example and cursors or uh down here the uid module um was not building them which uh still didn't break the whole build but it's nice to have all those modules available so i had to do a little patching here um also might upstream this one day but for now it's fine so let's see in the setup.py file if it's the present yes okay so that's just adding uh the last path here basically like that and then also here so you see it's user include by default and we just as a user local include um which is now present here which is nice see so you can already get rid of this line and um oh so is this one i'm after this video i'm gonna look um who actually did this change because like this really looks like uh what we do here um so it's possible that someone just looked at all the patches we have and if they are like legitimate improvements and not just uh fixing stuff that is broken on our end then um it's definitely nice to have those upstream uh let's see for curses that is not present anymore okay yeah someone someone just took this um yeah you see this is now present in the upstream file so this can go and it's very likely that this also is now not present anymore yes did someone change it to this i should probably look for these lines no okay that's not let's have a look then um protect you id okay did the include files change uh i mean this might still be fine i'm just gonna try it so i think what we're gonna do now is just remove the whole patch because most of the stuff has been upstreamed it seems um so let's delete that and try again so this is a tweak setup.pi gone and then let's also change the readme and remove that and try again okay another patch failing which is now this one and i wouldn't be surprised if that's also upstreamed because someone seems like someone has been doing that so this is just adding maybe it was rodrigo um we edit some of these changes so let's look in lib xml rpc client in the python source lib xml rpc client and search for this basically yes it's been added as well that's really interesting and that's really cool so uh if that was rodrigo uh thank you very much for doing that makes our life much easier another patch gone there was this one and then remove it and try again yeah and now you see all the patches applied and now it's doing the configure step which takes a little while and then it should build actual python cross-compiled for serenity using our tool chain and then we should be able to use that with all the new bug fixes let's give that a moment to build uh so i don't know if they have like release notes somewhere for the bug fix changes um i only know the major three or ten changes i think i can look at the change look here oh no it's also three or ten final oh okay so here's a list of all the all the fixes so let's look at maybe xmlrpc yeah so there's been a couple of changes um let's see if we can try the first one and look at the actual upstream change so this one um who did this yeah rodrigo so thank you very much rodrigo for upstreaming these changes uh mentioning serenity specifically and i'm glad they accepted it and we can remove our local patches um so i mean eventually the goal would be to have it all upstreamed so that you could just download the source code compile it with no changes and it will just work um that might need more changes on our end as well for compatibility stuff like that um but this is definitely another step closer still compiling that's really cool i like that gonna make sure to mention him in the commit message as well there we go okay edit python3.10.1 to the database of install port so what we do now is just rerun which will regenerate the image and then run that in qmu and it should have the newer python in it let's see yes that worked beautifully okay so we're up to date just to test it's working hello friends yes okay let's put that into a commit so have we actually changed um three patches removed updated early versions updated all the markers and updated this one okay cool so we can just all add that um is that my pre-comments i might need to uh reinstall pre-comment um oh it's not happy about shell check okay let's just run it with dash n skipping all the commit checks and then i'll make a pull request instead of pushing it to master just to make sure uh it's fine with all dc islanding parts update python to 3.10.1 um when did it release on december 6th this was released a couple of days ago on to sixth and contains various oh is that not auto wrapping oh that's nano so on a different machine i have my vim config um properly configured no let's do that in c line instead because you know c line got a nice commit editor even with the marker line this was released a couple of days ago and contains various changes um that we previously previously needed custom patches for so we are now able to remove those and compile more unchanged upstream sources very cool thanks to all right rodrigo for making that effort all right that should be it selecting all of these just double checking patch okay remove removed yes that's not going to work because of pre-comment right so we run a pre-commit to do some local linting and then um we don't end up with a failing prs for all the linting we have but the issue is that we also run child check and that doesn't like many of the pod scripts so i need to run with dash and here to skip that i am not used to nana yes like touch okay so um that's going to be it for today i will be turning this into a pull request just to make sure it all builds fine and we're up to date now i hope you enjoyed this again if you know how to make this process a bit easier please let me know and see you soon bye

Channel: Linus Groh

Views: 3,242

Keywords:

Id: 1Gt8gLhgw6U

Length: 34min 45sec (2085 seconds)

Published: Sat Dec 11 2021