How I learned to love build systems

as a teenager i was exposed to code in two different ways the first way was video games when i got into high school i got my parents to buy me a ti-89 i wanted that particular model because i knew that you could make games with it and so in math class and french class and history class and philosophy class i would spend hours writing ti basic to make simple games and later i graduated to making games on the computer using the java programming language in the eclipse integrated development environment and in both cases the process was kind of the same i would write some text in an editor and then press a button and then the game would start the second way in which i was exposed to code was also video games one day a kid brought an ubuntu cd to school next thing you know i had linux installed on a crappy old laptop that my father had rescued from the dump and i wanted to play games on it and the linux people back then were like sure we have games all you have to do is head on over to sourceforge download the tarball and then extract it and then run configure and then make and then sudo make install and then run it and for the longest time i had trouble understanding why these activities looked so different why was it that when i made a game i just had to press a button and it would just eventually start up but when i wanted to play someone else's game i had to run all those complicated commands into a terminal eventually i reached a point where my games were good enough that i felt like i wanted to share them with other people you can imagine my surprise when i discovered firsthand that just because something ran on my machine didn't mean that it was going to run on someone else's machine and you know i was a kid but i understood that a game was going to be made up of all these assets like graphics and music and sounds but also an executable that would be produced by a compiler when you give it source code it was difficult at first to see the link between a bunch of boring text and a fast-paced action game with snazzy graphics but it made sense to me just like it made sense to me that when my parents looked at a recipe book they were looking at a bunch of boring texts but that eventually would have something really tasty to eat for dinner what i couldn't make sense of is why could i take a pie that my parents had baked and bring it to the neighbor's house and eat it but i couldn't take an executable that i had made on my laptop and do the same part of the answer is that games are not food but also there really weren't that many kids messing around with linux at the time they just had a windows computer and they would play games that they bought at the store or got in a cereal box so i did what we all did back then when we couldn't find people like us in the area and i went on the internet sure i had to learn english but i also got a lifetime of trauma from being extremely online so it's impossible to tell if that was good or not but even the internet people that were running linux even the same flavor of linux i was still couldn't run the games it was always oh i'm missing this library and it says it can't find that file and i really dig the minimalist look but have you considered adding textures but there were textures they just couldn't see them so there i was at 16 wanting to make a 3d adventure game that was cross-platform i could sort of build stuff on linux but it was hell to distribute as for windows they had great games but i did not have the slightest idea how they were even made it took me a long time to realize that games were actually made the same way on both platforms just using slightly different tools it seemed much easier to do art on windows because it had photoshop and 3ds max and cubase and all those things and it seemed much easier to do code on linux because its guts were wide open for you to look into and interact with anything you wanted around that wasn't part of the distribution you had to build yourself to the point where i suspected that programming for linux was not really programming because i didn't see anything for linux being sold in a box in stores so i suspected that it was really more like writing games in ti basic which was accessible but it wasn't like i was making mario clones in motorola 68000 assembly like other people were just like i was getting basic 3d models to move on the screen using java and opengl but it wasn't anything like rayman 3 for example no the real games were on windows and they used direct 3d but i had no idea how to do that and my java games again ran great on my system but as soon as i wanted someone else to run them i had to tell them well first you need to install java which runs on three billion devices and then you need to double click the dar and then uh wait it's actually missing some native libraries because it's using opengl which is not part of java is a mess finding out about msis and mingw was a turning point i was finally able to generate a windows executable and it felt really similar to what i was doing on linux as well i had a terminal i had a bash shell i could run commands it even said gcc which i knew from trying to play some damn games on linux i started to draw parallels on linux executables are just those files that have the executable bit set so they show up as green on the terminal and on windows they're just fast with the exe extension that you can double click and both of these are just binary nonsense on linux libraries are.s so files and you usually get them from a package or you build them yourself on windows they're dll files and they're usually part of windows itself or you download them from sourceforge i couldn't quite tell where they came from still and that's because at the time windows was definitely a consumer operating system it was made for you to run software that you would purchase it was essentially a console whereas linux was made just for fun it was designed to let you tinker with it and so developer tools were much harder to come by if in 2006 someone had come to my parents house and dropped the cd from microsoft visual studio express they would have removed a huge blind spot for me but by playing with mingw i started seeing the value in that configure script because eventually i was able to build some libraries for windows using the same sources as on linux and for a while that was my model of what cross-platform development looked like as long as i had gcc and a bunch of libraries that i was used to i could make an executable but even that didn't feel like real windows development there were a bunch of libraries associated with msys that i had to distribute with my games and it was always such a hassle to find exactly which ones i needed to distribute finding out about the dependency walker tool was a big win finally i didn't have to guess an easy to use graphical tool would just tell me these are the files that you're going to need and i eventually found out about msvc the actual compiler that the makers of windows have been using all along but it didn't make sense to me because just like eclipse it was a complete development environment it wasn't just a compiler it was graphical it wanted me to write code in it and then press a button and then the game would start and i knew to stay away from that because i knew that was going to make it harder to distribute the game on different platforms and everything i had learned about the configure script didn't apply here it felt as if i was about to be locked into a single platform and i wasn't willing to commit to that and i went pretty far in life just ignoring msbc and not knowing anything more about build systems and then one day i had to build and install a piece of software that was using cmake and i thought that's new and also annoying i was already not super pleased to have to run a configure script and then on make and make install but at least configure was a script it didn't have any dependencies i could just run it i had no idea where it came from in fact i wrote similar scripts by hand but at least i didn't have to install an extra tool like cmake just to configure the build until i discovered that cma can not only generate make files it can also generate microsoft visual studio solutions and xcode projects and that meant that it could be used to build the same sources on different platforms using the tool chain that was native to the platform and at that same moment i discovered that visual studio solutions can be built from the command line with ms build and xcode projects with xcode build and everything slowly started to make sense not to say everyone was using cmake at the time there were definitely still a lot of people just manually opening solutions in visual studio and building them and that was what they were selling in stores but i was never going to be happy with that workflow and so i was delighted to learn that there was a solution to automate everything a solution that would let people push changes to some server and then have the server produce binaries for linux windows and mac and then run tests on them without any human interaction eventually i went back to my ti 89 that i was making games for in high school because i discovered that there's a version of gcc that can build software for the ti-89 and that was my first instance of doing cross-compilation when the build machine and the target machine are not the same and in doing so i realized even more of the value of that configure script for my games i was only worried about linux and windows compatibility and i was reliant on libraries that abstracted a lot of the differences between those two systems but those libraries had to be portable sdl for instance had to interact with different timer apis depending on whether it was built for linux or for windows and when building for something even more different like a programmable calculator a lot of the assumptions that i had been making no longer held some fundamental things like the size of integers in c were just different and that's exactly what the configure script was checking for all along is running tests to determine the characteristics of the compiler and using knowledge of the target platform to conditionally enable or disable parts of the code and the configure script itself was a program that had sources that were compiled with the new auto tools and if it had been generated by an older version of auto tools i could always regenerate it with a newer one that had knowledge of more platforms later in life i discovered other build systems like meson which a lot of gnome projects have migrated to i like that one because it has pretty colors and also because you can generate ninja files which are like make files but wicked fast it can also download and compile dependencies if they're missing and on windows it can discover visual studio installations and invoke ms build itself so you never even have to get your hands dirty at this point you hopefully have a pretty clear mental map of a bunch of different components that for me once appeared as a single thing but were actually distinct when i was playing around with java i was interacting with a code editor and a compiler and a vm and a bunch of runtime libraries not having to know all these concepts allowed me to just jump in and start making stuff which was very helpful but in the long run it slowed me down because the abstraction was leaky i could feel that there was some friction in the interfaces between different parts i just didn't know exactly where or why and there's always this tension between presenting an all-in-one package that's very approachable for beginners and exposing all the basic building blocks that are involved in the making of actual software if you look at the rust language for example it is very much an all-in-one experience you can write some code and then run cargo run and it does everything it downloads dependencies it lends it type checks it borrow checks it compiles it links and it runs the only thing it doesn't do is go space karno do that cargo road and that is amazing because in all three major operating systems people can have the same exact experience i did as a teenager playing around in linux but it doesn't quite work for everyone there are companies that are dealing with a humongous amount of code an amount of code so large that building all of it on a single computer would take literal days and that's on a top of the line workstation something like cargo gets you part of the way there because you rarely have to rebuild everything cargo knows the dependencies between files something make is also supposed to do so it knows only to rebuild what needs to be rebuilt for that change to appear in the final program but you still need to have every developer build everything at least once and when you're dealing with that much code and you have developers switching across different projects and you're trying to move fast that is unacceptable so this solution is like sc cache which by the way is looking for a maintainer right now which allows you to cache build artifacts remotely so that developers only have to download and already build objects instead of building it on their own machine but it can cache everything and it's limited to rust c and c plus plus so very large companies often will have their own build system that leverages caching as much as it can and even delegate the building itself to build farms and those build systems have to know about all the different languages that this company is using whether it's c c plus plus rust java kotlin objective c objective c plus plus go swift et cetera and now the question is how do they manage to build something like rust which would usually be built with cargo which again is an all-in-one package well cargo is actually just kind of like cmake plus make plus a package manager plus a bunch of linters and what it actually does at the end of the day is just call other programs like rustzy and linkers so if you parse the metadata in cargo.tamil files you can translate these into things that make sense for the build system and those very large companies and that system is what's going to invoke rusty on individual files and eventually a linker and then all the different languages come together to make the software that we know and love like chromium which everyone loves to hate but actually has a bunch of fascinating technology inside of it and is doing a lot of work that we now just kind of take for granted i talk about this in another one of my videos called a fistful of megabytes if you want to learn more about this now i realize you may not have any immediate use for all the information contained in this video but here's the lesson i want to leave you with every time there's something that doesn't quite make sense to you there's a possibility that it's just accidental complexity maybe it really is the case that nobody could think of a simpler way to do this and you're the first one to figure it out congrats but it's also really likely that it only doesn't make sense to you within your current world view when i first ran configure i was just building software for one version of one operating system on one computer that one time and as i gained more experience and started playing with different operating systems and different computers and building software again and again and again and again over the course of several years all the different pieces that i had encountered started making more and more sense and i hope that this video was helpful to you to realize that those things you just bang your head against and wonder why the heck would anyone come up with something like this actually have a place and a utility that you maybe just don't know about yet and actually if you're wondering right now why does that thing exist i would love for you to leave a comment asking about it because this is the type of video i love to make [Music] thank you for watching this video if you liked it you can like it you know youtube has the like function it even has a dislike function but it doesn't show the numbers anymore you can also throw me money on patreon if you have too much you can throw me five to ten dollars a month ten dollars will get your name in the credits so you can force me to write anything in these credits which i think is very neat in case a friend of yours or the youtube algorithm which is not a friend of yours has made you discover this video i also write articles that are equally long and hopefully interesting you can find them on my website which is linked in the description if you ran out of videos to watch there are definitely enough articles on there to keep you busy throughout the whole winter season i hope you enjoyed this and i will see you next time in the meantime please take excellent care of yourself as always there's only one of you so make it last [Music] you

Channel: fasterthanlime

Views: 18,750

Keywords: build system, cmake, cargo, autotools, software engineering, software development

Id: 7_DExGdUw7w

Length: 16min 31sec (991 seconds)

Published: Mon Nov 29 2021

Reddit Comments

[u/NeaZerros]

Great video, as always!