"Behind (and under) the scenes of the Meson build system" - Jussi Pakkanen (LCA 2020)

it's a great pleasure to introduce our next speaker who is UC Pakman and he works on the meson project I said that right and his talk is on behind and under the scenes of the mess on build system please welcome you see today that's the little off my Australian so that's correct but so I'm gonna be talking about stuff that we found in mission built system different source of surprising things that happen when you have a bit longer running project but first I'd like to say a few words about LCA I've done a lot of conferences and LCA is my favorite conference or like of them all for quite certain personal reasons and also because LCA was the first conference that accepted my talk so it's entirely possible that if they hadn't accepted the entire project would exist so therefore give me great pleasure to to introduce again something new for the first time ever which is a full manual for the Mason build system which is thank you to our 99 pages which is both a usage manual and a reference manual and this text is more extensive than the one on the website this is all written from scratch so there's no reuse of the existing documentation and this is only available as a PDF to save trees and it's only available as direct purchase and it's not in any bookstores or the like and here's the website you can go that in white now I pushed it to production about 20 minutes ago so it should be there it doesn't have HTTP because the hosting service has some problem enabling HTTPS but it should be enabled shortening now the obvious follow-up question this is what if you hate blocks and if you do the in Mason we don't have a patreon or something like because due to reasons it's almost impossible to run a patreon for a thing of the public good so if we feel that that the messing bill system has improved your life to the tune of twenty nine dollars and twenty nine ninety five euros plus a tax that depends on the country of purchase is you're totally fine you can buy the book and not read it so one mores the actual presentation let's start this with this simple programming challenge or design challenge suppose you have some data and you need to insert that into an SQLite database and some of the data that you insert should be up converted to uppercase so how do you do this do you do a so you have your data in your programming language you do the conversion and then you do the insert a motion to yeah SQLite has a function for this so it's an upper case function and you can use that in your sequel commands to do the things that you want to do right or did it go for option C which goes like this you get SQLite source code then you add a global variable to it it's called uppercase if I and then whenever you do insertions in the database inside of SQLite code you check for this with using get Enver and then when you insert data you set the global variable then you call in the SQLite and then you unset the global variable and then you say submit a patch upstream to add this to the the upstream with repo so should have hands how many would you would choose option C the single joke vote in the audience okay no well obviously but for some reason this is the way people want command-line tools to work every single time there's like any sort of needle to things ever hey could we add environment variables to this it's like no no so if you look at software engineering and think of your favorite two anti-patterns that you have like top two are global variables and string typing and environment variables are both and and yet for some reason people want these and then trying to support like configuration with environment it's just terrible because there's no guarantee that they stay the same they can change at any time and and so this this gives rise to what's known as the the law of bad solutions so if you have a bad only thing worse than having a problem is that you have a bad solution to your problem because then the bad solution will prevent a good solution for being a doctor and there are a bunch of these these around one of them is compilation flags we all of them use them but how do you ever thought about how you would combine them so suppose you have some compiler flags that come from source a and some compiler files that come from source people and they tell you to do something so how would you combine them so do you do global argument first the local one do you do a or B depending on whether you want to do something else or do you do a and B at the same time and to no surprise to anyone the answer is of course D because most compiler Flags write presidents but some of our left presidents so in some cases the one this comes first it's the one that decides what happens and in other cases well the one who comes last decides what happens and then if you if you have this problem then you need to have all sorts of MAGIX like what are these compiler flags actually trying to do in order to actually make them work together and this is a an altogether terrible idea and configuration by a compiler arguments doesn't really work because they don't compose if you have only one source of compiler flags everything is fine but when you have more than one then things get problematic and they usually work that the problem is that then at some point they fail in some weird way and you have an interesting debugging session in your hands ok so some other other things interesting fun things that happen is that as an example for the the D programming language which is used quite a lot there are three different compilers with this so this DM D which is the reference implementation as GDC is LDC which is placed on LLVM and they all take different compiler flags but like slightly different completely different they're all further on windows it would to do the link the DMD reference compiler might use the Visual Studio linker or clang C else linker or its own linker and the first and third of these have the same name it's linked of exe so if what if you want to use both of them at the same time maybe something happens like this might work your compilation might work if you run it on this terminal but doesn't if you do it in this one because the order of things in path is different so you get different results right so if you want to mix both them with decode how do you do that because there's things might break but it gets better always better so if you're not use linker flags with GDC as an example how you would set the Esso name here so you this is exactly the same as here with you if you're using GCC but for LDC the syntax is different it looks like this so um I don't know what the correct solution to this problem is but I can tell it's that's probably not it like - L - elves like what's happening and this is also a problem is that every now and then we run into people who are of the opinion that build systems like they should just take everything that they define all the flags and then they just just use them now and the the follow-up question that's like if you are this person is this sort of thing like maintenance burden that you want to take on because you have to know all the quirks of all possible things to make things work is that is that the really the thing you want to spend your life on I have spend some of some of my life on it I wish I didn't but but if you want to do it don't don't do it even then and then it gets even more fun because there are also people who compiled their like compile open source code for different purposes as an example there is a major organization doing security and he compiled many OSS projects with address sanitizer to find box the way they do that is that they inject compiler and linker flags using environment variables which are the official way to do things and it's risks this is how like Debian packaging and so on works and when you have external compilation flags then you use them for testing because when you test the compiler to do different things and you have flag that's always used then you need to you also use it in the tests and then the test compilation fails because among other things they inject flags that change the way linkers symbol lookup looks it's like if they're missing symbols that's totally fine and then if you have a test like it is this symbol available you always get a pass because someone has switched off the part where you actually get the correct results so we had a chat about this and in a bucket occur with with one of the people in charge of this thing and it went roughly like this so the you're injecting all of these flags and and you are breaking things so could you please don't do that and it's like no we need to do this because we need to compile with that Reis analyzer which is a reasonable reply yes well we have both in functionality there's a toggle switches like enable address sanitizer and this is what upstream uses and and that's what everyone else uses a lot just use the thing that we already have don't try to reinvent that and then the answer that they gave back went roughly like this I don't want to do that I want to do this this already works for every other make file based thing in the world so you should change your system so that it works with this and like well how do you respond to that it's like we have the correct solution but you want to do the wrong thing I don't know but but what this then turn out is that have you ever considered like where does software complexity come from why is everything incredibly complicated when it should be simple and then the answer is Arak one of the answers is that people have the opinion that if there is an obvious solution and it's like that's one way to do it the mandate thing that is a good thing we should do that because that reduces burden and and it's it's shared across all processes projects and it works just fine except the weird non-standard thing that I personally am doing that must be allowed and that must be supported for now into eternity and this is where then this is where complexity comes from because people want to do their own things even when it might not make that much sense okay there are legitimate cases where sometimes you need to do something would be weird that's fine but most of the cases when this happens that's not actually the case okay but enough about compiler flag stuff let's go on with the tendencies how many people here are software developers to order just all asleep okay so package config is used for dependencies and we all use that for all of our dependence right Yes No maybe okay well packets caustic is used a lot and it's it works for what it does it's quite nice but there are many projects who don't use package convict and instead they roll their own it might amuse you to actually go and find out the reason so sometimes you'll write their reasons is what I do not use package config because X and their or every single one of them is always completely wrong but all of these things have one thing in common can I guess what it is they don't work there are surprisingly many things that you need to handle in your own whole whole hand-rolled dependency system if you want to want it to be actually usable with many people for example does your theme handle cross compilation probably not does it handle installing into custom prefixes probably not does it handle the file system layout on a Mac which is different from the one on a general-purpose UNIX probably not does it support both shared and static linking of your libraries probably not does it support the visual studio compiler flag syntax probably not does it support platforms ways UNIX shell is not available 99% of these things are shell scripts that just print out flags if you don't have a shell your thing doesn't work or does it really work on any other machine than the one you're currently on and the answer is probably not but suppose you'll have these things okay you have your dependency systems might there be side effects using it so here is a wonderful flex for the new step which is like a reimplementation of the objective-c parts of Apple and they have their own convict program and and here you can get the debug flags and I'll give you a few seconds to think about what you know the output of this might look like right how many of you got close and and this is a gift that keeps on giving is like it has - I dot by by default it has to include directories for things that don't exist in your home directory the first two are for dependency information which classes with wonder coming from the build system and so on so on and W all gotta have that and so on its own and and yes I hear groaning in the audience this is good alright so you know the thing we find again and again instant cooperating is actually hard if you want to write your own dependency system you can do that and it works for you but you are not the target audience of your own dependency system the people using it more and cooperating with them that's the hard bit but it gets even worse than that so I had a discussion with a fellow at the bar who was a proponent of a certain programming language a and in language aids it's quite difficult to get pre-built libraries because everything is tied to the way they're built system and dependency lookup works and the following exchange of words took place and this is word for word or exact as exactly as it can be because I had had a drink or two but the person in question said that okay this is just the way we'd have to do things it's like well no because this programming language B which is roughly similar to what you're doing and they have a system for pre-built libraries and and you can integrate things with that are built with different built systems that's his desperation language B has existed for years without any usage whereas language a is taking over the world it could be said that they are just being pragmatic and then for the final response was no they're just desperate word for word exact thing at this point I couldn't think of anything to say but one of the things which which we've had was two talks here at the CLC and so on about the open source community is about cooperation it's like people come together and they work on on things to work together but what does it say about the open source community with cooperating with with people using something other than your favorite tool is seen as an act of desperation it could be said that this is just one person speaking and that's a reasonable counter argument but this is one of the things that once you realize that this is a thing you start seeing it in places and in an other place where you like really don't want to see it so if I ruined your life sorry whatever does let's go onwards to some of the requirements that built systems have built is were actually quite special in a way that they have requirements that other applications don't and it looks like let's look at some of them so a Bay SD developer made the following comment at some point to me saying that we can't use messin now because we didn't already use five years ago now this seems like strangely backwards search circular logic thing but but this actually makes perfect sense when you think about it for a while and the reason it goes a bit like this so suppose you're doing something like to LTS releases and if it's supported over several years and there are new graphics cards that come out and you need to have support for those in order to keep running and these come from the Mesa project they are developed on in head I think keel is correct if they develop it back trunk right and then the people responsible for the LTS releases will take that code and in back port that the old releases and make them but but build systems packages that are in LTSs don't get updates so the version that was there when it was released is the one that's kind of stay there because if you do change that then you get into all sorts of interesting problems so in theory the message trunk could only use miss inversion that was available and the oldest supported LTS that they have because if you use newer versions then you also have to rewrite the build system and this is again a big big maintenance burden on on on people doing that work and and because of this really adding new features to a build system makes it less capable of doing the job that it's supposed to do basically this is why why other tools was was a big thing for a long time because nothing happened in it it's the same now as was ten years ago so it kind of worked out okay but there's things get even stricter so in bsds I hope that I'm I've got this correct if there are any BST people and nobody's errors please point out but bsts have the the core package set and they have worst accruals that if you are in the core package set you're not allowed to build depend on anything that's outside of the core package set and python is not in there and I don't does anyone know if there are any plans to add and point them to the be any of the BS days okay no one's do you know about but it probably it's not gonna happen for a while so then a a BSD developer said that well we can't use mason in BSD until you rewrite it in either parallel C just for the record I'm not gonna do that hey someone else want to do that go ahead but this these are like the sort of limitation so BSD can't really use methane in their core because of this which is unfortunate but this is just how things go and and then these sort of things when you get deep in the in the bottom layers of the core things that you that you think you knew are actually completely like the opposite and it's strange and you can't really rely on anything okay let's let's talk about users because users are always fun and specifically how people adopt new tools and technologies and the thing that I noticed is that when you are given a new tool what people do is that they recreate the old tool exactly like all the things that you did using the old tool you recreate in your new tool because that's like the assumption is that you have to use it in the same way and this is not something that people do on purpose it just seems to be like a thing of the human psyche so as an example imagine we have a files and objects the like it's very common that you have your files in one directory and then you do define targets using them in a completely different direction so what we did is that we have file objects you just remember where they are like the full path and you can use them anywhere at all and the system will automatically calculate what the correct path is and all the documentation that we have all the example code that we have we always have this for mascot files and then list the files directly and I will say just use this and then the end result can be used anywhere and yet how did people use it they used like this I've seen this in tens of projects and and always filed bug reports say hey you know you know you don't need to do this it's like just just use it directly and and then there's like oh well that's convenient but there's an interesting psychological thing because if you are used to using make then this makes perfect sense because in make everything is a string and and probably you have never experienced the system where you know where things are other than strings so you have a preconceived notion that you need to take care of this problem and then only after when you are you are told okay this is what I'm actually a thing and you can you don't have to take care to do this and things I worked out for you there's ah okay well good but this is what this is is the problematic for for new if you do a new tool of any kind that has a workflow that differs from the old one this sort of effect is a bit of a problem because it when people write code like this and then they think that this is how you should use it they get a bad view of it even though the the actual system is more capable than how they are using it so this is an interesting psychological thing I don't know if there's a name for this this but if someone knows that if there's a name for this psychological condition please let me know but this what other what there is there is a law which is the uses lot of programmers is the problem with programmers is that once you give them the chance they will start programming and then there's a corollary for this which goes like this there is no limit to the amount of work programmers willing to do to avoid reading documentation so let's look at an example of this in Mason you can have you can build projects as sub projects of other projects and what we want went from word on is that you want to isolate the sub projects from each other and from the master project yep so if you try to grab files directly from a different sub project the system will detect that and say no you can't do that you have to use the mechanisms that are in place to do this because if you if you permit this then you force a specific layout between a master project and the sub project or two different sub projects because they have hard hard hard coded paths then they always have to be in the same orientation look relative to each other and then all operations are sandbox and this is a very nice reliability and flexibility improvement because then you can because you know that these things are isolate from each other all right so what did people do well whenever you have a restriction some people dislike that and they really want to do their own thing so they went through the source code and found a bug in the isolation code where you could construct paths that would actually pass through this thing and you could grab stuff from somewhere else and they exploited that and they grabbed files directly and then we found this bug and we fixed it and then they complained like how dare you I would I relied on this to work and you have to keep keep doing this since like I don't know when other container developers do you have the same problem and that people wanted to grab stuff from the other containers and their get annoyed when you can't I'm guessing yes a little a recurrent thing every single time we have tried to enforce anything for consistency there have been people who complain about it usually quite loudly but then this this leads into the the usability dilemma of software development in general is that short term usability comes from the fact that you allow people to solve their own problems and long term usability comes from preventing people from solving their own problems it's like you have tools and solutions and you have people use them and then they can use their energy on something efficient rather than everyone always reinventing everything but the the hacker mentality favors the first one it's like oh I can fix my own things and that it's a very alluring thing and people want that power and when and and even further is that once you permit something it's really difficult to then take it away so the so like software design is a very tightrope-walking of like what are the things that you permit which are the things that you don't permit and and how you go about doing that so that you drive people towards working on solving the problems once and then using that instead of every single project running off in some direction okay but let's look at something else so when you have users you get feedback and one of the feedback that we got some every now and then people send me private emails where they write I have this book from problem with Mason could you please tell me how to fix this and I always reply them saying that I'm sorry but I don't do email personal service over the puc account to the the IRC channel to discuss and then I got this answer which is that is very sad but I expected to hear something like that one day even from the open source community good luck with such the noise attitude I don't like the Gmail instant replies usually but in this case it's also because it says very sad indeed I was tempted to press that just just for the that one but then I didn't write okay so the time is time is running so let's look at a future of it so what is the biggest problem by in in free software development and at builds and so on in the future well one of the things which we could say is that it's mixing of programming languages and dependencies the old way was that everything is in C and using C would see is quite easy you just put this together now what's what seems to be happening is that people are getting more and more disillusioned by C and then they are moving on to other languages C + + D rusty linsey your favorite here Swift and so on and what probably will happen is that the debase of software will it's gonna get like fragmented and you have different parts implemented in different languages then there's the problem because every single programming language that you invent nowadays has his own build system and their own dependency manager and they are all optimized for the very common case of building only your own thing when you use some single apps from the library but that's it and for this use case if these are the things that you want to do it works actually quite nicely and they don't work quite as well in any other case one of the original design points of messin was that it should be support mixing of a bridge of languages and also that it should be re-implemented all so if there are people who want to write their own programming language like a recommendation is that when you have to design your new own build system and your own dependency manager consider taking the the messin definition and just write your own implementation from that kids use that to bootstrap your thing and then when your if when your thing is ready or asked whether use you can submit support to upstream messin and then your programming language which will work with all the other programming languages messin currently supports which is kinda nice the existing languages could also do that it's kind of like the naive dream that this would happen the pessimistic inmate says that it won't happen they're probably a bit too a trench to their existing tooling and they don't want to give it up so it's like it doesn't look very good but that the the big problem is that if you have if you want to have multiple build systems in the same builder Ector at the same time that doesn't work it works for simple use cases and it's a bit inefficient but if you want to make it actually actually scalable and reliable and all those sorts of things it's not really possible and this is an hour's talk on its own all right if you wanted to do really reliable it can get really really complicated so everything is terrible and we should just give up maybe but I really thought there was the mention that that a bad solution will prevent a good solution from coming up so so is we have these tools or like our pads and all in the load library pads and custom previews and all that to to build things with dependencies on your own and have the work now independent of the one that's on your system so what if all of these were they are the bad solution and so what what would a better solution look like we don't as far as I know we don't have one yet but there are things that you could beat and flat pack is one of the examples of this a flat rock builder has a a staged build concept it's a bit similar to how docker works I mean that you install your appendices one by one and you have a sandboxed environment so it looks like you're installing to the system which we know that works because that's how the next systems work but the the kernel will will make it so that looks like you're installing in your own thing but it's just over here by the side and the way flat-pack specifically works is that when you run your application all the dependency stuff comes from your own bundle but you can still see the file system as if you with the user so you can see the user home directory all that sort of thing and and it actually works pretty nice because then you can if you have something like package config where you have a build system independent way of getting dependencies then you can mix languages and build systems in completely arbitrary ways because it just install them one by one and then stop install the system and then they are usable out of the box and they don't need to know about how all the other things are built and with kernel file system isolations it's quite nice and convenient i was i'm just given the ten minutes science also closing up there are things we can use help with again if you are a Python hacker or other things all things are are needed user features percolates and so on but one of the things is that we have a dependency management system of ourselves because we've gotta have that it's called the wrap TV it's a bit understaffed at the moment so if there are people who want support work on that early great either just come to building new wraps misogyny like right the build system for a project which doesn't use messin and then submit that and then this they will combine them and you can build them transparently or you can view things that are there and even better work on web service reliability currently the rap DP is down and the reason that the server hasn't been restored is that it has apparently vanished and they work wonders of cloud technology so our goal is to get it like proper reliability and so on so if you have that or if you are a company and you won't want to give us provide hosting for the thing please come talk to us and we'd like to know about that okay so in conclusion let's talk about the year 2000 so we have gotten many things since then it's like LLVM and android and all these massive big projects which have which didn't exist at the time and then have been built from scratch and they are all much more difficult to write than a build system and yet build system haven't changed all that much since since the year 2000 at least not as much as you would expect expect and why is that and then maybe it's because a righty build system is hard but not hard is it difficult it's hard isn't tedious and and people are drawn to problems that are hard as it difficult but not really so much to things that are hard and tedious and this is a problem with with open source development especially volunteer one because for companies you pay people to do the tedious work but for open source development this is a big problem is like how do you get people to contribute in projects that are useful and needed but which are continuous to work on maybe you find some crazy weirdo from Finland to do that for you but it's like not really scalable solution that's the that's all I had I will take questions and the first for people to ask your question will get one of these which is a Mason mug only 72 have made made so this is your only chance of getting one and this is a rare collectible at least $10 on eBay go for it come on down if no one asked the question I will and I'm free Nene is it considered to build the colonel with messin okay so the question is can you build the colonel not so you can't do it now the the problem it's compiling the colonel is not so much in the compilation itself it's the K convict system and we do have support for that so you can if you do your K convict and you can create the configuration in and then you do your build system thing there so if someone were to do the work you could do that it would be interesting but the kernel is very special in that it does a very specific thing and it doesn't need to cooperate with anything else so the thing that they have might be the smart thing to do because there's a lot of work involved in changing it that that being said if someone is wants to do the why is interested in changing it I'll be happy to help you in all the ways that I can and but but if you have your very specific thing that you're doing and you have a single it works maybe just stick with it you can't get your thing do you encourage do you encourage to copy lib the Mason into the project or do you encourage that it or exist on the system a bit like see make which like an army wife does the you should copied into the project that's what Samba does so so we recommend that you use the system one and install from pipe eix own system and how do you cope with the I mean you alluded to issues with built that you know you've got an ancient one we're constantly having to fix bugs in laughs in in order to fit halves amber's build improve how do you cope with that long round-trip problem we haven't really had that or like people haven't filed bugs to us saying that this is a problem so somehow hope they're doing whatever it is then how their system works they're a co-pilot somehow but you would assemble would probably have the same problem if you need to run on the old old things and so we just we have a full copy of laughs in our third-party directory right we update it to as we as new releases coming out but the person is installed running set growing a semi build has the current wife already you know a tree okay the one that we tested with and we built quick so if you want to ship your own version of messing with the thing you can do that it works you can just running from whatever check out you want have if it works for you go for it okay okay okay so on the bsd five-year program five-year problem you mentioned in enterprise destroyers we have a slightly different yet similar problem in that we have some projects using mason the one have the latest and greatest features all the time and yet in our stable enterprise Detroit we might have a two-year old version of Mason without that feature do you think that there's anything we can do to kind of reconcile those two ideas well let me tell you a story of an imaginary Linux distribution that shall not be named system D adopted Mason and it's used on this thing that should not be mentioned by name distro which is something-something 6 and PI 3 is not in the default install for that it's only in the extra thing and things in the core are not allowed to build depend on things that are outside of the core same thing as with BSD and it and yet system Lee manages to do that because somehow magically Python 3 appears when you need to build system Lee and then it goes away so usually people can get around these things but this this is a like a known problem is like if you have a core infrastructure thing and you need to run on all things this is a big problem and I don't really have a good solution for this okay one last mug what thank you hey thanks for the presentation I have a question about integration with other built systems right in in Android the AOSP project they recently switched to a new build system called blueprint and problematically some of the open source projects used like Mesa for example there now mason based which is really nice but there's no good way of integrating the two like you're forced to use android blueprint - yeah basically build a software inside of Android but you can't really reconcile that with Mason being used in the Mesa project for example what's this story is there a solution to that kind of problem well I haven't looked into Android all that much but it's there disco they make the call they say this is how you do things and then we just have to cope and try to work from us from what I can tell from memory is that there is there was interest by the Mesa folk that you could have the list of files in some special way so they could read it both from the Android blueprint 1 and the the mission file which is something we could do so then then you would have the list of files in one location and it would be easier but if your if your operating system vendor insists on doing things and you have to do those things so sorry we are out of time thank you you see thank you [Applause]

Channel: linux.conf.au

Views: 2,447

Rating: 4.6923075 out of 5

Keywords: lca, lca2020, #linux.conf.au#linux#foss#opensource, JussiPakkanen

Id: iLN6wL7ExHU

Length: 44min 31sec (2671 seconds)

Published: Thu Jan 16 2020