Build a TS/JS project with Bazel (MobX case study)

hi I'm Kyle cordis at Oasis digital here's my contact info and today I'm going to walk you through a conversion we did here a team of us converted the mob X open-source project to build with basil so to clarify we do not recommend using basil to build mob X we're doing this solely as an example as a way of offering explanation to sort of see how that conversion process works I think it's important on things like this to share credit where it's due so these are some other people on our team here who were involved either coding or reviewing the work you're gonna see today for the agenda I'm going to tell you what mob X is I'm going to show you how its build works or at least explain how its how its build process works and then I'm going to show you how it works after our conversion to basil to sort of explain the differences and similarities okay let's go to code and just I'll show this again when I get back if you're interested we do training consulting and on many folks tech topics now including basil okay so to the code so mob X is somewhat popular in the react world although it's not react specific it is a state management system so for example for if you're familiar with redux this is a very different way to achieve the same state management goals that you would achieve with redux if you're in react or with ng rx store if you are in angular you can sort of think of as being maybe like a different take also on some of the same ideas that are in rxjs but we don't really care what it does for the purpose of this build work so we don't need to look anymore about it so if you're familiar with mob X that's great but you don't have to be to understand the rest what I'm going to show here okay so this is the source code for the mob X project just so you can see the URL you know exactly what we're looking at and I'm gonna look at this code in an IDE in a moment if you want to look at the at that code after our work this is our fork you can see the URL in here so on to see our fork where I've made this conversion and then just to clarify I'm gonna close this repeatedly again we're doing this for explanatory reasons this is not intended to be PR Dormer or anything else the my ex project is perfectly well suited to build with its pure JavaScript build machinery that it uses right now Basel is a great choice for huge scalable difficult builds not a great choice for small relatively straightforward things like mob X okay so I am on easy I'm sure I'm in the right place here so this is the code before any work has been done so my ex is a JavaScript project so the starting point for all things is typically the package JSON file it's build is orchestrated with a combination of packet NPM package scripts which you can see some of here and JavaScript files that actually operate the bill primarily this build that J's file so they define two types of build they have a a quick build which all that does is the typescript compilation I guess to spot any type script errors and then they have a small build I think the idea there is to produce a smaller build output that might be why it's a named small build but this is the one that actually gets operated if you're going to say publish a new version of mob EXO what they call small build is the real build we aim for the scope of our conversion to be targeted primarily at this essential build and then the Oh point about the right test the jest tests because those are the primary unit tests here and most and more importantly what they call the web pack tests those are tests that verify that the NPM package build output from this process can in fact be consumed in a JavaScript project downstream so that's that's the scope we aim for there are some other parts in here for example they have a website a documentation website that we saw a moment ago here they have some some packaged scripts that run that that's built with a tool called docu Saurus and that would be its own interesting different problem to convert that to be done with basil and it's some sort of out of scope for the the core conversion we also let me see here I don't think we converted the lint target because that's typically is something people mostly look at in their IDE although we certainly have certainly no obstacle to running the lint as part of the build I just I don't think we did that okay so there's a bunch of packaged script here that sort of orchestrates what parts flow together to form to build but the most the work actually happens right here in this bill Jay s script now this is a reasonably representative example of how some JavaScript projects build so it's it's relatively ad-hoc it's just a JavaScript file it doesn't use any of the the various JavaScript build machinery for example there's a pretty decent thing called broccoli never seem to get a lot of momentum but broccoli is a great take on how to build there's the very popular in the old days grunt and then gulp and of course there are many other ways to build your stuff with JavaScript this does not use those that this just uses a JavaScript file like most JavaScript projects it's kind of fundamental view of the world is that you when you build you mutate the contents of the project directory so that they've made some attempt to separate build output so for example oh you don't see a Lib directory here but some of the bill up it lands in a directory called Lib but other build output live lands in these other build output directories it's kind of an ongoing challenge when you go to kind of impose a tight order on a JavaScript build process that just much of the NPM package community doesn't really have a view of the world where you have ICANN in an unchanged source directory and then a totally separate output directory it's much more common to see inputs and outputs interleaved either directory by directory or even in some cases file by file so with that context I'll explain a little more about what this build script does this old way and then we'll move on to seeing what we did okay so in this build script it has no notion of incremental ad so every time you run it it simply gets rid of any previous build and then all of this code is to run the typescript compiler and so it actually you know what I'm gonna make ooh that's a little much here before I get too much into the code I think this this is prime this will be yeah we'll call this good enough so hopefully you can read this sighs okay so it runs the typescript compiler now they're choosing to configure the typescript compiler by having a static TS config file and then based on some parameters they override that configuration and then they will call the typescript compiler here programmatically um this seems like a pretty low-level thing to do but this was probably the the most straightforward programmatic way to configure the exact set of typescript options they want it would probably be a modest improvement to remove some of this code and instead just to have more than one TS config file and have your build process use those different TS config files that'll be a little bit more declarative but this works so there's nothing inherently wrong with this approach okay right here we have some error checking whether the build actually happened and then down here we have the the top level of the build right so we call this async function and we basically build the typescript twice so we use this function up here to do the typescript build it's quite a bit of quite a bit of code and then it's not super clear to me but I fear that this might be a bit asynchronous in some cases I'm not 100% sure that this is legit thing to do I just haven't studied the typescript compiler API enough to know whether it's always okay to check that output immediately after okay off the pause mode recording for just a moment here but that pause okay we'll come back to this function I was gonna show the high level here the top level code okay so we run type script a couple of different ways and this is intended to be synchronous code then we come up with a build directory and then we loop through and generate I guess six different bundles so this is kind of nice if abstracted out how to generate how to generate the bundle sometimes they do it off their es5 builds sometimes off of six and then they they target different file names that so they do a reasonable job of encoding the the type of build like I probably would have named this you know what like that if I was doing it but you could read through the detail the details don't matter too much just to know that the type scripts cuz is compiled to different ways and then there are six different build outputs created and then there's a separate script when we won't do as much with but they have a run publish script fundamentally their publish script will call NPM publish which calls NPM package behind the scenes okay so that's a pretty high-level explanation there was one bit in here I thought might have been worthy of explanation though let me see if I can spot when they call roll-up right so in this is the code that generates a bundle it does this by calling roll-up um they call tercer inside of roll-up and this is a very very common thing to do in the JavaScript world you sort of have this order of N squared problem that may be why there are so many NPM packages out there but in the JavaScript world it's reasonably idiomatic that if I have a tool and I have another tool and they're going to be somehow used together then there will probably be yet another tool that takes one and plugs it into the other and so for example if I'm using roll-up and I want to apply to sir to the output the most common way to do that in a JavaScript build process is to use the roll-up tercer plug-in to embed the execution of tercer inside the execution of roll-up we have a different way of thinking that in the basil world that avoids needing this you know order of n number of different tools but this but this is a reasonably common an idiomatic thing in JavaScript okay so let me just uh I'll go back to my slides for just a minute and talk so now we're going to switch to the Basel version of most of that so we're the idea is to take that existing build and then purely for a kind of explanatory purposes here maybe future teaching purposes to convert that and build most of those same things or several the important parts of that same thing using Basel so let's go back into the code and see how that works okay so I'm on master and that master corresponds to what you can see right here as the current master but just in case I make some further tweaks after this video if you go in here to tags and you click in here I made a tag called video 2019 so if you if you go in and click on that you can get the exact version of the code that I am looking at in this screen video ok so we have a Basel program typically the place you start trying to understand Basel is with a Basel version file so you know what version of basel is intended to be used I'm using version 2.0 it just shipped a few days ago but basil is using semantic versioning so 2.0 is really not very different from one dot oh you know that they changed a couple of incompatible flag defaults but for for most code and including what we're doing right here it wouldn't matter if I was using 12.1 or 1.2 or probably even one dot Oh so just be aware it's one of those semantic versioning things where you should maybe be less scared about going to the next major version than you would on a project that had a different version encase oh this is my converted code I have removed the old Bill J's file because I'm willing to do something else instead if something hasn't start with Basel version two now I need I need to go gather up my tools to use basil and that is always done in a workspace file with basil so I'm just gonna walk through this briefly if you are deeply familiar with basil you could probably like skip forward two minutes but just in case you're not next explain what this means so when you mate when you're setting up a workspace you don't have to give it a name but I chose to but I also have set up this managed directory capability so that when I have basil handle my node package fetching it can make that available here for my IDE to access that's a nice capability I think that was added somewhere around 1.0 with basil there are some built-ins some of those built-ins are still always available by default like the operation load but many of the built-ins are actually sitting inside a kind of library toolkit built into basil one of those is the ability to retrieve code via HTTP so we go make that available so now we have the HTTP archive rule available so we're going to use that to go fetch the the rules no DJ s so this is the the basil support for JavaScript it probably should be named rules JavaScript maybe most ideally because it's actually not specific to note the rules node.js contains both the node support but it also contains a lot of support for browser side JavaScript and it looks like I'm on version 1.0 of that I don't know if that's quite 100% up-to-date if you went and you search for rules nodejs you could go find out okay I'm gonna pause for a moment just verify that that's right verified that this one that I don't know this is the current version as of three days ago as I record this so it just went 1.0 stable that means they'll be very careful about API changes there will not be any breaking API changes until 2x I would expect they'll probably continue to be reasonably rapid development rules node.js has been evolving and improving quickly over the last six plus months and I imagine it will likely continue to do so next from these rules nodejs we grab the ability to set up node and node packages so we bring a note again so this is not relying on a version of note I happen to have on my machine this is a basal managed node installation and then it's not relying on me running yarn basil will will run yarn and it will manage the resulting files and then based on this mechanism it will expose those to me so that I can use those same files it might excuse me IDE um when you've done this when you've brought in a when you when you've used the basil rules nodejs yarn install or NPM install rule to bring in NPM packages sometimes there will be further basil code basil rules inside there so let's take a quick look here again this is my edited version so sure enough there are several basil rule sets being brought in via NPM now some people find this very surprising at first it's not yet clear to me what all idioms will emerge is like the best way to do this but at the moment this seems to be the way to do this in basil is a to package up basil support for libraries in a certain ecosystem in a way that they can be brought in in the same way as the other tools in that ecosystem that seems to be emerging as at least at least one reasonable way to do it so when that happens there's going to be a bunch of dependencies that need to be brought in for those things and that machinery is right here typescript specifically has some additional set up and so that is performed here it seems to me at some future version it it may be possible to have that happens part of this operation but at the moment it does not you have to do this next skylib but I really thought I had removed the need for skylab let me go double-check I do use skylib in one place well leave it so that curious part about skylib is its name so today's basel has emerged from or descended from an internal google tool called blaze blaze had an extension language configuration language called skylark well when the taint when the time came to send those out into the world there was apparently something wrong with the names blaze and skylark taking a wild guess I don't know if they were said but my wild guess is there was a trademark conflict that somebody else had a product that was a plausible trademark conflict with those names so the external tools became basil set ablaze and star lark instead of sky lark well skylib was the sort of standard library or kind of common library for skylark that has not been renamed and so I kind of wonder if this will end up being renamed star Lib in the future but for the moment just basil skylib this is a set of common things you're likely to need as you use star lark next we do some webpack testing and we wanted to do web pack testing in sort of isolated NPM package sand boxes with two different versions of web pack so this is basically echoing something that was done in the old build and ability that you could tour in the mob ex source code they build or they run some tests with web pack versions one and two well those are both quite old now and so to kind of to echo that same kind of complexity we have built with web pack versions three and four but we want to make sure that with just web pack and the package we've produced for mob X you can get a program up and running and so we need a separate workspace feels like an isolated workspace so that we have absolute confidence we're not accidentally relying on that other workspace and to do that we've just used this yarn install rule again we'll go over this in more detail when we get back to how those tests work okay so if you just ignore everything from line 46 down this is everything we need to bring in this basic toolset so that we can build javascript in typescript code with basil okay so this is our main build basil file so it's it's the one at the root of the project so I call it the main one typically your project level stuff will happen here and then you'll take certain parts of complexity and you won't pile them all into this file certain things you'll push down into other build files and that's what we've done so as we go through this we'll have to take a detour to look at those when you use the basil typescript rules those typescript rules need to be provided ATS config it's idiomatic to attempt as much as possible to have just a single TS config so I've placed that here and I've used this export mechanism to make it so that other build files anywhere in the project can use these TS configs that turns out to also be true the package.json file there's a couple of rules that are at least one role that needs access to that so I've also chosen to expose that so this export mechanism makes these three files available as targets and it makes them publicly available so that any other basil build machinery anywhere in this project can see these files this is an interesting interesting bit of complexity but it depends on this thing called slash slash SRC so let's go take a look at that first okay SRC this is where the source code for mob X resides now it is a typescript program and that by the way if anyone notices this file I'll return to this file later this is a type script program so we're going to build it using the basil typescript TS library rule now the steals library rule might not be perfectly well suited to the job at hand um at the moment TS library sort of encodes how things are done at Google apparently and and they just have a set of practices that have worked well for a bunch of projects there but one of those is that for whatever reason I know the reasons but I don't want to spend an extra 20 minutes here what they consider their development mode gets es 5 files I'm at they consider their production mode gets es 6 files you could easily argue that in late 2019 as I record this for production mode they really should be bringing in like yes 2018 I guess es7 maybe but that that's just not what's done yet also we're gonna end up misusing some of this es5 output as part of our production build and that's okay because some some of the excess stuff that'll be in there will get pulled back out by tercer but anyway at the moment there's this disentangling between what's development in most production versus what's es5 and es6 and maybe in some future TS library those ideas might be teased apart a little more but that's why you see this kind of oddity if why is it talking about es5 and dev mode this is just connecting you back to as of right now as I record this at the end of 2019 this is what TS library considers these things okay so a TS library so this is a rule that will output the result of typescript compiling some stuff so here's how you tell it what files to go compile this is the glob syntax to get everything from here on down in the project so that'll get these top-level files as well as a whole bunch more files that are in here there are bits of code in here that reference the node API and so we need to allow that dependency and there are bits in here that are not bits in here typescript is going to compile this in a way that you know it needs the that TS Lib which is sort of the oh it's it's that mechanism for factoring out common code so that typescript doesn't have to admit as much repetitive code when compiling a bunch of files that's called TS Lib and so that needs to be available to TS library as it compiles this syntax right here so a 10 p.m. / slash and then something like this if I wanted to I could actually do something like this and that would mean the whole contents of the node modules directory has an interesting bit in the JavaScript world this is essentially universal every build step everything you do in JavaScript automatically they're in node automatically has full access to read and use any code anywhere in node modules that's sort of intentional but in the basil world it's it's really all about increment tality and if if your rule is that all of the source code can depend on all of the libraries then if any library ever changes you have to rebuild all of the source code well if all of the source code is you know 50 files or whatever so what but if all of the source code is a million files then that would be really bad so in Basel this is workable you can do this but it is bad form if your project is small enough that this is a good idea it may not be big enough to really be working with Basel yet at least in 2019 so instead with Basel you get this syntax and maybe in some more and detailed training it could go over exactly what the syntax means but with this syntax you're saying out of everything that's available in my node modules via my package JSON and yarn lock whatever grab this subset just these two subtrees of that and make them available for TS library to compile against okay so we have a thing called a target called source and that's in a directory called source so if we just say slash slash SRC so that's our source in Basel that that label so if I say oops so slash slash SRC sled so this would be the like this would say within my workspace find a directory called source which is a basil module it has a basil build file in it and find a thing in it called we're finding a target in it called source but you can do that as a shortcut so it's very very common for sort of the the main the primary target in a given build file to have the same name as a directory that build file is in and that's what you're seeing here and then oh for for various reasons ts library it's default output is the DTS files so just sort of the public API of that code and then there is this machinery here this mechanism to go grab a reference to get a handle on the other two alternate outputs so TS library has a main output of the DTS files then it has a couple of all turned out outfit outputs for the actual compiled code and there's various machinery there too you know to actually execute typescript in an efficient way if it needs to do both that sort of thing without reloading everything from scratch okay so now that we have this context so we know about that target or that yeah that target with that label this target and this target now we can understand what's going on here so um one of the more surprising things when first adopting basil is that I kind of assume you're really super familiar with Python that's because at Google from where this emerged at least from the early days of Google a whole lot of important code was written in Python and so it was it wasn't is reasonable to assume if you meet a developer works at Google they're probably pretty familiar with Python because they probably stumble into it and everything you know all the time even if the project they're working on is not Python so you could sort of think of Python is almost like a lingua franca of inside Google and therefore this star lauric language which is kind of a syntactic and semantic subset of Python you could think of that way it has inherited some Python constructs one of those constructs is this thing here with the brackets this is called a list comprehensions this is a oh it's sort of like a way of defining a lip list and then calling a map on it so here is a list the list of strings and then I'm going to map over that and basically I'm gonna run this rule once for each of these so this what this actually means is make me an array of the output of calling roll-up bundle once for each of these values for format now the outputs actually discard it because the output of these things doesn't matter we're calling these these functions too to define targets to use a rule to define a target but so that this has that import of somewhat imperative effect of making of invoking this rule three times to make three targets because there's three entries in this list or array down here now format this becomes the looping variable over this loop and so oh this is a really fun line of code the format on the right is this format but the format on the left is a named parameter to roll-up bundle name parameters is not a common language feature but it is a Python language feature and it is therefore a systolic language feature so that's what this line means it means that this looping variable format becomes the value of the format parameter when invoking the roll-up bundle rule to create a target and then when we create the target we're actually parameterizing the name of the target this here is the the list interpolation operator sort of like a printf from see if anyone's ever use that but it basically means that the value of format will be stuck right in there that means you can sort of if you if you've read these a bunch before you just instantly read this as oh I'm defining three targets named mob x CJ s mob XTS m and mob x um b and when i and the only difference between those is the value of the format parameter when using that rule so this gets me three targets now when this grab the three targets that explicitly asked for the es five source and then but but some of my desired build output you know again echoing back to what the old build is scripted so some of that build output once es six output and for that one I have to go ask for a different output from that ts library and so I have I have looped to call roll-up bundle three times to make three targets and then I just manually called it one more time to get a fourth target and you'll see there's a lot of other config you know commonality here like the config file is the same the entry point is the same the format you know this format uses you sort of imagine putting in that list I'm like doing an if statement or on this arguably I might be like two or three lines of code shorter if I had added more complexity inside this list comprehension but it seemed better in terms of being able to explain this code to go ahead and call this one separately okay a few more bits so when you specify the entry points this is a rollup entry point if you're familiar with roll-up you know that you know when you roll up code you have to give it a path well within the kind of body of Java JavaScript code you've given it of the JavaScript entry point is and you might look at this and say well that's weird that's a TS file that's because there's machinery in the Basel rules nodejs rollup bundle rule that uses the typescript output to translate from this source directory to the destination javascript file behind the scenes and that's really nice that means that as the user of roll-up bundle I can just point to like my source code level entry point and now you know let it worry about figuring out where in the file system that entry point ended up after the compilation and then the same idea applies down here lastly if we look in this roll up config file so in the roll up config this is basically bits that were borrowed from that old build script and they well they used several plugins but the two that we're really necessary are the node resolve and the replace plug-in the details don't matter this is just how roll-up works but importantly when I call roll-up bundle it does not magically or automatically get access to all of the node modules right this is that Basel idea of controlling your inputs so we have to give access to the specific subset of all of our NPM dependencies that we want this rule to have access to as it runs okay now this this to this point line 46 now we've only rolled up the code I guess four different ways now when I showed the the build vag is the old way the kind of mob X master main line way of doing this build they configured roll-up to call tercer which is a minnow fire as part of the roll up process and that actually would work here because I think most likely whenever I rebuild whenever I read a rollup that probably exactly matches when I have to redo the tercer but it's definitely more idiomatic in Baselworld to split those apart so that in theory at least in an in reality and a larger project the more you break the separate tool in vacations apart the more basil can divvy out the work to a distributed array of build I guess build worker machines and the more it can skip work so right if it like if it redid the roll-up because it's thought something might have changed and then the output had the same secure hash as it did last time it would skip retiring it so I did not mean to do that okay um the key point though is that we want we want a total of six build outputs to match what the old build made and two of those were simply minified versions of two of the others and so I've just done that here I've just defined the two outputs there's some interesting bits here so like when you when you call tercer and you give that rule a name you give it a name without the J s on the end and but it actually produces a file with the J s on the end that you can ask for so that's a bit of an oddity so you you know as these names are both without the J S extension which is fine just happens automatically mob X has some flow types it looked to me like that was a piece of work in progress and that they weren't they weren't actually tested yet but since the since the existing file was copied into the NPM package output it seemed responsible to do the same in our recreation of that build and so we use the copy file to do that copy file oddly enough that's actually not not not a built in basal feature that's part of that skylib so you can just there it is for anyone that finds that irritating that I had to go you know import that rule and then you know call it this way I invite you to go read the stack overflow page on how to copy a single file in maven it's something like 27 lines of XML to copy file a to file B so this is not bad if you've been traumatized by that experience okay so at this point we've compiled the typescript code we've rolled it up four different ways in different package formats we've minified two of those now we want to bring all that together to make an actual NPM package the rule for that as of a few days ago is pkg and p.m. I want my package to be called knob X and then you just tell this thing what's gonna go in now I think this is really a huge improvement over kind of the default command my thing you get with with NPM with NPM package you know you you have to look at the NPM ignore file or maybe the files key in the package JSON and there's there's this sort of there's a lot of complexity and scattered nough sand how the system decides what files will go in the package I like this rule because I simply tell it what goes in the package I tell it these two files go in exactly as they are and then go grab these dependencies and these are all basil targets now some of them like this one it'll be a target that produces a file of exactly this name this one will be a file that it produces a target of this name or this target results in a file of this name with the j s on the end so there's a little bit of naming things going on but roughly speaking those are the so here are the six things we were looking for and I could follow each of these up this file and I could figure out where they came from for the sake of helping me translate between the old build a new I had annotated these with where this file landed in the old build machinery but these are the key six files this gets you the flow file and then this brings along the typescript types this little wrinkle with that that we'll see when I look at the build output in a minute okay okay so so this point gets us to having an NPM package now let me see if I can get a big enough come there we go so I'm gonna go ahead and just verify this still builds which I assume it will since I I just recently got it all tested here okay so this will go actually performed the and it'll produce that output I'm gonna go take a look at that output so we go too much farther so that output is gonna land in a directory called basil bin and then it'll Nate it'll be landed in here in a directory that matches the directory structure of the project so at the kind of root of the project there will be a a mob X here we go a mob X directory and that mob X directory will contain to build outputs let's see it's about that there we go so this mob X right here just clicked on this is the actual generated package so it contains the package JSON six files and then the source that has the types now this is what I said there is a rough edge a lot of rough edges around at the moment this is all I Chi said this just reached 1.0 three days ago um but uh it appears to be a bit out of spec or it may be I'm just somehow holding it wrong but for some reason we're bringing along the J's files here they're assuming we're bringing along the J's files as part of this source and I believe these should not be here per the way it's described as working but I've left that as an open issue I've not I've not tried to hack around that here I've just left that as a point for future discussion so there are some extra files in this mob X directory but modulo those extra files this mob X directory this will be the output this will be the NPM package ready to consume now how do I know this is any good well we'll see that in a minute we're gonna actually test it but we'll pick up where we left off mm-hmm okay so for reasons that will become there come-come clear later we need to go make basically a copy of this mob X build output using the file group rule and I've named it untainted as a hint Y basically this rule will have something called a basal aspect come along with it that will make it possible to determine what copy of node modules was used to build this but I intentionally want to go this with a totally separate node modules installation totally isolated from the environment where I built it and so to do that today the workaround is you use file group to go make a sort of untainted copy so to speak available of that and we'll see what happens there if I'd look around I could figure out where that land is probably an out or something probably poke around here and somewhere find that untainted one but I don't guess I don't want to get distracted there okay so I wanted to lastly the last top-level thing that the old build that does it check the flow types I don't think this actually does anything because I think if you go look in the flow types you're not there if you look where it's test flow star I believe it's it's not actually doing anything yet this fixed this name I think makes it not count as a J's file and therefore not actually test anything because I couldn't get the flow types to fail either with my of my original or their original or with this the this changed way of running it via basil it seemed like it it never never failed okay so if you just remember this hook so this mob X untainted this is a a copy or a way of accessing the NPM package output that we've created so far and now I'm going to go consume that elsewhere so where am I going to consume that we're gonna consume that from tests actually you know what there's two kinds of tests let's start with a simple of kinds of tests first the first kind of test is in this test based directory these are just tests and if I go look at this this uh there's another build file here I put the the basil machinery for running the tests in a build file in the tests directory but uh-oh the important thing is that I am going to compile the tests that are written in JavaScript or written in typescript excuse me which I believe there's just one and then I'm going to get the es5 output from that and I'm going to include that along with just the plain JavaScript when I call gests so this is a rule it's actually the this just test rule is actually it's actually ship right now as an example of how to call a of off-the-shelf javascript tool from basel so this this rule I'm calling is actually considered example code it's not considered like a and a shipped proven artifact yet nonetheless it works fine I've been using it so this is how we run Jess tests we give this target a name if I go here and I type basil tests dot dot dot means everything at some point yeah you'll see this name so there we go and that passed small means expect it to not take more than some number of seconds I forget the threshold sources sources are the files containing tests so these are the ones that Jess should try to find a test in and run that consists of just grabbing the JavaScript which is sitting in there and also of grabbing JavaScript which is compiled from typescript and this is basically a copy of the same or a copy of that an echo of the same machinery that we already saw it's pending how to build typescript you have to tell just where its configuration is you have to tell it what it's allowed to depend on now dependencies are files that just will need to access to run so that's different than source files so you know test source files or files that have tests in them dependencies or other files just will have to access and that's quite a substantial list so for example everything in this util directory and the you know this whole all these folders are unchanged from the original by the way so everything that's due till directory these are just extra files that are used to run the tests and so to compile or to execute these tests you'll need access to those just works on the basis of comparing previous output to a snapshot and so it needs access to where the snapshots are stored it needs X of this is that slashes SRC and these access to the code under tests so this file right here let's the jest tests see the es5 compiled output of you know running the typescript compiler on mob X so this right here is the only connection between the you know compiling the code and then running it now the reason this grabs it at the point of that SRC es 5 is that these are unit tests in a moment we'll see some other tests which use the NPM you know the the NPM package build output but unit tests just generally run against the file by file compilation this just configuration uses babel to provide some extra javascript language capabilities and so when it runs it will need access to a bunch of babel things let me see if you just look at the jess config this is a standard just config to special about it but that's that's why it needs access to those things and then the test code actually uses a thing called it all and a thing called serializer and i guess also conditional in the the typescript tests use conditional type checks and TS lib so basically this ends up being a list of all the dependencies that are needed to run this set of unit tests now at some point i could just reach a point of frustration and replace all of those lines with one line that says i give up you can look at all of the node modules and in a small project that would be a good idea but in a large project where you again have a sprawling huge list of no bottles you really don't want to allow hardly anything to depend on all the node modules because that would mean anytime any node module changes in any way the whole world has to be rebuilt that you don't want to do in a very large sprawling project lastly the the babel that's run from within the jest config needs and our c files we need access to that and then one of these tools i think and i'm not quite sure i just don't remember if it was jest or babel wants to look at your package JSON so we had to give it access to see that ok so the upshot of all of that is this line of successfully running the tests and they say cache because i've run this repeatedly but i could easily do a clean that would throw away at least that layer of the cache and then if i do a basic tests it'll go and do a bunch of work it'll rerun those and it'll give more realistic okay so I've been holding off in the most interesting part and that is it's pretty easy to imagine you know compile the typescript code and have the unit test work because that only had like one one that just had one or a couple of rules here needed for that but imagine I had made some kind of mistake because III I'm the contributor for this last little bit of doing the NPM package bit so imagine that then I made a mistake on this NPM package NPM part right imagine I I screwed this up easy to imagine how how would anyone know well the answer should be some kind of test and the the way that was done in the original code here I want to just want to verify I did not do the right kind of invocation to throw away the right kind of cache sorry about that okay so the way this had been on in the original codes let's go back there I made a couple of edits I'm just gonna roll all these back okay so in the code from before the upstream master of mob X there is a very interesting type of test here let me try to get the source to it they call these webpack regression tests now these are kind of ingenious I I did not look back to see who wrote these whoever wrote these was very clever so they wrote a program that imperatively and I don't want to walk to all the details but it takes several combinations of webpack versions and module formats and then it makes a temporary directory named to match those and then it takes these strings and it basically creates it and a whole project three or four files and then having done so it installs the right version of webpack it executes sweb packet and executes the bundle that webpack produced and it verifies the output is what it expected so this is basically an end-to-end tests where it's creating you know a comparatively creating three web pack three projects using webpack to bundle code from the mob X package Ehsan so we were inspired by what they did here but definitely did not want to take this completely imperative approach in Basel that would be a I mean it'd be very very unbeli do to try to recreate this imperative promise training way of doing this in Basel so let's look at what we came up with with Basel so what we came up with was to essentially more statically set out these these examples these workspaces so there's a directory called web pack tests now we need let me say to reproduce the same kind of difficulty as the old build had we wanted to use two different versions of web pack so we just we made a couple of directories one called the three a month called V for each of these has a package.json and a yarn lock and that package.json just points to a very wide version range of web pack for odd reasons you need web pack COI version three dots on thing to work with web pack version 4.7 it's just an oddity of the web pack world okay so these two directories v3 and before these essentially serve as sort of a sandbox a sandbox NPM package places and places for that to live so by having these versions sitting in these package JSON files statically we don't need to have that imperative bit of code we saw in the the you know upstream master web pack test that it's just sitting here today I think this is a more idiomatic way to express hey we have some tests and they depend on web pack version 3.4 a package lock to lock in an exact version but I looked at the the old code for this mmm the upstream mob ex master code intentionally runs on the current version of web pack you know version X dot y as of when you run the tests and although you know you probably make an argument why that's a bad idea I looked around to find a way to reproduce that and here's I had to go find this one there was someone talking about on Stack Overflow if you don't have a yarn lock thing gets angry at you because when you do your build or into your workspace we had to specify the yarn lock and this is not optional there's no way to say don't use a yarn lock but if you just make a yarn lock and you just delete all about those first couple of lines it works so um this is an intentional bit of non hermeticity introduced into a basil build because when I run this build it will get the current version of webpack 3 as of when this is executed not as of when some files were checked in and the same thing here for web pack 4 so this V 3 and V 4 directory you think these as being it's right word sandboxes to run to build a project with web pack and a couple of things in it okay and then these build files these don't have anything in them the purpose-built file is to make this directory be a workspace or to be a basil module so that you can refer to files in it when you need to okay so here's our first web pack test the the original test did three different variations we decided to do a few more variations just because you know once we have the pattern worked out why not so this is a test that uses web pack 3x and commonjs modules so it needs a source file that fetches or you know loads mob X in a way that will be compatible with floating it as a common jeaious module and then well it mmm just a moment here it didn't seem necessary to recreate the complexity of running the program capturing the standardout and comparing its results it seemed easier to actually write this index.js to act as a test so this essentially acts as a command line test in which the process exit value will reflect whether mob x worked correctly so if mob x is working correctly you can in you can import up with require you can grab the observable and auto run identifiers from it you can make a mob x observable you can tell it to auto run so means call this whenever anything I depend on anything whenever anything that I depend on in it changes so anytime store account changes this will be run and then I'll just append that onto this counts array and so if I increment count and decrement count I should get the original value of the store because this function will be called the first time you know when this autumn execution precedes through and runs this line and then when store count goes up by one this code should run again and when it goes back down by run by one this code should run again so I wrote this you could sort of think of this as a equivalent of an expect it's a one-liner well I wrote this line um so we just take those whatever values got pushed on there and joined them and then check the string and we do this comparison in such a way that if everything is okay it'll be a successful a zero ish exit code and if this doesn't match it'll end up being a non zero exit code so I had took a little playing back and forth to get this this inequality rights and okay so the upshot is if I bundle this program and then I run it with webpack if it returns if if I take it or if i bundle this with webpack then I run run the resulting JavaScript as a node program it will return zero if the tests pasts so let's see how we achieve that so to build with web pack we normally need a web pack config so this is our starting point very very simple web app config we need a file name for the output and you tell it what what right word here tell it which field to use when it's consuming a package.json file and this is a way of forcing it to use the commonjs code then we need to target this code for nodes of it for example process process works okay so let's see how do we build that in basil okay so there are rules in in this rules nodejs for running arbitrary code okay so one of the ways of running an arbitrary code is what they call NPM package bin and it can do a bunch of things but one of the things it can do is it can just grab some JavaScript binary some java script executable some it came in via that whole package JSON mechanism and it can run it so parsing this back apartment that means go look and find out what this is in the workspace if I look in the workspace this is the package JSON and associated files node modules that came in from that v3 sandbox that we saw a few minutes ago so this says go use you see here go use that V that web pack v3 sandbox get the web pack package and then get a binary that that provides you know command line next who writes call the web pack and then we need to configure it appropriately so outs this defines the name of the output file that basil should look for and use as the sort of output of this rule so we're sort of saying basil please trust me we're gonna call web pack and it's gonna make a file called bundle j/s and that's the output now this is a mechanism of passing a bunch of arguments to web pack and then data this makes files available for that web pack execution to see so let's uh well we'll look at these in that order so of course it's gonna need that index.js file right because like that that's the file we want to bundle with web pack but index j s and then of course it needs the web pack config cuz you know web packing it's you you always start by configuring it with a web pack config and then we really want to compile this against that that that mob X NPM package we built right because this is supposed to be sort of an end to end or integration style test that will verify that the result really did work that it can be compiled and it runs so that's why we need access to that bit and everything up to that point was was easy this is where it got really really difficult so this part right here is like the trickiest the whole thing but for webpack three here are the arguments that had to be passed first we had to put it in development mode because you have to tell it development or production mode or else it complains at you next we need tell it about its webpack config now the way basil executes s-- command line tools it does not merely execute them like in the directory where that source file is sitting right if this does not run things in this directory it goes and it makes this sort of symlink based sandbox and runs it over there and then it tells you where you can find the files you depend on so here's the here's a file I depended on here is the syntax and basil for saying the path from the point of view of where this program is being executed in that sandbox to where that file is located so I couldn't merely let it find webpack config by default because it wouldn't be there it would not be where it looked I had to tell it where to find the config file next I had to tell it an entry file right so this is sort of like that roll up entry file idea you got it to a web pack here's the file you start bundling from well the same thing I can't just say index that j/s because I don't know where that's gonna be I mean basil is gonna make this sort of sibling key sandbox it's gonna put that file somewhere and this is the mechanism by which it tells me where it put the file so here I gotta tell web pack there's what you found the file so something you're probably already noticing I configured web pack partly using a web pack config jeaious for all the static stuff with the dynamic stuff where I had passed I had to get from basil I did that configuration with the web pack command line next I got it to a web pack where to write the files well we've already told basil web pack is gonna write a file called bundle and we've already told web pack to write a file called bundle question is where's it going to be written well as part of that sandbox mechanism basil will tell your rule here is where the file should be written and that is the syntax for how basil tells your rule where it's files should be written so I'm telling web web pack your output should go where Bay's told me you're out book should go and then lastly um if I left these two lines off down here I would get an error because I would try to import or require mob X and it wouldn't be there so this is a way and there's you know there's something you can do in web pack config with this is the way where you can tell web pack where to go resolve a certain identifier so I can say oh that that web that mob X if anybody requires that here's where you should go look and this should be you know again basil is gonna tell us where it put those files right so we said we depend on the files now this is how we asked basil where did you put those files that we said we depend on and here's the really frustrating thing this bit required simply experimentation and study of what was going on behind the scenes that's why I said like all of this was the trickiest part to get work and of that this was the trickiest part to get work to get to work I think the ideal solution would be to go write a web pack rule for basil so if if you are going to use webpack and basil together substantially instead of this code being in an application there should be a web Peck rule that understands this bit so that definitely a rough edge or a piece of unfinished business for basil plus web pack use but after a little thinking this was found to be the actual path to get to that code the whole thing works so the output of this will be a bundle that J s file that as a result of taking this index and compiling the against the NPM package output that we created for mob X in this new basally way ok so now we have a program well no js' test this is a rule that takes just an arbitrary thing you can run with node say so a javascript file and runs it and if it out it's a zero exit code that's considered a test pass and a non zero is considered a test failure so since this is crafted to process exit with a zero if all is well is just a matter of calling it so bundle it tricky call it trivial okay so that's webpack three with common Jas all the rest of these are approximately the same but slightly different so like this configuration is slightly different you say mode development here instead of on the command line because webpack four is different from web pack three in the build file there's a couple of differences you don't have the - dev up there for example so I'm going to avoid going through all of these build files in detail because I'm already at an hour here but suffice it to say they are slightly different in irritating ways it would be very idiotic as a basil user to write a basil macro like if you weren't going to go to the effort of writing a full fledged web pack rule you would at least write a basil macro that would boil out the close but not exact like the near duplication between these different files the fact that they're almost the same but not quite that could be taken care of with a basil macro but that would make it even more difficult to dig in and explain and and and work from this as an example if anyone needs to of calling webpack from basil so it chose not to do that okay so I think that is everything so there's now this basil way to build the typescript code run the flow types on it although it doesn't seem you actually do anything yet run the gests tests including some JavaScript and some type script based just tests and then run five different combinations of web pack features to take the NPM package output and verify you could actually consume it and have it work so I guess that concludes our tour I guess I'll go back here and uh thanks again to is not just me other people both wrote code and reviewed code on this and if anyone needs help of basil out there you are certainly welcome to call us here's the way to reach us so thanks for watching to anyone who got this far bye-bye

Channel: Oasis Digital

Views: 2,241

Keywords: Bazel

Id: AegYQnhMn-g

Length: 64min 22sec (3862 seconds)

Published: Mon Dec 23 2019