Setup a monorepo with PNPM workspaces and add Nx for speed

so all the common package managers that are out there like npm pmpm and also yarn have a concept that is called workspaces and such concept allows you to have multiple packages in the same area in the same workspace and you can link them locally together meaning you can have an application you can have a library you can also have multiple libraries and you can consume them locally and reference them without having to go through a publishing step such as like an npm registry and this is exactly what you usually want to have if you want to set up a monorepo and so today in particular we're going to have a look at pnpm how a pnpm workspace is being set up and also once we have that how we can optimize it for speed and developer experience so first of all let's start and create a folder let's call it pnpm mono for mono repo which is basically where we will host our monterey publication and so then the first step we can do is just run pmpm in it and this will just create a package json that we can then use for the root level of our monorepo so let's open up visual studio code for this so far in our workspace we just have that single package json that we just created and this is the root level package.json for our monorepo and things like the main entry point here we probably won't need because all the actual applications or libraries will live in their respective folders let's leave the scripts for now but probably we don't need those either a good practice that we probably want to do is initialize git here and so let's create a git ignore to get started and then basically open this up and specify some of the most common things that you would usually have in there which is like the node modules folder which we don't want to have cached in git or don't want to have committed to get this folder and also potentially build photos those are usually the folders that contain the build output and we can add more as we go or we can also have that then individually in the apps or packages that we have in the workspace then let's in it get and so now we have a new git repository so let's just add all of it and comment it so let's now go ahead and actually structure our monorepo workspace now monorepo workspaces most often come in two different shapes there is like the app centric setup which has usually a folder structure with apps at the top level and then packages and then there is the more like package centric which is very often used by open source frameworks for instance check out like how vue.js or react.js or angular structure the workspaces and that's usually a folder called packages where all the various packages live in there you can have both structures you can mix them we will today go mostly with an app centric structure where we have like a remix application live in that apps folder and we have a react library living in that packages folder so let's go and create that so let's create an apps folder and then let's create also packages folder and these two will be where our applications live in the end there's something more that we need to do so in order to tell pnpm that this is actually a workspace and we're not just using pmpm for installing our packages we need to create what is called a pnpm workspace.yaml file and this is created at the root of the workspace here and we define basically what the structure of it looks and so we have the packages that are living in this vmpm workspace basically live first in the apps folder or in the packages folder and we probably want to have a glob here so let's do something like this and you could add here other folders as well or obviously if you just have a package-centric monorepo you would just have a packages folder in here and this is basically just used for pm to understand where the package is that it needs to link locally within the workspace live let's actually commit that as well and it's the next step we can already go ahead and create one of the applications that we're hosting in here so we can host any type of application in our pmpm workspace here whether that is like react angular or as well or whatever you want to choose now in this specific example i'm going with remix so let's just go to their website go to the docs page and understand how we can create a new remix application so you can see here there's the npx create remix at latest so let's go here we want to create that in the apps folder so i'm cd into that and then let's paste this in now i might want to change npx with pnpx which is the equivalent just for pmpm workspaces so remix asks work to create application and we definitely want to do that in my remix app because we are already in the apps folder we just want to create the basics so we don't need the pre-comfort setup stack and we can also just go with the remix app server obviously choose whatever best suits your needs let's also do the pmpm install so it immediately sets up remix and should be good to run so let's explore what got created here and the apps folder you can already see now we have the my remix app folder this is a full remix application that just got generated by their command and there's a package.json which is just the default package.json remix comes with now one thing we might want to do is add a name property here so let's just call this my remix app this is because it will be used by pmpm workspaces when we address commands against it here are remix specific scripts that we can run friends for building developing or just starting a remix application locally and we can directly invoke them here so we can just cd into my remix app here and then run pmpm run dev and this would kick off the script and run remix locally so we can actually go to localhost 3000 and we have our remix app running now usually what you don't want to do is like cd into every application or package you want to execute or run a command against but rather you would want to run this from the root of the one reaper so let me just go back out to the very root of our monorepo and for this purpose pmpm has so-called filter commands so what we can do is we can run pmpm dash dash filter we can give it the package name which in our case is my remix app and this basically uses the package name specified in the package.json and then we can run the command or issuing a command against that application so in our case that would be def and you can see it does the exact same thing as before just that now we can do it from the comfort of the root of the monorepo rather than having to cd into the single package so now that we have the remix application running let's actually create a small shared library that we can then consume from within that remix application so again what we do is we just go into that packages folder here and we create a new package so our libraries or packages you live inside this packages folder so let's create something that is called shared ui and the main goal here is that we would want to have like a shared ui library for instance that can be consumed by different applications that live in the monorepo so let's actually see the into their packages shared ui and let's do pmpm in it again which will create a package.json for us and so we can go ahead and customize it to our needs so this is name shared ui again this will be a private package this could totally also be a public package if you intend to actually publish that to some npm registry which is totally fine you can have a package just live locally you can have a package also live locally as well as being published for being consumed outside the moderator which is a totally valid use case for the rest of the things here let's just leave them for now as is and we will then tune them as we progress now what i want to do here is i want to create a react package and we want to probably use typescript and also just use the type of compiler to compile it for sake of simplicity here so let's install those dependencies just issue pmpm add create a filter again and say shared ui and i want to install react and i also want to install typescript let's actually pass the minus d here because times will be a development dependency not the runtime dependency and so now you can see how we have the dependencies set up and note this is the local package.json of our shared ui library here so now let's create something useful let me create a button component let's call it button tsx file and let me paste in here a button implementation so this is a very simple one you can see here is a button element with an on click handler which we can pass in nothing really fancy but it works for our small demo purpose here another thing you also want to use in a monorepo usually in those shared libraries is to have a common export so as your public api of the library basically and so this is usually the index tsx file where you can explicitly export things from that library that can be consumed by other libraries because potentially you could have some components react components or some other logic that just is tied to that specific package and so you would want to have it local rather than exposed to the whole world and so here we can now export everything from our button component because that's what we're going to use from the outside so with that we should be good to go now another thing we want to set up is the compilation for this simple purpose i'm creating just a typescript setup so we use the title compiler that will compile it the single files and they can then be consumed note the goal here is not to create a fully real-world proof compilation output of a react library you would want to use some more sophisticated setup for that purpose so in this case let me just paste in here our typescript setup and we will place this in the tsconfig.json another note is also that many of these things as we have specified here you would probably want to abstract into a higher level ts config and reuse it across different packages so keep that in mind while you set up your monorepo now here there's nothing fancy we have some jsx stuff we have the module set to common jazz because that is what our remix app will consume and we have here output set to a dist folder so that we have everything grouped into one place now having set up that we want to also adjust our package.json because it's now the main entry point we'll live in that this folder so we want to set that to dist index js and we also want to change it is test script which we don't use right now to build script and what we do here is just delete what has been created before and then just invoke the tsc compiler and so with that we should be ready to already test that again we can use pmpm filters to target the shared ui project in this case and then invoke the build target and we should spell this correctly otherwise it wouldn't work and now we can see we got that this folder output here which contains our button.js which is now compiled to javascript and it also contains here our index.js which is the one that we are going to consume so far we we're set up now we can go to our remix application actually install our shared ui package and then consume it so first step let's install the shared ui package in our remix application so to install our share ui package we could go to our remix application package json and just manually add it here or we can also use the pmpm add command and target that to our remix application so we can say pnpm add shared ui we filter and give it a my remix app and we can also pass the workspace flag here which means the package needs to exist locally in workspace otherwise pnpm will error and so we install our package if you now go to the package.json we can see that this shared ui dependency has been added in our dependencies part here of our package.json now we probably don't want a specific version but usually always want to rely on the latest version and you can also see this workspace part here in front which is a specific protocol to that pmpm workspace setup which indicates pmpm that this package lives locally within this workspace so how does it work if you go to the node modules folder you actually see that our shared ui package is in here and it has been linked to our local shared ui package here so this is basically a sim link which connects the two packages together and so with that we can actually now go into our remix app we can just go into a specific route here and let's just actually remove everything that is in here because we don't really need this let's create here a div and then we want to use our button so we can just go ahead and import the button from our shared ui package and use it in here and we can prioritize click handler which is on click which would then just do something like console.log let's try and see how this works again let's run our my remix application we can use pmpm filter my remix app and run the dev target so guys this will now install under package and run remix we can go to localhost 3000 and refresh and now we see our remix package loading the button and the clicked handler worked properly so so far we have seen the pmpm add and filter commands now you can also run multiple commands against entire monreal so for instance let's say we want to run the build for all the packages that are present in the monorepo here we can use something like pmpm run dash r for recursive and then say the commander we want to run so in this case this would run the build on the shared ui package as well as on the remix app package similarly we can also add here a parallel flag and now the commands will be run in parallel so to recap to set up a pmpm workspace monorepo all we need to do is actually define the structure of our monorepo so whether we want to have an absent lips folder or apps and packages or just a packages folder and then we need to create a pm-workspace.yaml file where we tell pmpm what our workspace looks like and from then on we can just use the normal setup of the applications that we set up within the workspace so basically for remix we went to the remix docs copied their command and then set that up within the apps folder and similarly for shared packages such as our shared ui package we can just use the setup we are accustomed to for building our libraries this could be just a simple typescript setup as i have done in this demo here or we could also go ahead and just use some more sophisticated things like beat for running it or es build or swc so this works fine as long as your workspace is relatively small but as you grow your monorepo you might want to have more sophisticated features for running tasks within it that comes with better developer ergonomics but also likes things like running things in parallel only running things incrementally what changed in a given pr and also things like caching to make sure that you scale even as your workspace grows so let's have a look how we can do that in a very straightforward way by using an x so if you haven't committed the changes yet this is a good moment to do that and add that to our git repository so to install nx all we need to do is use pmpm add give it a package name which is an x minus d because it's a development dependency and we also pass minus w because we want to install it at the root of our workspace because this is not dependent on specific packages but rather we want to use nx to coordinate and orchestrate execution of the commands across our packages so this will live at the root package.json in fact if we go now to our root package json here we will see that we got a new development dependency which is nx with the version that is currently the latest so with this we can already start using an x instead of the pnpm commands to run tasks in our monorepo here so rather than doing pmpm that's just filter shared ui or whatever you want to target we can just use npx and x note npx in front is just to reference the local nx installation within the node modules folder and so i'm running npx nx we give it the command that we want to run which is for instance build and then the package we want to run it on which is shared ui and x automatically figures out how the workspace structure looks like it finds the package and it runs the script and according package json similarly we can just do an xdev remix for the my remix app and this would just run the dev command that is present in a package and run remix at localhost 3000. now one of the main features of an x is its caching it allows you to cache an operation and you don't have to execute it again if the input sources didn't change and this is actually what speeds up your mono reaper quite dramatically now by default nx does not have any operation just to be cautious because potentially operations could not be casual so what does the caching operation look like it is like a side effect-free function so whenever you get a certain input you get always the same output consistently let's have a look how to convert that with an x so first of all we create an nx adjacent at the very root of our repo to contain that configuration and let me paste in here some simple configuration this is the task runner configuration for an x so it tells 10x which runner it should use you could potentially switch them out and have different ones even in different environments and then there's a key part here which we are actually want to do now is the casual operation setting and so for our specific case what we want to cache is our build usually you would also want to cache something like test or linting but since we don't have any of those scripts in here right now we are not adding them for now so now given that we have added that cache for the build we can actually try it out we can run npx nx build shared ui and you will see now it runs the build again it took about a second now if i rerun it it is immediate so you can see the whole operation took like six milliseconds and you can also see this note here and x read the output from the cache instead of running the command one out of one tasks and the output however is exactly the same as before as you can see nx just restores the same output it also restores the cached artifacts such as our disk folder if i'm going to remove the disk folder and run it again you can see how it is being restored and our files are still there so next uses a couple of defaults for where usually the output is placed for such build commands but obviously you can configure that another thing you want mine to customize is how nx computes the cache so basically forever operation that is being executed and x creates a so-called hash out of that inputs that got contributed to that operation and it then uses that hash again to invalidate that cache if some of the input changes now looking at the input files here specifically nx by default just uses all the source code files that live in our repository in our shared ui package here however that might have some undesired side effects for instance let's create here a readme.md file hi there now let's run the operation again let's run mpx nx build shirt ui and so you can see now it gets recomputed because we changed our source code however if i rerun again it gets cached now the thing is if i go here and create another exclamation mark so i change the readmd file and then i run the shared ui build again it will not be pulled out of the cache but rather will be recomputed same thing goes if i have test files in here and i change the test files now for a build output however i probably would not care necessarily about the change in the readme file or i would not necessarily want to recreate the build output just if i change some test files so in order to customize that we can again go to the nx json and we can add here what is called target defaults target defaults allow us to specify default configuration for our packages that live in a workspace and so in this case i want to create a target default for our build scripts that live in our packages and what this does is basically it allows me to pass in defaults for things such as like here the build script as well as the build script also for my package in the remix application here and what i want to customize specifically is the inputs and as the inputs here i can provide globs and in order to disambiguate the paths here i have some placeholders such as like the project root so for our shared ui package that would be basically packages slash shared ui slash whatever i specified here and i would want to target all dot md files and i would want to say i want to not include them in my build so now let's see again let's re run the shared ui it will rerun the first time because we changed some essential files such as the nx json configuration it will cache the result a second time now if i go to the readme and i change something here and save this again and then i rerun the command you can now see it is still being pulled out of the cache so an x now ignores that markdown file and therefore doesn't invalidate the cache if we change a markdown file for the build command it would still change it for let's say like the surf command because it might be useful for that or for the publishing command because for publishing we want to have the updated readme included in the actual output now since you might want to reuse such inputs across different targets let's say we have another one here that is called test and we also want to make sure that markdown files don't invalidate our test runs you would duplicate them but you can also do better by using here such a named input section and the name named input section is nothing more than a section where we can define reusable inputs and so we can say something like no markdown and we can just copy the same configuration here to that no markdown configuration up here and then here rather than using the final glob we can actually reference now that no markdown configuration from up here we can even do more to say this doesn't just hold for the specific build command of our own project but also all its dependent projects and so we can basically prefix that with no mark down at the head in front so that sign here and express that it would also hold for all inputs of our child libraries that we depend on so basically our remix app depends on a shared ui and so it would also hold for that one and then we can also copy that over here and you can see how we can now reuse such globs and just change them in a central place in that named input section so alongside these inputs you can also provide outputs if they differ from the default ones that nx comes with and so we can just go ahead and either provide an empty array here which indicates to an x that this specific operation does not have any outputs which would then up to help optimize actual restoring of the cache artifacts or alternatively you can just provide some glob or path pattern such as like instead of this you call it build or something different in our case we don't need it because we have the default output path that nx comes with so another common thing you have in pmpm workspace or monoreapers in general is task dependencies meaning if you run a build or def command for instance in our case it would depend on the output of some other task for instance if we go to our remix application we know that we earlier referenced our shared ui package which internally links again to that shared ui and since the package.json for that reference the dist index.js output that one needs to be present when we run our remix application now let's have a look what happens if we remove that this folder if we go and run our remix application and we can use an x for that so we use nxdev my remix app it just boots normally but then if we refresh it in the browser we can see an error here and it basically indicates that that in this index.js file is missing so how do we fix that well we need to run the build for the shared ui package first so this built it in this case actually we stored it from the cache now if we run our remix app again it would work just as we expect obviously we don't want to do that manually every time we change something and so we can set up such task dependencies and we can do that in the target defaults here so we can go to the build command here and say that this one depends on not just the build command of the specific project but also on the build command of all its child project or dependent projects and we can indicate that with the symbol in front here just as we did before for defining the inputs of the power dependent projects and we would want to have a similar thing also for our dev command so whenever we run remix in depth mode so we would also want that whenever we run dev command for instance for our remix application it will first run the build on all the packages it depends on in our case shared ui so now having that let's actually remove the shared ui package this folder again now let's run again or let's better surf our remix app so what you see now an x here prints out that it's running a target dev for project my remix app and one task it depends on which is our build command and in fact you can see it runs first the shared ui build and then it runs the dev of the remix app and so if you go to our browser refresh we can see it just works as we would expect now such sophisticated caching actually helps a lot in speeding up the operations either locally but also in ci but what you also want to do is you don't even want to run cache hits against something that didn't even get changed in a given pr so to simulate that let's create a small package it doesn't really do anything rather than printing out something to the comment line and so let's create here my custom feature which is just a new package that we have in here and let's actually cd into that packages folder and create a package json in there now we don't really do anything we can remove here the entry point and all we want to do here in the script for instance for the build is you want to print out an echo here saying building and so now again if we do npx nx build my custom features it would just run that echo so behind the scenes nx creates a project graph over here monorepo so let's go ahead and open a graph which we can do by using nx graph which will open up a browser window which is showing an interactive version of our project graph here so you can see now there is the remix application which depends on the shared ui and there is our my custom features library it doesn't depend on anything and no one depends on it the dependency graph is also interesting because you can actually see why such a relationship exists by clicking here on the edge you can even go ahead and do some more advanced filtering by saying i want to start on this one here and i want to understand how the connection between those two exists and why that exists now this is obviously a very simple example but in a larger monoreport it's very interesting to just find all the paths or it's just the shortest path between two different libraries so why am i showing this mostly because our workspace visualization here can help us understand what we need to execute when we change something so meaning if i change my shared ui package i want to also make sure to run for instance tests against my remix app but i don't actually have to touch my custom features package because if that didn't get changed it is not involved in any of these operations or linked to any of these packages or applications so how does that work if we go back here and actually commit this to git and then let's say we create a different branch my feature and we start changing some packages in that branch we could go for instance to our button component here and just insert a new console log if we now add that to git and comment this we can now use the factor commands to just run them against the packages that got changed and so this is done by using nx affected colon and whatever command we want to use now to get started we can also apply this to the graph so let's start with that one first you can now see it shows here a show affected projects rather than just the show all projects and if you go here we can already see that just the remix app and the shared ui package got affected because we changed the shared ui and therefore also remix app is affected by that change if you again show all projects we can see the my customs features is not being considered and will not be used for running targets against it so going back let's actually run some useful command we can run the build command in this case and you can see that it runs against the shared ui build and also against the my remix app build so this way we can avoid running commands against projects that didn't even get changed obviously also this operation gets fully cached and so you can see the output is being restored just the same as it did before you get a note here that this was fetched out of the cache as well as at the end you can see two of two tasks got fetched out of the cache another thing you might have noticed while running this is actually that dynamic terminal output if you remember before when we ran pmpm run in parallel the output would be interleaved so you can see there is one output for the remix app then there's a couple for the custom features app and again remix app so it's kind of hard to parse specifically if you have like 50 or 100 projects in the same pmpm workspace which is totally not like unrealistic so when we run the same command with an x in this case using affected or we could also use the run many command say we want to target the build and we won't run it for all projects we could even filter it just to a couple of projects but let's just run it for all to see the dynamic terminal you can see how the operations at the top here is what got already executed while at the bottom you can see what gets executed in parallel dynamically now in this case again these two have been fetched out of the cache my custom features built which just echoes has not been fetched out of the cache but it doesn't show us any logs that are not relevant for our current operation so to recap what we have done is we have set up a full pnpm workspace monitor we perform the ground up we have configured it with using the pmpm workspace yaml then we have added an x which is really just installing the nx package here at the very root package.json of our workspace and we have done some fine tuning on nx by using well-defined inputs and also defining such a build pipeline or task dependencies usually depends on flag so if this sounds interesting subscribe to our youtube channel we will have more of such content in the future also make sure to check out our twitter account where we push out news and infos around mod repos front-end tooling and javascript

Channel: Nx - Smart Monorepos - Fast CI

Views: 40,031

Keywords: Nx, Monorepo, PNPM, devtools

Id: ngdoUQBvAjo

Length: 32min 59sec (1979 seconds)

Published: Tue Jul 12 2022