3 Gotchas with WSL 2 around Disk Space, Memory Usage and Performance

hey there nick janetakis here in this video we're going to go over three gotchas when working with wsl2 so the first one is wsl2 using up to 80 of your memory by default uh we're going to go over how we can keep that in check the second one is going to be around wsl2 not reclaiming disk space on your drive after you delete files within wsl2's file system you know this means over time your disk is going to get more filled up and you know we need to keep that in check as well so we'll go over some ways to control that and the third one is going to be how to comb that really poor file performance when you're accessing mounted drives over uh wsl2 now i don't want to make it seem like this video is click bait as of july 2020 there really is no reasonable way to get really good file performance if you're working over mounted drives so it is highly recommended that you do copy your important like source code files directly into wso2's file system and then it's lightning fast but the workaround for right now if you need to deal with uh you know doing file operations on like tens of thousands of files on a mountain drive uh you might just want to keep a wsl1 instance sitting around just to run those commands from there so right now i am running ubuntu 2004 in wsl2 that's what we're looking at over here but i do keep my old ubuntu 1804 wsl1 instance around when i do want to grip through you know tens of thousands of files that's on a mounted drive so when i say mount to drive like if you run mount l here you can see all sorts of uh you know different mounts that are available within wsl2 or you know wherever you decided to run that command but you can see here i do have a couple of drives here that are you know mounted into wsl2 you know i have my c drive and i have a you know that's my ssd then a d and e are like a backup like one terabyte hard drive and then fng is actually an external usb drive and you know if i want to access these i can just go to cd like uh d over here and now i'm in that d drive and i can just you know do whatever i need to do in this drive you know this is where the uh slow file performance lies like if i were to grab through you know a massive directory here with thousands of files and you know searching through text files or whatever then that is going to be very slow but you know if i go to something like my source directory here which is in my home directory this is actually living within wsl2's file system and the performance here if i want to access any of these files are very very fast so don't worry about that if you have everything copied into your drive then you don't really need to worry about slow file performance but now let's go over the other two because those are super important and they will require you to do a couple of things to keep them in check so if i head over to github over here and i'll make sure to link to both of these issues that we're going to go over here in the description you know you'll see that this is pretty uh severe i guess like there's 275 comments and basically the idea is as of right now as of july 2020 wsl 2 will just consume like 80 percent of your system's memory by default and you know we're not going to go over this entire issue but there's some comments here with like tons of upvotes and like you know like this one uh basically it shows you how we know what you can do to maybe get around this so this one right here you know it says you need to create this dot wsl config file in your user's home directory so if i actually drop back down to my terminal here you know i did do that so in my case you know my windows username is nick so in my users directory for nick i created this wsl config file now the really important line here is this one here on line two and by the way just to be clear like where this exists inside of windows you know if you go to your windows drive here and you go to users then you go to your username right here is that wsl config file and you can see here this this line is the one that's really important so memory equals six gigabytes on my system right now i have 16 gigs of memory and before i created this config file uh wsl 2 used something like 12 gigs of memory just because i was running a couple of docker containers and then i stopped them but then i you know i'm a freelance developer so i'm working on a number of different projects so you know i hopped over to a different client's work and you know i did a docker compose up i ran a whole bunch of stuff there you know i was done for the day and then like you know i i did this for like literally like two or three days and i got up to using 12 gigs of memory and my system started to actually like it became noticeable that like hey you know something weird is going on let me you know bring up task manager and it was like holy moly like i went to performance here and my memory usage was at like you know like 15.1 gigs it was like basically everything so what you can do is you can just set this memory equals six gigabytes or in your case you know if you have less memory you know maybe if you have something like eight gigs of memory maybe you can set this to three or possibly uh even four maybe uh this is basically going to limit wsl two to only ever ever ever use a maximum of this amount of memory and nothing more so that will prevent wsl2 from overtaking most of your uh your memory and then this line over here the spot file one i like to keep it around just to make sure that you know if this memory does get full you know things are not really going to start to crash but instead they'll get written out to a swap file and i like to keep mine on one of my regular hard drives not my ssd so you know i did a custom path here the only noteworthy thing here is you do need to use double backslashes here so be be mindful of that now when it comes to this memory usage if you bring up your task manager you can just hit ctrl shift escape and you go to your list of processes here and you sort by what's using the highest amount of memory we have vm mem and this is the culprit this is your wsl2 vm now right now i am only using about 650 megs of ram here which you know isn't that that that bad you know it's the whole entire os is booted up really i guess you know whatever wsl 2 does behind the scenes and i've run a couple of containers in the past but uh i recently ran a command that will reset this basically to the lowest amount like normally in my day to day if i'm working with docker and i'm just running like docker compose here and there you know this number will be closer to that six gig amounts pretty much all the time so there actually is another way to control this so let's say that you set your limit to six gigs and you just don't want it to sit there at six gigs all the time and you want to reclaim some of that memory back so there is this alias that you can run and this alias by the way is inside of my dot files over here so if you go to my aliases file and scroll to the bottom then you can copy paste this here and drip you know drop this into your like bash aliases file or whatever alias that you have and let's just go back to the editor because it's a bit easier to read there so this also came from this uh github comment here not this guy's comment but you know there was something else i forgot who wrote it but it was definitely taken from here and they even referenced that here in the comment right so here we go right that's the actual comment and i mean the tl dr in this one is you just basically run drop underscore cache as your alias so if i opened up like a new window here i can just run you know drop cash here i'm not going to do it right now but if he did do it in about 30 seconds or so all of that memory that wsl-2 was holding onto is going to be freed up so you can see this command here i'm not going to bother walking through it completely but uh yeah this is basically what you need to do i hope in the future that microsoft does solve this problem this feels like something that could be managed at the wsl2 level so we don't even need to think about it i really hope that exists because right now you know unless you're like a hardcore power user it's kind of hard to recommend someone to use wsl too because like it's on you to keep these things in check so that controls memory usage you know between this config file you know limiting the memory usage and you know maybe dropping the cache then memory usage is going to be pretty good honestly i don't even really run this drop cache that much because on a 16 gig system using six gigs of that still leaves me with 10 so very rarely do i ever get to the point where i'm totally pressured for memory on my system you know even if i'm running you know like video recording tools and video editing tools and things like that and some games like it's it's pretty much hands-free so now let's go over the second issue here which is uh wsl 2 not reclaiming disk space back to the host os now this one is super kind of annoying because i got killed by it a couple of days ago so and this actually leads into another thing but let's focus on the disk space for now so the basic problem with this is let's say that you're operating inside of uh you know some wsl2 file system right you're you have all of your source code you have blah blah blah you know maybe you copy in some like multi-gig files for whatever reason maybe some like you accidentally copied a directory that has a whole bunch of like wav files in it which is kind of what i did at some point uh for my backup script and the issue with this is you know let's say we were in this directory right now like you know the home directory of my website or whatever not important at all but you know let's say i was in this directory and i just had like i don't know 20 gigs of wav files just sitting here or a zip file or something like that or you know mp4s and you know i just run rm to delete these files now what you would expect is you would expect that if there were 20 gigs being used and you deleted these files those 20 gigs would be reclaimed but that's actually not the case when it comes to wsl2 because the way it sizes the the virtual uh the vm's disk or whatever it just doesn't give you that disk space back so what i found was one time you know i just woke up and i i opened up like windows explorer and my main c drive here with me you know which is where the wsl2 vm is created on instead of having 112 gigs free there was like 16 megabytes free and this bar was all the way to the red here and uh it was pretty bad like somehow the machine still worked and it booted up and and all that but man like you got to be super careful about that one and like the sci topic about this one is and it's sort of related to the disk space 2 is wsl2 will not auto mount external you know or drives that you connect after wsl2 has been booted up so for example you know if i run amount l here then we can see all of these mounted drives here right and this f drive is my external usb like one terabyte you know western digital passport drive or something like that and this is my backup drive so i actually do run a shell script that r syncs files over from both wsl2 and like my d drive onto this and this is where like my backups occur so the problem i had was you know i just moved into this new home office area and part of moving my computer you know i disconnected my external drive and you know i booted up into windows and i'm like oh you know now's a good time to do a backup i just finished doing some work or whatever and you know i plugged in my my usb drive and then i ran my backup script like i do normally and then i basically you know just closed the window and went afk because that's typically what i do you know i back up like my stuff uh once a day i don't worry about it i just run the command and walk away and uh you know when it came back the next day that's when i noticed that wow like our sync was pretty busy it just copied over like 150 gigs of files or whatever or you know just to that point where it almost filled up this disk and that's a problem because normally you know the way rsync works is it would only copy over the files that changed between what's you know on the remote and what's in the source so you know there was probably like i don't know like a meg of data or something that should have been copied over but all of that stemmed down to uh a pretty subtle thing where the wsl2 did not auto mount my external drive when i plugged it in however when i go to cdf here then this is actually a directory that exists but here's the interesting thing if i go to the root file system here of wsl2 and i do an ls there we can see it's a little bit weird to read this because the coloring is a little messed up but if you take a look here you know there is the f over here like this is the the f drive or the f uh mount but when when uh to get the amount to work like i had to like explicitly create this f directory here so like long story short what really happened was it copied everything from a different directory in my wsl2 file system and my d drive into this f directory here but it wasn't like literally mounted to my external drive it was just like copied to that folder so it just filled out the wsl2 file system and uh to get around this problem of just generally reclaiming all of your space let's actually jump over to this github issue someone had a pretty good like semi-copy pasteable solution here which is this over here and you know i'll link to this one in the description but the basic idea is what you need to do is open up powershell as admin and then you you run these commands now these commands this specifically the optimize vhd one is only going to work when you're using uh windows 10 pro edition uh maybe enterprise as well but if you're running windows 10 home there's another solution you'll need to do i'll link to that one or you know i'll find it up here in a second so we can go over it but the basic idea is this so you would want to copy this command here just up to the point where we're in this packages directory then you'll want to open up a new powershell prompt but you can't really do it regularly you'll have to open up as admin now you're not going to be able to see this but you know off this like recorded area of my screen i'm actually going to right click my windows terminal icon in my taskbar and then when it says windows terminal you can right click that and then open as administrator actually like i'll throw up like a screenshot of what this looks like when i'm editing this video but you will need to run this thing as admin and if you just saw the screen go a little dark that was because window was windows was like by the way you know do you really really really want to run this as admin and uh the answer is yes so let me just zoom in a bit here so we can read this but i guess it doesn't really matter but anyways you'll want to open up powershell here and i'm just going to resize this a bit my resizer command yeah it doesn't seem to work here because i'm running this as admin but the sizer tool is not running as admin so now i need to like carefully manually resize things that's actually in my wsl2 like tools i use video if you happen to watch that video the sizer tool saves a lot of time because it saves me from having to do like what you just saw so if we go back to this github comment here you know it says you need to cd into this directory cool so we'll do that and you can just paste that in you know this dollar sign env is going to make sure it goes to your windows username and then from here it gets always confusing when you're working with powershell because you have to use dir so if we go back over here to the github comment you know we can see here that if you are running something like ubuntu 2004 you'll have to cd into that specific directory but the path is going to be a little bit different you know this number on the end here is probably going to be different on your end versus this comment here and what's on my machine but the takeaway here is we need to go to uh ubuntu 2004 so if you are running ubuntu 2004 then you can just copy up to here or if you're running a different disk drill then you'll just need to replace that but the idea here is we can just do a cd into this and then we can hit tab to complete that and then that will give get us into the correct directory here and then you can just cd into local states and then from here you know this is where this you know ext4 this is basically the vm file on disk and it is pretty big so if we go back to the comment here the first step that we would need to do is to run wsl-shutdown now i don't feel like running that on my machine now because i have a couple of things running in wsl but this step is super necessary because what it's going to do is uh in order to run this command which is basically shrinks down that ext4 like you know this vm file here uh the vhdx file ext4 is the file system type but in order to run this command there absolutely needs to be nothing being used from uh wsl so the shutdown command is what uh you know ensures nothing is running like wsl 2 is not running nothing's connected to it etc etc so you would copy that command and it will take about a second you know it's very fast to run and then you would run this command here which is the optimize vhd command and we're going to run it on that file and basically this is going to shrink this vm file to be whatever it really needs to be based on the disk space being used so if uh you know if i go back to the other terminal i had open here you know if you had all sorts of like you know wav files or 20 gigs of files and you remove them then after you run this one shrink command here this optimized one that is going to reclaim that space and you know i do recommend that you keep an eye on your disk space because you know if you're copying files over you know here and there then your drive space is going to continue to get used and used and used so you know i wouldn't say run this on like a schedule like every week or something like that but you know if you notice your disk space being used and maybe this gets a little you know maybe you're at like 90 gigs and then like 85 gigs it's like whoa you know maybe i can reclaim that because you know every couple of weeks if i run this command i've noticed that i've gained about like 15 gigs back so it is totally worth running this every once in a while and once you do that then you'll just be able to launch your you know favorite terminal here like he says and then you'll be able to just run wsl 2 like normal now if you're not running windows 10 pro then there is a second option here i don't know here it is for windows 10 home alternative so in running instead of running that optimize command you know you'll still need to do the shutdown command you'll still want to go to that one directory but this is the part that we'll need to change so you'll be running these commands instead and you'll have to pass in i guess like the full path to that vhxd file which should be in the current directory if you followed along to this point here where we're inside this directory here you know there's that file over here so you can just pass that in as the path you know i haven't run this command personally but you know it's got a lot of upvotes seems to be working successfully and if you read through all of these comments here you know there are people that were like yo thanks a lot like this worked great so you would just run these commands here and you would be good to go on windows 10 home so that's just a couple of gotchas of working with wsl2 for now i imagine in the future both of these issues the memory one and the disc one will be taken care of automatically because i would i would imagine microsoft can write some scheduling thing in to do this in like a non-destructive way you know every you know x amount of hours like whatever you know any whenever it needs to be done so that will be great when that happens but until then you know just keep an eye out on the memory and disk space usage because the last thing you want to do is wake up one day and notice that you have like 12 megs free and like your uh you know your memory is at like 99 so with that said hope you enjoyed the video if you have any questions or if you have any issues running these commands uh let me know and i'll do my best to help anyone in the comments also if you like the video please give it a thumbs up because it really helps thanks a lot for watching and i will see you in the next video

Channel: Nick Janetakis

Views: 24,894

Keywords: Windows 10, WSL 2, Development Environment, Performance

Id: 4PwClrUCqJM

Length: 19min 52sec (1192 seconds)

Published: Sat Jul 11 2020