Docker integration with WSL2 | Moby Project | LINUXKIT | HyperV vs WSL2

creating a virtual machine let's check if that's the case but before that let's try and install docker on windows and let's see if we face any issues i hope we don't so in order to install docker for windows just type in install docker and windows in google and you will get the first site and you have to click on that and once you click on that you will come back to the docker site and where you will get the download option and you will see some of the information here regarding the docker desktop just click on that and save it and here you have both the options mentioned like wcl2 backend and the hyper-v backend that we will be seeing in a short while so don't worry about that for now we'll just install the docker desktop on windows and we'll see how it works so now that it has downloaded just click on that and start installation install required windows component for wsl2 okay just click ok not a problem we'll go ahead with the wsl2 integration wait for it to complete the installation okay so we have encountered an error so what is that so hardware assisted virtualization and data execution protection must be enabled in the bios okay so what it is saying is for your docker to work you have to enable hardware assisted virtualization and that's what we are going to do right now just copy the site and paste it here you will find the options to basically enable this this can be done using the windows feature on and off so you can just go directly to the windows feature on and off section in your operating system and you can enable this feature and that's what i'll do right now and once it is done what you can do is you can actually validate it using your task manager by going to the performance section you will see the virtualization as enabled so let's see and check the task manager here we'll go to the performance section and we'll see the virtualization here is disabled yes so not a problem we'll enable this go to your windows feature on and off and here you will see the option to enable hyper-v i'll just expand it for you to see clearly what am i trying to do here so first one is virtual machine platform there's already enabled so not a problem with this the second one is windows subsystem for linux so i've already enabled this and the third one that we have here is for the hyper-v so i'll just go to the hyper-v section and i'll just check this so we have a lot of options here but i don't think so we have to enable everything right now just click on this check that and just click on ok and it will start applying the changes once it is done just close this so you can see even if we have enabled the feature on the windows feature on and off the virtualization still is disabled because we have not enabled it in the bios so the hardware resistor virtualization is currently not working so for that what we need to do is we need to go to the bios and we have to enable it so for that what you can do there are two options you can just restart your machine and just hit f12 or f11 based on your motherboard and it will go to the bios settings and the other thing that you can do is you can just go to the recovery section in your windows settings and under advanced setup you can just click on restart now so once your system actually restarts it will ask you for options so you can either continue so that you can exit and continue to windows or you can just move on to the troubleshooting section where you can actually reset your pc and see advanced options so that is where we will go click on the troubleshoot button and you will see advanced option and here you have to select the firmware settings just click on that and restart so once you have restarted your machine you will reach the section where you have your bios settings so this is the bios so once you have restarted you will reach the bios settings and this will be different based on your motherboard so currently i'm using a msi meg x570a so if you're using this motherboard then you are then you are already in the right place you can follow the same instruction that i am seeing right now but for the vmware virtualization of the hardware virtualization settings may vary from motherboard to motherboard and the bios to bios and the version as well the bios version as well so you can just check the manual of your motherboard before actually operating on this one for me i know what exactly to be done so i am doing that you can follow the same if you just want to learn i don't have much to do here so i'll just go to search so in the search you have to just type vm and you will reach to a point where it will show a setting for svm mode so what svm mode actually does is it allows you to enable or disable cpu virtualization so currently as you can see this is in the disabled state so what we have to do exactly we have to enable this just double click on this and click on enable and close this and come back to the settings and click on save and exit save changes and reboot so it will show you what are the changes that you have made you can just click yes and it will restart the system once again so once you go back you go back to the task manager you go back to the performance section now you see the virtualization is enabled so once the virtualization is enabled i uncounted one more error that was wsl-2 installation is incomplete the wsl-2 linux kernel is now installed using a separate msi update package please click the link and follow the instruction to install the kernel update and please make a restart after installing the linux kernel so we'll follow the same procedure i'll click on the ak dot ms wsl 2 kernel link and i'm going to install it just click on the link right now and here you will find the setup for wsl to linux update package just download it save it and install it so this is the setup for wsl2 so once the installation is done you can actually make wsl 2 as the default setting or the default version by using the wsl hyphen hyphen set hyphen default hyphen version space 2 command that you can see here the partial command so that's what we will do right now so just open up the powershell and hit the command and just press enter and then just restart the machine so once it is restarted what will happen is your docker is now started and just click on start you will see all the options here and you will reach the no container running page because we don't have any containers running isn't it and now if you click on settings you will see use the wsl2 based engine so just take a moment and think this is the setting for the docker desktop and if you haven't used docker on windows before you may not see any change to this but in case you already have then you will relate to what i'm pointing at right now if you see it is mentioned here that use the wsl2 based engine as it provides better performance than the legacy hyper-v backend it sounds confusing isn't it but just for a moment if you look at the right hand side we have executed the docker version command and it tells us that the client is having an architecture of 64-bit windows but on the other hand the docker engine itself is running on linux so how is that possible is it a linux vm or is docker still using a hyper-v as i already told you there is a lot more to docker for windows that we actually think let's keep digging on that and don't worry we will do the demo on the tool at the end of this video to understand all the options i know some of you might be thinking i didn't show things that actually matter so don't worry we'll get to that so please watch this till then else you might just miss out on some things that are really very important but before that you need to understand a concept that will help you get to the root of this investigation yes let's talk about linux kit and the legendary mobi project before moving forward what if i tell you that the docker engine is just a linux vm made with linux kit now that we have reached to the moment of truth i want to tell you that docker has always been able to run linux containers on windows desktop since it was first released back in 2016 using a linux kit based virtual machine running on hyper-v yes i said it and it burst your bubble but we can't do much about it so let's understand how we can create one using linux kit so to create containers or customize containers docker came up with a solution called the mobi project it's an open source framework or open framework to assemble specialized container systems without reinventing the wheel of course so with mobi project you will be getting all the necessary library components and what we call the layers to create specialized containers like the operating system container runtime orchestration infrastructure management networking storage security and the build image distribution and it also provisions you the tools to run on cross platform architectures and if you are an enthusiast and want to learn more about the internals of docker you can surely make use of it now let's see how does the linux container running on windows looks like so as we already know running linux natively on windows is not an option we will make use of yes virtualization so you will be thinking every time there is something related to linux and windows this guy just slaps in virtualization everywhere i am with you on this okay have some patience and that's where the hypervisor comes into the picture next we need the placeholder on windows that can support the container host so this is the windows container host where you have your docker client which sets the properties and settings and calls on to the docker daemon which is present on the mobi vm or what we also call the linux kit vm os which is also your linux container host you should understand by seeing this image that docker client sits on the windows host but actively communicates with the docker daemon on the linux host but why because it helps the docker demon to listen to the api calls and manage images and containers and other docker objects and as this is a linux vm running on the hypervisor the linux process containers on this linux host will have consistent storage and networking handy isn't it and that's the reason why when you see an older version of docker you will find resource allocation available to you on your docker engine properties when you install the docker desktop for windows with hyper-v that's why a lot of people had doubts as to why do we need to limit resources docker engine and when you change these settings why does docker need to restart if it's not a vm or if it's not using hyper-v but if you look closely the path to the disk image location is pointing to the mobi linux vmdk so yes docker is or was running on the linux vm which made use of the hyper-v for resource allocation but wait a minute i just said it was running on hyper-v or should i say it's still running let's keep that thought in our mind so now we have reached a position to answer these questions so can you run docker natively on all the operating systems are containers same as virtual machines and is docker for windows and linux vm installed on the windows machine and the answer would be no no and yes but having said that do we have a solution for this virtualization concept and for that we need to discuss about wsl or windows subsystem for linux so is this the ultimate matchmaker for windows and docker it's not that simple so keep watching so when we installed docker you might have seen we ended up with errors related to wsl and finally we got it sorted by installing a piece of software called windows subsystem for linux or wsl so what does wsl actually help us with so this is something that you can see here it's been mentioned that run linux on windows install and run linux distribution side by side on the windows subsystem for linux interesting isn't it but what is wsl so wsl is a feature of windows 10 that enables you to run native linux command line tools directly on windows alongside your traditional windows desktop applications so you might be right in thinking that if it is a feature then you might have an option to enable it on windows isn't it yes you are right and you can do that using the windows feature on and off like this so let's see what else wsl can help us with so with this you get the capability to run linux in a bash shell with distributions such as ubuntu alpine and other great distributions but of course you need to install the distros first and that is a very important feature that wsl provides us and it's really a boon for many developers who wish to work on linux alongside windows so wsl actually helps us to run a gnu linux environment including most command line tools utilities and applications directly on windows unmodified without the overhead of a traditional virtual machine or dual boot setup so you remember the days when we used to install a linux operating system along with the windows using dual boot and we had to choose the os before logging into the system yes that's old now so with wsl you get to run your favorite gnu linux distributions like ubuntu alpine fedora as i already mentioned just now and if you want to run linux command tools you can do that too and you can run bash cell scripts and command line applications as well and you can as well install additional softwares using the distribution package manager and most importantly you can invoke windows applications using a unix like command line shell that's something we can try isn't it that should be fun and also we can invoke linux applications on windows but as we discussed before docker went for the integration of wsl2 but before that we had a wsl generation one so let's understand why wsl1 was not a good option and what is the difference between wsl1 and wsl2 so let's start off by comparing both the architectures so i know some of these topics might be very tricky and you might feel you aren't able to catch up but trust me i'm trying my best to keep this as simple as possible for users who are new to this because each word that you see here can be made into a separate video so don't worry if you have any doubts we can clear them in the next one okay let's get back so regarding the differences between wsl1 and wsl2 what microsoft tells us is that the primary difference and reasons for updating the windows subsystem for linux from wsl1 to wsl2 was to increase full system performance and support full system call compatibility but what was the problem and what was the solution so let's find out by checking what was the architecture that made the difference here so if you see the architecture here you have the nt kernel and the wsl sits in between the linux distributions and this wsl1 here acts as a translational layer which helps interpret the system calls okay if you don't know what system call is so let's suppose you want to execute a program so you have to tell your operating system by clicking on the gui that please execute it by performing an operation isn't it but in reality there will be a service that will execute it and for that you have to talk to the kernel of the operating system who can give you the service that actually does the job and the programmatic way by which a computer program requests a service from the kernel is basically the system call so here the wsl1 acts as a transactional layer which helps interpret the system calls so that they can work on the windows nt kernel and this is why it became slower and slower when the system calls increased when it came to performing operations such as accessing files requesting memory grading process and that's where we went on to the wsl2 the no transitional layer approach so here you get back to the vm approach using the hypervisor these guys just went back to the hypers i'm sorry let's let's continue with this so now the window subsystem for linux 2 or wsl 2 started using a virtual machine with an actual linux kernel that can respond to system calls much faster than wsl1 and i was thinking what is with not letting the virtual machines go and if we had to always come back to the virtual machines then why did we criticize them in the first place but hold on it's not just any other vm it's a lightweight virtual machine which uses a microsoft developed open source linux kernel so it better be good isn't it yes absolutely it's good and i have tried it myself so and now that wsl 2 includes its own linux kernel it has the full system called capability so now you get better file io operations you get better network operations and as well you get better performance than wsl1 so now that you have the linux kernel you can use it with the docker as well isn't it so let's see some of the comparison between wsl1 and wsl2 so with both you get the integration between windows and linux faster boot time and you have a smaller resource footprint compared to the traditional virtual machines that you have so footprint actually means how much space and resources are consumed by the software or hardware so that's why a reduced footprint for resources is always a better thing to aim for and both actually can run on the current versions of vmware and virtualbox and next is even though wsl 2 runs on vm you don't have to manage it it only uses resources when it needs and the best part is it runs behind the scenes so here you will get full linux kernel support in wsl 2 and full system call compatibility and the last point that we have here is performance across os file systems that you see it's not that great with wsl 2 so what microsoft tells us is that you should avoid using cross operating system file systems for your files and for the fast performance speeds store your files in wsl file system if you're working in a linux command line and if you're working on the windows command line store your files on the windows file system makes sense isn't it and now that you know the vast improvements with wsl2 that's the reason why around october 2019 we had wsl2 integrated with docker and docker had mentioned as once wsl 2 is generally available the wave will automatically switch on to the wsl2 backend on compatible machines like windows 10 of course and basically moving away from the hyper-v backend and that's what we experienced we are now using the wsl back-end and what docker did was it listen to its users and that's something that i'm happy with as well but having said that they actually wanted to change a few things regarding the wsl integration so the first thing running in an isolated environment so in order to avoid interlocking and other side effects from other apps running with wsl2 they wanted to run it on a separate namespace for network pin and mount the namespace is like the kernel partitioning its resources so that one set of the process sees one set of resources and another set of process sees another set of resources or in simple terms so that they don't overlap or collide with each other there are various namespaces like username space mount namespace network namespace and more and you can read about them in the documentation second they didn't want to re-implement everything that they had already done with the hyper-v vm they did not want to take the additional overhead for that and the third one was to have complete integration with the existing ui so that it does not create confusions for the users currently using the application that's good isn't it now let's see what and how things changed from hyper-v vm back-end to wsl-2 based engine for docker so before moving on to the wsl-2 integration with docker let's get a better context and talk more about the hyper-v back-end architecture so first things first by now we are already sorted and we are fully sorted and we know in the hype we backend the linux vm that docker used ran on hyper-v which was built using the linux kit which we already had discussed a couple of minutes back i hope you remember that so here running a linux kit vm gave docker the flexibility to incorporate the components that they wanted which could be used for both hyper-v and mac vms so what they did was they packaged all the components into the docker desktop like lifecycle control service diagnostic service aggregation log service into a iso that was docker desktop iso and on top of the base distribution they mounted another iso that was version pack iso which contained the binaries deployment scripts and upgrade scripts related to the specific version of docker engine and then in order to store container images and configuration files they attached a virtual hard drive before starting the virtual machine and they did a very interesting and logical thing by introducing a proxy that would expose the unique socket as windows named pipes using the hyper-v sockets so that the services mentioned above could be reachable to the windows side smart isn't it so if you're not aware of what name pipes are you have to ask yourself are you aware of pipes in general so the easiest way to remember pipes is to imagine a real pipe so you can push water from one end to another using a pipe so it tells you the pipe velocity and the amount of water that can flow from it so one opens the tap and the other collects but in windows the pipes are a bit more complicated than this so pipes in windows are used for inter process communication or ipc where a section of memory is used by process for the sake of communication so the process that creates the pipe is called the pipe server and the one that connects to that is the client pipe so one process rights to the pipe and the other one reads from that and there are two types of pipes so one is anonymous and the other one is named pipes so anonymous pipes or what we also know as pipes in linux are used for simplex communication or in other terms we say we use it for one-way inter-process communication and this is unidirectional and here from one end the process a can either write and read and from another end the process b can read and write but even though we can perform read and write operations at the same time this cannot be used over the network that's why we jump onto the named pipes in windows we also know this as make fifo for people who are well versed with linux so they might know this as make fifo so this is more popular because it supports client server architecture or kind server communication and it also as well supports two-way or duplex communication so this way the server can communicate with multiple clients and if the server is communicating to the client one then client two and client three will be in the wait state but don't worry there are both synchronous and asynchronous name pipes that you can make use of but that's a topic for another time and the advantage here is that name pipes can be used over different systems over the network as well now let's see the wsl2 implementation of docker so here if you see it may not be that obvious but unlike the hyper-v back-end the linux kit vm or the distribution that was running in a vm is now changed and is running in a container but for the process to work we need to create namespaces isn't it and as docker mentioned before they wanted to have separate namespaces so for this docker actually creates two wsl distributions which are docker desktop and docker desktop data so docker desktop here we'll call the bootstrapping distributions which eventually will create the linux namespaces and docker desktop data we'll call the data store distro which will help us for yes to act as a backup store for container images instead of using the virtual hard drive it's more evident when we see them side by side so you can as well understand just by looking at the comparison here that docker is hinting that the bootstrapping distribution replaces the hyper-v and the data so distro replaces the vhd or the virtual hard drive and moving on for the file and network shares the bootstrapping distro manages the mounting for windows 9p shares so if you know 9p or plan 9 is a file system or mounting protocol that is used in a distributed environment so the linux container can make use of it and this also manages the lifecycle of the linux kit container and last but not least the version pack iso that you see here remains the same as it was in the hyper-v back-end and what are the advantages to this yes it makes docker achieve 15 times faster start time it provides dynamic resource allocation and now it can run on environments with lower memory as well cool isn't it when we installed docker we installed it with wsl to enabled and that is why we did not get the option to provision spaces and resources to the docker engine but how does docker get this dynamic resource allocation so as we now have realized the main objective for docker was to move away from the concept of managed vm so with wsl-2 you get a feature for dynamic memory allocation which helped the cause for better performance what wsl provided was a property of memory reclaim previously when we created vms the memory would be kept occupied even after we were done with the workflow cycle or by the linux kernel and thus the memory requirement would increase for the wsl2 vms but with memory reclaim the amount of memory which is no longer needed by linux will be returned back to the host and thus reducing the footprint this as well includes the caching and you may ask how so listen to this very carefully so this is a linux kernel feature and what it does is that it allows block of contiguous memory to be returned back to the host if and when they are no longer required by the linux guest so what wsl did is that they incorporated this feature to the linux kernel or wsl2 using a kernel patch and updated the hyper-v to support page reporting so if you know linux already had this feature long ago which was free page reporting which is basically an api by which a device can register a receive list of pages that are currently unused by the system so that is the same thing that they have tried using but by using the patch so in wsl the host would periodically compact memory in order to ensure free memory is available and it's not that you can't change it yes you can do that now as well by modifying the dot wsl config file if we have enough permissions and this actually happens when the cpu is idle and you can validate this by looking for the message performing memory compaction in the d message command output but what if we still want to use the hyper-v back-end approach can you do that yes you can let's check it out so what you can do is you can go to the settings and here you can see the option in general use the wsl-2 back-end engine wsl-2 provides better performance than the legacy hyper-v back-end what you have to do is you have to just uncheck this and you have to apply and restart okay so the docker engine has started and i can just go to the settings and now as this has been disabled the wsl2 based engine is disabled it will fall back to the hyper-v back-end yes now you see the resources available to you here is where you can change it docker desktop vm data so you can assign the number of cpus that you want you can assign the amount of memory that you want it's currently at 2 gb swap is at 1 gb disk image size is 64 gb there's the file sharing settings there's the proxy there's the network this remains same but the option that you get is the resources that you can change going back to the docker engine so this is the configuration file and here you can enable the kubernetes as well which will start a kubernetes single node cluster when every time the docker desktop actually starts so you can make use of it but we'll see this after the docker session is complete so we'll move on to kubernetes after that so that was interesting isn't it but you might still think that is it that we are going to do away with the hyper-v back-end approach but among these two which one is the most practical solution and for now i would say it depends and we also haven't discussed about how process isolation actually works among the docker objects and i know you might be thinking we still haven't executed our first docker command but for that you need to watch the next session which will be coming up shortly on this space that you are right now because this is the end of part 2 for the blueprint series so make sure that you don't miss out on any of these sessions and for that please hit the subscribe button right now and and please press the bell notification icon as well these videos take a lot of time to make so please make sure that you hit the like button and you let me know on what you liked and what you didn't and if you wish to support the channel or buy me a coffee then you can check the links in the description below so that's all from my side today don't miss out on the next session because we will be learning something very very interesting until next time stay safe stay healthy its pytholic signing off

Channel: Pythoholic

Views: 1,089

Keywords: docker for windows 10, Install Docker on Windows 10, Enable CPU Virtualization on MSI ACE x570, Understanding LinuxKit and Moby Project, What is Windows Subsystem for Linux 2, Difference between WSL1 and WSL2, Docker Integration with WSL2, Difference between Docker HyperV Backend vs Docker WSL2 Backend, WSL2 Dynamic Resource Allocation, windows subsystem for linux 2 install, windows subsystem for linux vs virtual machine, How does Linux Distro work on Windows

Id: QCGaI1bh4eM

Length: 30min 1sec (1801 seconds)

Published: Mon Jun 14 2021