Proxmox GPU Virtualization Tutorial with Custom Profiles thanks to vGPU_Unlock-RS

with as many videos as i've done about virtualized gaming pcs people have asked how long am i going to milk this concept i think i can get at least a couple more did you know that two out of every three men will experience some form of male pattern baldness by the time they're 35. every network admin worth his weight in scotch knows the value of preventative maintenance so why not apply that same mindset to your hair that is where keeps comes in keeps is a subscription service designed for men who are dealing with uh premature deforestation with keeps a licensed doctor will review your information online and recommend the hair loss treatment that's right for you and don't worry about forgetting your prescription at the pharmacy while you're on a beer run instead your prescription will be shipped to your door every three months now the hair on top of my head is so thick i can barely get a comb through it seriously my barber charges me double and i want to do my best to keep him employed keeps uses generic fda approved medications and since there's no name brand markup you'll save money while keeping your barber in business for years to come if you're ready to take action and prevent hair loss go to keeps.com craft or click the link down in the video description and receive 50 off your first order that's keeps.com craft and thanks again to keeps for sponsoring today's video sorry i was enjoying that beer so much i forgot that i needed to film something welcome back to craft computing everyone as always i'm jeff and this is the cloud gaming pc that i built just a week ago for less than a thousand dollars now i get a lot of questions as to what actually is a cloud gaming pc this is not a pc that you build to be able to hook up a monitor and keyboard to and play games on it rather this is meant to be a distributed computing system a server that you can place in the corner of a room hook it up to network and then stream games to it to devices as simple as a laptop a chromebook or even a raspberry pi what that means is you don't have to build every single person in your household a pc who wants to get into pc gaming instead you can have a single system to run everything last week we focused on the hardware section of this that is putting everything together and making sure we have enough power to run four 1080p gaming sessions at the same time today i'm going to focus on the software aspect what's required and how you can get up and running yourself as a quick refresher this system is running an intel xeon e5 2690 v3 12 core 24 threaded processor on a jinx shot x99 motherboard we've got two nvidia tesla m40 enterprise gpus with 24 gigabytes of video memory each on top of that we have a one terabyte nvme drive and 64 gigabytes of system memory now one thing i've removed from the system for right now is the giant blower fans that were keeping the tesla m40s cool this system will not be able to run games without some active cooling on those cards but if these were plugged in you wouldn't be able to hear a word i was saying in a future video i am going to try one last time to cool these graphics cards and make it room friendly so you can keep it in a living room or a bedroom and not lose your mind to get started there's only a couple pieces of software that you'll need to get for starters we're going to install proxmox 7.1 which as of the filming of this video is the latest version you'll also need to create an nvidia enterprise account and apply for a trial license for their virtual workstation software this is actually the software pack that we will be using to enable vgpu support on our definitely not enterprise hardware last but not least you'll need to download the pve headers that match the kernel version that you have inside proxmox now this can normally be done through the app repository and even though proxmox 7.1 is using the 5.13 linux kernel the latest version in the repo is only 5.11. so a manual download is required links for where to download all of the required files will be down in the video description and with all the prerequisites out of the way let's go ahead and get started on the system right now is a 100 clean install of proxmox 7.1 and we're going to start off by installing all of our required software lucky for you i've set up a text file where you can simply copy and paste pretty much everything in the instructions again link is going to be down in the video description to start off we're going to set up the non-subscription repository for proxmox to make sure our system can stay 100 up to date there we go then as you should on any new install we're going to do an app update followed by an apt upgrade once everything is up to date we do have a couple required packages and that includes get build essential pve headers dkms jq and mdev ctl we'll also need to clone a couple git repositories starting with the secret sauce for this whole thing which is the vgpu unlock script by dual coder there's also a new rust based script by user m bilker and that is the vgpu unlock rs so go ahead and clone that repository as well and since the rust script requires rust we should probably install rust and that can be done with a very simple curl and sh command we're going to select one to proceed with the installation and then just wait for it to finish last but not least are the two packages that we had to download manually so here we have the nvidia vgpu driver which is just a 30 megabyte driver file and also the pve headers for the current kernel version which is 5.13. and we're going to simply copy those over using winscp with all the required packages ready on the system we can start moving to system configuration and the first of which is enabling iomu and pci express passthrough to do that we're going to edit our grub file so we're going to type in nano slash etc slash default slash grub scroll down to the line that says grub command line linux default and we're going to comment that out with a pound sign or a hashtag then depending on your cpu architecture we're going to copy in either the intel or amd i o mmu setup in my case i'm running an intel xeon cpu so we're going to use quiet intel iomu on and inside nano hit control x to exit y to save and enter to confirm the file name to apply those changes we're going to type in update dash grub there are also some required modules for this that we need to start up at boot so again we're going to type in nano and then etc modules it should bring up a file that looks like this essentially blank with some commented outlines at the top and we're going to copy over the vfio modules that are required so vfio vfio i o mmu type 1 vfio pci and vfio vir qfd and again ctrl x to exit y to save and enter to confirm the file one of the things included in the linux kernel is the nuvo driver which is essentially an open source replacement for nvidia graphics drivers but as we're going to be using an official nvidia graphics driver we don't want that one working so we're going to blacklist that by typing in the following command that is echo blacklist nuvo and we're going to place that in etc mod probe d slash blacklist.conf and to apply that change we're going to type in update dash ini t r a m f s space and then dash u this will update all of the changes that we just made to the kernel and let us proceed to the next steps once the update is done we're going to do a quick system reboot so reboot once your system has booted back up and you've reconnected with ssh we can move on to the really fun part which is the configuration of the nvidia drivers first thing that we need to do is install the pve headers that we should have installed before i rebooted the system but we can go ahead and do that now to do that we're going to type in dpkg dash i followed by the pve header file name next we need to make the nvidia vgpu unlock script as well as the nvidia driver installation executable so we're going to type in chmod dash r plus x and then vgpu underscore unlock for the directory and then chmod plus x followed by the nvidia installation package with that done we are ready to install our nvidia driver so we're going to execute that by typing in a period and a forward slash followed by the nvidia package installer file name but then we're going to add dash dash dkms to install this as a kernel module inside the installer you'll just have to answer a couple quick questions starting with would you like to register the kernel module as a source with dkms and yes we would when that's all done you should see a message that says installation of the nvidia module is now complete and go ahead and click on ok inside the newly created driver files there are a couple of lines of code that we need to change to hook them into the nvidia unlock script first of which is going to be modifying the file at user source nvidia 510.47.03 or the driver version that you just installed slash os dash interface dot c and this one is pretty simple we're just going to scroll to the bottom of the include lines and paste in the following line from the v gpu unlock script to include a c file within that directory and yet again hit ctrl x to exit y to save and enter to confirm the file next up we're gonna add a line of text to another file and this one is in user source nvidia 510 47.03 nvidia dot k build for this one we're going to scroll all the way to the bottom of the file and paste in the following line and yet again ctrl x y and enter with those two lines added we can run our rust installer and complete the vgpu unlock installation to do that we're gonna go into the rust directory so cd vgpu underscore unlock dash rs then you're going to type in cargo build dash dash release next up we need to make a vgpu unlock service so the service will actually start up when the pc does to do that we're going to create a couple directories starting with nvidia vgpud.service.d and the nvidia vgpumanager.service.d inside both of these directories we're going to create a vgpunlock.configuration file and that can be done by typing in nano followed by the directory and vgpunlock.conf and inside these files we're going to paste the following line so it will actually start up the gpu unlock service yet again control x y and enter and we're going to do the same thing for the vgpu manager service paste in the same line control x y and enter with all that done our vgpu driver should now be installed and running but we do need to reboot the system first so go ahead and do that now yet again with your system booted back up and reconnected to ssh we can verify that the nvidia vgp unlock script is actually working to do that we're going to use the utility mdev ctl which is a program that will let you split out different aspects of your hardware including nvidia graphics cards to view our available profiles i'm going to type in mdev ctl types and hopefully i get this list right here which displays all of the possible bifurcations of our nvidia cards if you see a list that looks something like this congratulations vgpu profiles are now unlocked and ready for you to use if you don't see any output from this command you'll want to go back and check all of the steps from this tutorial with all that out of the way we're going to move away from the ssh session for a little while and focus on the web gui the next step is setting up a windows vm but before you can do that you likely need to upload a windows iso so your virtual machine has something to boot off of to do that click on the local storage from your proxmox server and then go over to iso images and click upload now i can't supply you with a download link for a windows iso but pretty much any windows iso will work so upload whatever you have and we'll move on to the next step to create our windows vm i'm going to right click on the proxmox server and go down to create vm first couple things you'll need to do is create a vm id which is a three digit number and this can be pretty much anything you want so i'm going to type in 501 and then give your vm a name i'm going to name this one m40-001 click on next to go to our operating system setup and this is where you will link the iso file you just uploaded to this virtual machine in order to boot off of it i however i'm not going to use that method i'm going to say do not use any media because i plan on booting off of my windows deployment server and if you're curious on how you can deploy windows through the network i do have a tutorial for that and i'll link it right up there under guest type go ahead and click on the drop down window and go to microsoft windows and then click on next starting with proxmox 7.1 it will default to creating a virtual machine with whatever the native bios of your hardware is using you should be using a uefi bios if you plan on running this tutorial which i probably should have mentioned at the beginning but it is at the beginning of the notes that you can download from the video description below if not go ahead and make sure your virtual machine is set up using a uefi bios by clicking on the q35 for machine type and ovmf uefi under the bios drop-down you'll also need to add an efi storage disk and for this we're just going to add that to our local repository if you're running a default windows 11 installer you'll also have to define a tpm storage directory however if you have a modified windows 11 installer that doesn't require tpm or are using windows 10 just uncheck the add tpm box with all that done go ahead and click on next the next page is where you're going to set up your hard drive capacity for your virtual machine now i'm using a one terabyte nvme drive for these four virtual machines so i'm going to give each one a 200 gigabyte max capacity with that set go ahead and click on next now again this system is using a 12 core 24 threaded cpu and since i'm going to be splitting this up four ways i'm going to give each system six cpu threads to work with and that goes under the cpu cores next up is the amount of system memory you want to assign to each virtual machine and i'm going to give each of these gaming machines 12 gigs so 12 000 244 under network leave it with the default intel e1000 adapter click on next and then click on finish you should now see a new virtual machine showing up under your proxmox server go ahead and double click that vm to open up a vnc window start up your virtual machine and install windows as you normally would once you have windows installed go ahead and shut it down that way we can attach our virtual graphics card and since i'm an overachiever i went ahead and installed four windows vms now that we have windows set up and our nvidia driver loaded we come to the very tricky part and that is how do we actually configure our nvidia virtual graphics card and what even is a virtual graphics card with a virtual gpu enabled card you're able to actually divide your graphics cards into equal chunks and present them to a virtual machine as if a graphics card was physically plugged in that is giving a vm direct hardware access the easiest way to explain this is let's say you have an eight gigabyte graphics card you can divide that card into eight equal chunks with one gigabyte of video memory each four two gigabyte chunks two four gigabyte chunks or one eight gigabyte chunk the amount of video memory and the features you have available from the graphics card to the virtual machine are determined by what nvidia vgpu profile you enable and these profiles are what we listed out before using the mdev ctl utility if we go over to nvidia's website you can see that there are a number of cards that support vgpu modes but pretty much all of them are either tesla or high-end quadro cards we have cards like the quadro rtx 8000 the tesla v100 the tesla p100 or the tesla m60 now if you note the tesla m40 that i have installed in this system right now is not on this list and that's where the vgpu unlock software comes into play it allows pretty much any architecturally compatible graphics card to use vgpu modes so that is any maxwell pascal turing or volta based gpu using the mdev ctl types utility you can view all of the available profiles for your particular graphics card in my case the tesla m40 is a maxwell based card and so we see the profiles for an m60 based gpu i've gone ahead and highlight the nvidia 18 profile which is a maxwell-based profile with two gigabytes of video memory you can also see that it has a frame rate limiter enabled of 60 frames per second and has a max resolution of 5k by 2880 and a max number of instances of four on a gpu you can only use a single profile type so you can't mix and match nvidia profile 18 with nvidia profile 22 to get different sized virtual machines using this nvidia 18 profile i could create up to four 2 gigabyte virtual gpu cards and assign each card to its own virtual machine each vm would have access to two gigabytes of physical memory and could use up to one hundred percent of the gpu's other resources such as nvinc encoding or cuda cores now unfortunately you can't mix and match profiles on a single gpu so in this case if i assign the nvidia 18 i have to use nvidia 18 for all gpu instances and this has a max number of instances of four so i can only create up to four two gigabyte cards the tesla m40s though have 24 gigabytes of memory so a lot of that memory would just be sitting there wasted that max number of instances has to do with the maximum available memory on supported nvidia graphics cards and since the tesla m60 and m6 are the only maxwell-based cards that support vgpu they are only 8 gigabytes so they figured they would never need to support anything more than 8 gigabytes with these particular profiles luckily the new vgpu unlock rs script allows us to modify these vgpu profiles and define all the parameters ourselves and that's what we're going to set up now first thing i need to do is create a new directory to hold all of my vgpu unlock profiles so i'm going to mkdir to create the directory forward slash etc forward slash vgpu unlock then we create a configuration file that holds all of our profile overrides so that is going to be in vgpu unlock slash profile override.toml now again you're probably just going to want to copy and paste my profile data from the instruction file down in the video description i've gone ahead and copied off my profile 18 data and will paste that into the new file now there's actually quite a bit to go over in just this very short configuration file starting all the way at the top we are going to modify profile nvidia 18. again the nvidia 18 profile is only for maxwell-based gpus which my tesla m40s are you may have a different vgpu profile that you need to modify if so you can follow this as a template but you will need to make sure to enter your own information the first line i've added here is the number of supported displays and i've set this at just one now by default this profile actually enabled up to four virtual displays but we're just using this as a gaming pc so one display will be just fine next up we have our supported resolution which is defined by width and height and i have set at 1920x1080 again the default resolution support is 5120x2880 so why would i lower that down from 5k to just support 1080p it's actually pretty simple if you plug in a 5k display to a pc it will try to default the resolution all the way up to 5k and sometimes games will see this default resolution and try to set their own resolutions to 5k which often leads to crashes so i actually recommend setting your maximum resolution in a profile to something like 1080p or 1440p or even down to 720p if you're aiming for lower resolution streaming the last line for the display is the max number of supported pixels and this one's actually pretty easy for you to figure out at home just take your width multiply it by your height and you get the maximum number of pixels next up is cuda acceleration on or off i went ahead and turned mine on by setting the value to 1. frl is a frame rate limiter that you can enable inside the nvidia driver itself and i would actually recommend turning this on as you're probably not aiming for super high frame rate gameplay on a remote machine from a desktop in the other room secondly by capping your gameplay to 60 frames per second you'll actually save resources on the pc and allow more people to play on the same graphics card up next is the frame buffer and this especially matters for older graphics cards like the tesla m40 or newer generation graphics cards that have more video memory than their consumer counterparts in the case of my tesla m40s they have 24 gigabytes of video memory each however maxwell cards are limited to distributing 8 gigabytes of video memory per card which leaves 16 gigabytes unutilized by modifying this value i can change the amount of physical memory that we are passing through to a virtual machine instead of the default 2 4 or 8 gigabytes available to maxwell cards in fact you can assign all 24 gigabytes of this card to a single profile if you really wanted to in my case i'm going to split the card pretty much down the middle and give each profile 12 gigabytes of vram to work with down in the video description and the instruction file that you should have been using to this point i have a full list of vram sizes that you can paste in and last but not least is the pci id now the reason we need to change the identifier of the vgpu is vgpus actually require licensing by nvidia the vgpu unlock script is allowing consumer or non-certified nvidia cards to utilize vgpu drivers now there's two pieces to the pci id equation there's the device id and the subsystem id if we look here at my configuration file the 17f0 is the device id and the one one a zero is the subsystem id when you look these up online these are normally presented following the vendor id and an nvidia's case that is one zero d e in here though we don't want the vendor id we want the device id and the subsystem id as one single line and i know it's a little bit confusing but don't worry again down in the instructions you can get in the video description i have a full list of maxwell pascal volta and turing based cards that you can get the device ids for just copy the values that match the architecture of the graphics card you're using and since i'm using the tesla m40 i have a maxwell-based card so i'm using the quadro m6000 once you have that pasted in we can exit and save so you've created your vgpu profile what are you gonna do next well let's attach the vgpu to a windows vm and play some games because that's kind of the whole point of this right as you can see i've set up four windows virtual machines right here and there's really only one thing you need to add to these vms to allow them to use virtual gpus and that is a unique identifier or uuid now unfortunately i haven't found a way to do this in the gui but getting to the configuration file is simple enough we're going to type in nano slash etc slash pve slash qemu dash server slash and then the vm id number so in this case 501.conf and this part could not be simpler first off we're going to create a new line of text and paste in argument uuid followed by a long string of numbers however it's only the last three that you need to worry about and all we're going to do is replace those three pound signs with the vmid number of our virtual machine or five zero one control x y and enter over in the web interface i'm gonna click on vm 501 i'm gonna go down to hardware and then add a new pci device under device i'm going to grab that pull down menu and scroll down to our nvidia tesla m40 then under mdev type i'm going to select nvidia profile 18 which you'll remember is the profile that we've modified i'm going to click on add and then i'm simply going to open up this vm and start it now over on the ssh side of things there's a really nifty command that you can use to view the process of a virtual machine and a vgpu we're going to say watch nvidia smi and if you look on gpu 0 it's using 11 gigs of ram right now why would that be well it's because i assigned a virtual gpu to a virtual machine and it's using around 11 gigabytes of ram per the modified profile all we have to do is wait for windows to boot up and then install our nvidia drivers once you have windows booted up you're going to want to go to nvidia.com drivers and download the quadro driver that corresponds with your gpu architecture yet again if you have a maxwell-based gpu like i do you want to download the quadro m6000 driver for pascal it's the quadro p6000 for volta it's the quadro gv100 and for touring it's the quadro rtx 6000 but i've got a quadro m6000 so we're gonna hit search now the latest version from nvidia is 511.79 and that is the driver version that i have been testing through this tutorial there has been an issue in the past with some vgpu unlocked tutorials where newer drivers will not work so keep that in mind if you're following this sometime in the future this is being tested with the 511.79 driver only and go ahead and download once it's downloaded i'm just going to run the installer as i normally would i'm going to click on custom installation click on next and we're going to install the graphics driver and might as well install the rdx desktop manager as it really doesn't alter anything click on next and hopefully the drivers will install and just like that we should have a working quadro m6000 of sorts with 12 gigabytes of video memory inside of our windows 10 vm i'm going to go to the device manager click on display adapters here is the built-in virtual machine graphics adapter and then our quadro m6000 success with the driver fully working let me go ahead and show you one oddity when working with vgpu systems inside of a virtual machine so i'm going to right click on the desktop and go down to display settings and you can see two different displays here although my window is only showing me one that is because we are currently looking at the qemu kvm virtual display we actually cannot see the quadro display from this vnc session so we need another way of accessing this machine where we can see both virtual displays now one option is just using windows remote desktop client but i went ahead and installed type vnc as a server so we can use a third-party vnc client so i'm just going to get this machine's ip address real quick 100-162 awesome so we'll go ahead and close out of that window and then using a type vnc client we're going to connect to 100.162. now through vnc you won't get the desktop background but as you can see i now have the 1024x768 virtual display which is the built-in vm graphics card as well as a new 1080p display over on the right side which is connected to the quadro graphics card and to make sure that the quadro is the primary display we're going to go down to display settings again and i'm going to disable display number one we scroll all the way to the bottom we find the multiple displays pull down and i'm gonna say show only on desktop number two keep the changes and now we are only running the quadro display now unfortunately just using a vnc client you're not able to play games with this because the frame rate is well let's just say atrocious like let's go ahead and launch heaven benchmark here just so you can see what i'm talking about so here we have heaven running you can see we're getting about 170 fps upwards of 220 fps but what's displaying through the client is one one frame per second so this is obviously not going to fly to connect to for gaming but there are a couple other clients that you can use let's close out of heaven there i'm going to install parsec for right now which is a very simple client that you can use and it is web connected so you can access this pc from anywhere but in the next video i'm going to go over a number of different options to stream games from this server to the client of your choice so make sure you're subscribed so you don't miss that one with parsec installed i'm going to go ahead and connect to that machine now and this is what the pc looks like when i connect through parsec you can see it's a much smoother experience everything is moving around pretty much exactly as it should with as little latency as possible now i can still tell it's streaming from a remote pc but it's good enough that i can probably play most games so let's go ahead and relaunch heaven benchmark and let's turn up the settings just a little bit there we go and run now i'm only going to be able to do this for a very short time because i don't have my giant fans hooked up to the m40s so you can see even just loading the game we're already at 65 degrees celsius and here we are inside heaven and you can see that is a much smoother experience now obviously again graphics card is already hitting 75. i'm only going to show this for a couple seconds but we're getting 100 110 fps down there's 160 fps overall that's a win but like i said in the next video we're gonna go over quite a few different streaming options for your game clients and how you can actually use this as a four gamer one cpu system for your entire family household or friends that live remotely but that's going to do it for me in today's video let me know if you have any questions down in the comment section below also in the video description i'm going to leave a link to the vgpu unlock discord server where you can talk to the community that is enthusiastic about this kind of thing as well as the original authors for these two vgpu unlock scripts on your way down there make sure to drop this video a like and subscribe to craft computing if you haven't done so already follow me on twitter craft computing for daily shenanigans like this and if you like the content you see on this channel and want to help support me in what i do consider joining the patreon link is down in the video description that's going to do it for me in this one thank you all so much for watching and as always i will see you in the next video cheers guys beer for today is from new image brewing it is the can you milk an oat oat cream ipa clocking in at seven and a half percent [Music] can i milk a note well i'm certainly gonna try the great thing about starting a youtube channel and making craft beer a staple of it is you get to have a craft beer every time you shoot a video the downside with hiring a cameraman is now it cost me twice as much that is a delightfully creamy yet still hazy ipa lots and lots of fruit in that very juicy let me see if i can pick some of those out definitely a lot of pineapple right up front i'm getting a lot of pineapple boy this is gonna sound weird but lemons and passion fruit towards the rear of it this almost reminds me of in some ways a shandy midway through of it it's got a real similar character to it where it's a nice refreshing like orange aid or lemonade or something like that with a pretty solid hazy ipa recommended sorry three-point game three minutes left yeah i'm watching basketball sue me there now i can see the score i don't have to crank my neck anymore

Channel: Craft Computing

Views: 138,155

Keywords:

Id: jTXPMcBqoi8

Length: 33min 30sec (2010 seconds)

Published: Mon Mar 21 2022