The Holy Grail - Proxmox video passthru.

well greetings people of the internet hunky joe here hunky joe's playhouse did you miss me i'm back [Music] that's right i'm back and first of all let me apologize for not putting a video out last week i just i was busy look i'm a one-man band here and uh i've had a couple of clients uh decide to spend uh end of the year money at the same time so i've been busy trying to get these two major uh clients of mine get their networks expanded and frankly i've been doing a lot of i've been learning that's what i've been doing i've been teaching myself some new skill sets while i'm trying to run a business and trying to run a home and you know all the other stuff you know life but here i am now and what we're going to talk about today is for me so it has been for me the holy grail of of computer building and server making and that kind of thing has always been to build a machine that can host virtual machines that could have pass-through video i don't know why i'm obsessed about it but i have been for a long time it's been like something i've been trying to do but i never i never felt comfortable doing it myself i thought it was way out of my pay grade and um instead i decided to buckle down and learn out learn how to do it so that's what this video is about today how i taught myself how to do this and now hopefully what i've learned i can teach to you and you can do it as well before we get into this video i i i can hear people out there screaming well you could do it without raid you could do it with a bunch you could do it with vm where you can do it it's the same old song and dance and you and i have done this before i hell i got multiple video cards working on that x 3650 model 3. i never had what i would call stability though i i was never quite comfortable with the way it worked um an unraid man oh man i love unraid i really do but if you've ever dealt with virtual machines under unraid at least in previous versions of unraid it's always hit and miss when you go to edit a virtual machine and unless you're will very well skilled and unread and know all the ins and outs of it uh it was uh it was i tried it it just didn't work out for me uh but i did come back to proxmox and man i i tell you i've done another video on proxmox i'm going to keep doing videos on proxmox i really believe in the product it's an excellent hyperbat a hypervisor it it dots all the eyes and crosses all the t's as far as i'm concerned for a hypervisor and as we as i learn more about it and i do more videos about it i'll explain i'll teach that to you as well so today it's all about getting a machine up and running that can run proxmox that i can put a couple of video cards into and pass them through to virtual machines so i have the motherboard that can do it it's an older gigabyte motherboard but i and i have an intel chip that i'm going to be putting in here we're going to start out with an i5 3470 and we'll see what happens um and the reason i'm doing that is because that i5 chip also has its own video built into it so i can have the onboard video that the benefits of having onboard video plus this motherboard has two pci express slots in it one is at 16 ones at four then i could put some video cards into a new pass through and pass them through to windows 10 virtual machines and then give those virtual machines to people and actually let them test them and see how they work and see how the performance is so whoa that was cotton picking mouthful let's get the video started right now now if there's ever been a more prominent machine on this channel it's this one or i should say motherboard i just cannot for the life of me get rid of this gigabyte motherboard i have two of these now what i was doing is digging for a cpu cooler ram that kind of thing so this is that gigabyte gaz 7ds380 i believe if memory serves me and that thing just supports a wide variety of processors now what i was doing here is removing the extra special and expensive cooler i got you'll notice by the brown fan i'm not going to mention their name if you know who they are so i had to rob that cooler and put it on this motherboard with the i had an i5 3470 is the cpu that's in there and so i'm taking everything out of here we're gonna get uh we've already got the old cpu fan off i put the uh i5 processor away in a safe location back when i retired it and uh man hold on to old motherboards that's all i can say and then i was attempting to put the uh cpu cooler back on and then we'll get it installed back into the tower case now the the tower case i have for this thing is just a it's a really thin metal case actually i think it's a gigabyte case don't ever buy gigabyte cases they're cheap well the ones i've had dealings with are cheap they're not cut your fingers cheap but they're they do have rolled edges but the material is very thinsy flimsy and easy to bend so i'm going to get board standoffs and all kinds of stuff to get this board mounted back in here and i did i did a lot of work to get this thing going now originally i was going to use a spinning hard drive to install proxmox but then i came across a an ssd drive this is like a 750 watt power supply i'm putting in here it's totally overkill but i've got it on hand i'm going to use it i'm not going to buy i'm not going to buy a new power supply for this build that's the beauty of of these home labs is you can do them on the cheap so but i am a firm believer in a clean layout for power and motherboard and it's just so much easier if you take the time do a little bit of cable management it's going to make your life a whole lot easier so i'm getting everything plugged in all the front panel connectors getting out my viewing glass there so i can see and uh yeah getting everything ready to fire up for the first boot because it's been a while since i i had this motherboard up and running running [Music] uh i did find four sticks of uh four gig rams so i was able to give this thing 16 gigabytes of ram which is fine for what i'm going to use it for because we're basically this machine is just a proof of concept that i am capable of uh doing video pass through with video cards on proxmox but you just notice i just took the video cards out i booted it up and took the video cards out now why do you suppose that is well i had a dream and actually i had an epiphany i had everything up and running with the i5 3470 but frankly folks is that cpu has no hyper threading so i only had four uh cores to use and it just current you know just could not cut the mustard so i remembered i had purchased an i7 cpu six eight months ago from china and i decided i'm not using that home theater pc anymore we went another route so i figured hey it's it's actually a four core four thread cpu so let's put it to good use and so that's what i did i put it inside of this unit here i was trying to get the fan mounted again without having to unhook the motherboard connectors because it's such a pain to hook them back up isn't it so anyway yeah i finally just put my head up back on and took it removed it from my rear end and then i was able to figure out how to get the cpu cooler back on there and then the next step of course is to reassemble everything and then start from scratch because i'm going to tell you it's just good habit if you're changing out a cpu go ahead and reload the base operating system if you can afford the time because man oh man you're going to be a lot happier with the end result then uh then you're going to be trying to change your tail to find some problem that you may have created by swapping out the cpu now you notice there right there i took the board back at i i felt on that board i was like it's missing a standoff and it feels weird so when in doubt remove the board and make sure you don't have a standoff laying on the base for the motherboard take the time this motherboard is not replaceable it's almost impossible to find these motherboards on the ebays so there i was booting it back up with the new cpu and it saw it and then i'm gonna put a uh ssd inside of this thing we're gonna have an ssd to boot from uh it's only 128 gig that was all i had on hand and then i'm putting my video cards back in um i put the amd rx 560 in the first pci express slot and then even though this is not the k620 it's a that's a gtx card i did end up putting the k620 in here so there you go and now let's uh get everything working eh and here we are so we've renamed this node or i've named this node i7-3770 that way i can keep track of what machine i'm running on and i'm glad i went ahead and swapped out the cpu in here i had other plans for this i7 i purchased this about i don't know six months nine months ago on ebay the idea was the cpu was going to go into a home theater pc for the bedroom and then i thought better of it as time went on anyway long story short too late it is now in this gigabyte uh motherboard that i had the i5 3470 s processor in and the reason i went with the i7 is specifically because it's got four cores four threads so uh basically an 8x cpu um that i can then put over to virtual machines because when i did this with the i5 i wasn't even though i've got ssd storage in here and plenty of ram i wasn't real pleased with the performance so we're going to try this i7 now granted it's a it's a gen 3 i7 but uh hey you got what you got so what we're going to do now is uh we're going to get started with making the changes we need to for example i've got a couple of virtual machines created here but if i come out here to hardware and i try to add a pci device you'll see i'm getting an error message that no i o mmu is detected please activate it now a couple of things you need to be aware of one is you're going to need to go into the bios of your computer whether it be intel or amd and make sure that iomu io mmu is enabled and under intel that's vtx or is it vtd technology in addition to allowing it to uh run you know uh virtualization on there there's another setting as well um but anyway you'll need to go look that up for the individual board and bios you're using because every one of them is different and then you you'll need to come in and we'll need to edit some system files on proxmox so if that thing is if the thought of that is scaring you you may not you know you may not want to do this so um i'm going to go into putty though so let me bring putty up we know our ip is 5.10 here so let's go bring up putty and we'll create a session let's see 5.10 i said let's open it okay i'll accept it now then we need to log in but we need to uh we need to change some settings i'm going to change that font to lucidia console 12 all right we'll go ahead and log in as root all right i'm going to go ahead and clear the screen sorry i can't make it bigger for the life of me i can't get my brain to work today so the first thing we're going to do is we need to tell the bootloader that we have we want to turn on iommu so we're going to edit grub so in order to do that we need to do nano now i'll include links to the directions on how to do this after i get the video posted so etsy default forward slash grub so i'm going to use nano to edit grub so right now what i need to do is come down here to the grub command line and i need to add another command here so after quiet i'm going to put a space and i'm going to type in intel underscore io mmu equal on then the other thing i'm going to do is tell it io mmu equal pt that is a command i oh and i only i didn't mean to put those quotation marks you only want close quotation marks at the beginning of quiet and at the end of pt the iomu equal pt is a i'm hedging my bet that i don't have any trouble later down the line with putting um devices in iomu groups so that's what the pt command is so so i'm going to do control x and i'm going to go ahead and tell it yes to save it and we're going to write out to grub now the next thing i want to do is update grub so that gets written into the boot code okay and then i think what we'll do is go ahead and reboot yeah we'll go ahead and reboot this as well let's let's go ahead and do our other commands as well we might as well right uh yeah let's let's do our other commands so i'm gonna edit um i'm gonna nano into etsy um and we're gonna go into the modules file and we're gonna we're gonna add some modules we're gonna add uh vfio we're gonna add vfio underscore io mmu underscore type 1 vf io underscore pci and vf i o underscore v i r v i r q f d and those are the uh control x to save it yes and those are the modules we wanted to load now there's another command in here a couple of commands and they're used for interrupt remapping now you have a choice at this point you could try using those or you could leave those commands out and then come back if we have any problems later which is what i think i'm going to do now the next thing i want to do is i want to blacklist the drivers so that um uh there's no chance proxmox is going to use them but just to let it know hey you're not allowed to use these these uh modular these drivers so um and basically i'm just cutting and pasting or copying and then pasting these commands in here that i've written down and keep in mind i'm doing this on my kind of my own notes that i've taken i've done this so many times now that i've just got a set of notes that tend to work for me now i'm going to go ahead now that i've made these changes and i'm going to uh reboot proxmox and then we'll see if we've got our ability to uh have the iomu groups so we'll give this a minute all right so now we're back at the screen and uh it has survived a reboot so let's go out under the uh virtual machine and see if we can add oh lucky there are iom u groups error messages gone and i should be able to come down here and see i've got a choice of either a radiant rx 550 5560 or a nvidia gtx 650 all tucked away in their own iomu groups and that is a good thing because we're going to need them later when we go to assign a network or a video card to a virtual machine so uh let me show you those steps next now one of the things i have done uh previously with my proxmox adventures and if i haven't covered it uh before this video i'll cover it in a video coming up but i created a template over here called windows 10 ufe uefi so what i've done is create a virtual machine with all the settings i want in it i want it uh four cores i want a uefi bios 4 gig of ram it's a q35 6.1 you got a yada yada so what i did was i created this machine then i sys prepped it and then i created uh a template out of it which is what they call them under proxmox and then all i did was right click and then choose to clone this to a new machine so that's what i've done in the case of this windows 10 machine here and this windows 10 machine right here so one of these windows 10 machines has an rx 560 video card pass through to it it's working and i was successful and i'm going to show you how i did that same thing with the windows 10 with a nvidia k 620 quadro card pass through it's working it's working fine and right now i have both machines busy doing various things that's that's why you can see the cpu and ram and network traffic and whatnot so some of the commands i showed you a minute ago the ones that you need to be aware of uh in order to enable iomu you need to follow those um you also need to set uh the intel underscore io mmu to pt we covered that as well so now what i'm going to show you is what you need to do to make sure that this is all going to work for you i'm going to go to putty because you can do this through the proxmox interface as well by coming here and going to the shell but i'm just used to going out to it with uh putty so go ahead and put my ip address in here click on open and we'll log in as root and we'll give it a password all right now let me uh stretch the window here a bit and change the font all right there we go so now what you're going to want to do is there is a specific file that you're going to want to create and edit so the instructions for creating uh what we're trying to do here is isolate the two video cards that i have from proxmox and basically tell proxmox hey don't use these under any circumstances they're made to pass through to virtual machines so that being said we need to create a file to tell proxmox when it boots up to isolate those cards if that makes sense so there's a couple of ways to do this you can enter a couple of echo commands and it'll write the file for you or you can just go right out and you can edit the file now i'll put links down in the uh notes of the show that link you to this article that i one of many articles i followed but basically this article is really good at explaining to you what each one of these commands are doing now we've already gone through and configured grub now some of the things i did not do were disabling the frame buffer and using acs override for iommu groups i found that it wasn't necessary to use those in my setup now this is where the key is each one of your setups are going to be different i don't know what hardware video cards cpus you have what motherboard so you're going to need to do a little research on on your on this uh on this install on how to do this um the other thing i did was added the vfio modules to the modules um and now what we need to do um is if you need iommu interrupt mapping i did not need this you can put that in here uh i strongly recommend you blacklist your drivers the the radeon the nuvo and the nvidia so that uh those won't run and then you'll need to add your gpu to the vfio so this tells proxmox to exclude using those video adapters uh for anything so that's what this command is for now i have two video cards so you'll need to run this lspci minus v and pay particular attention to the ids you get assigned to your video cards for example one of my video cards was zero one colon zero zero the other one was zero one six colon zero zero then you need to take those numbers that you get the one and the six so you need to plug them into lspci to get the vendor ids you need to make a note of those vendor ids because what you're going to be doing is your if it were a single video card i would tell you just use this command here this echo command and that goes out and creates a file called vfio.conf under the modprobe d etsy directory i'm going to do that a little bit differently i'm going to show you how i did mine so if we bring up putty we'll go into nano space forward slash etsy and nano is just simply a text editor for um for uh linux so we're gonna go to the modprobe.d directory and then we're going to load uh vf io dot conf so what that you see my my line is a little bit differently that's this is basically what the echo command did right but since i have more than one video card i wanted to just edit the file myself now don't do like i do and think that each line is an individual line it's not you want to combine them so i've got my first video card the ids for that are right here um then they're separated by a comma this is the video card and this is the audio portion of that and then this is for my secondary video card and it's sound properties or sound card built into that for hdmi audio and then i've got disable underscore equal vga equal one so that's how easy it is to do that it's very important if you do not have this command in here and you don't have these addresses proper properly entered it's not going to work for you you're going to get all kinds of weird errors so that goes beyond the scope of this video but basically this is one of the very important commands that you need to to do to make sure you have done or it's not going to work right and then you need to run the update in it ram file system dash u and then you need to reboot the machine your proxbox machine um once you've done that then you can come down here if you haven't created a virtual machine and follow these directions on configuring the vm what i'm going to show you next is how to configure a vm that you already have created so let's go look at that right now now it's very important you do a couple of things first of all you want to name whatever virtual machine you're going to do the pass-through on you want to make sure you give that machine a name and it's got either a hard-coded ip address that you're aware of or you've got your guest agent in there so you can see what your ip address is like i do now currently and the other thing you want to do is enable remote desktop because vnc is no longer going to work as you can see i've got failed to connect to server when i try to go to the console you'll need to do this before you flip the video over otherwise you're not going to have any video at all so keep that in mind now this machine is running so i'm not going to change it but i am going to show you the changes i made so if you come under hardware a couple of things under the processors i've gone in here and i've turned on i've changed the type to host and i've turned on pcid i've given out a plus sign and what that does is these are some flags that tell it how to treat the cpu how to pass the cpu through to the virtual machine because you want the machine to think it's not running virtually you want it to think it's running uh you know in a non-virtual environment i can just help later with getting your video card to work right you've got to make sure and these instructions are in that article that i'm going to post a link to make sure you've got ov ovmf or uefi bios enabled and what i do here is simply pass through my pci device so i come here to add i go to pci device i choose my pci device which is number one here and it'll grab the audio portion then automatically and it'll put it in there so if we go over to that device you can see i've checked all these boxes all functions rombar pci express primary gpu and advanced and then click ok now after you've done that then you will come up here to summary and you will actually power on your virtual machine and bring it up now you'll know it's working because what'll happen is while you're powering that machine in fact let me just show you i'm going to turn this machine off for a second and you'll notice that this guest agent information will disappear after the machine shuts down it'll say guest agent not running so now you'll see the machine is stopped and nothing's on so if we come down here again to hardware same way with this one except i use card number six i've given it all functions rom bar pci express primary gpu but if i were to come here to the console and power the unit on i'm not going to get anything because it's got a pass-through video card so it's just going to tell me failed to connect to server but if i come up here to summary and i monitor this area right here where it says ips once that comes up and gives me an ip address then that'll tell me that i'm able to then log into the virtual machine so we'll wait for that to come up now if you have problems with this part go out and check that command we entered make sure you've got the right uh hardware ids in there if you don't none of this is going to work properly but now we have our our windows 10 k620 machine up and running again that's good let me see if i can log into it you're not gonna see that part but that's okay and there you go i'm back in it's up and running so i'm gonna put this back on summary air perfect okay so it's back up and running so again make sure you've got that command this one here entered correctly like i had shown you earlier we can go back into that just take a look those are the ids for for both of my video cards and it's working well but yeah make sure you've got those those commands in there correctly or none of this is going to work so right now this machine this is the one with the rx 560 video card you can see it is passing through the card you can see the temperature of the card cpu is running we're currently running folding at home on this unit just so i could give it something to do while i was testing it if we come up to the k620 and do the same thing here you will see that we have the k620 uh the cpu not doing much right now but um it is it is up and running and the video card is being passed through and we actually have the uh the software loaded for nvidia and for um for the rx 560 the other one so let's just go ahead and bring up heaven here i'm not going to do it full screen that way you can see it running in the background all right so if i just minimize you see we're getting about 20 to 23 frames with this older video card but it is uh it is using the video card and cpu it is firing it up so if we were just let this run believe we even have sound in this yeah we even have sounds that's a good thing and there it is running it does much better on the uh rx 560 but this one uh it it keeps the pace pretty well so here we are back at the uh proxmox screen and you can see that even though i've got uh four cpus assigned to each unit and four gig of ram assigned to each one and a video card we've still got a little overhead left this is why i switched i originally started this with the i5 3470 and it was just it's only a four core cpu it was just way too underpowered to handle this load so i remembered that about six months ago i ordered a a used i7 was it 3770 off of ebay and uh that has four cores four threads so basically an eight core product or an eight cpu processor i put that in here and it's doing much better because before it was uh even though the virtual machines were getting good frame rates it was a little laggy so it is up and running let me tell you i have been through the mill with this thing it took a lot of hard work to get this to work because i had to wrap my mind around how uh proxmox isolates equipment and what all those it's one thing to sit here and enter commands at a linux prompt it's another thing to know what they do what effect they have and what they changed and i wanted to make sure that i wrapped my mind around what they were changing in proxmox to get all this to work because as you know this has been like the holy grail of computing for me for quite a while now is to get video cards passed through uh to the uh underlying operating systems so as a test what we're gonna do is i'm gonna hand both of these virtual machines over to uh to j uh jamie who does some video editing for me and let him have both units and try i think he uses vegas video editing let him go ahead and use these virtual machines and see what kind of performance he can get out of them and if they're stable these have been up and running now uh well it's the machine's been up running 4 hours 49 minutes i shut it off over overnight but it ran all day yesterday just to make sure and i ran these uh folding at home in a couple of video tests just to make sure everything was working okay but like i said i'll hand these over to jamie as a virtual machine and let him have adam and see if he can crash him or you know consequently get him to work so i wanted to share some of the final scores with you um i did uh some unigine benchmarks on these virtual machines once we got the video cards installed and the proper drivers so let's start with the k620 um this is on a now all the virtual machines were configured the same way they had four cores four gig of ram that kind of thing so for the um uh quadro k 620 our score was 24.1 so it didn't do too bad its final score was 6 0 6. and let's go look at the rx 570 and you'll see that it it did pretty well i got a frames per second of 51.0 a final score of 1284 and as you can see it's uh the radian rx 560 series so yeah and then i did some other benchmarks let's go look at tropics uh did really well with tropics on the k620 and actually got 50 frames per second the rx 560 got a whopping 122. so an rx 60 uh 560 running the valley benchmark did almost 50 frames per second or a score of 2085. um let's see k620 heaven yeah we already did that one okay and then i ran sanctuary and it did very well that was on the k620 got an impressive 72.3 and on and on it goes uh superposition with the k620 got a 1507 and on the rx 560 together 24 44. so you see the scores were all over the place but i just wanted to share those with you so there you go now i was actually i am actually quite impressed with the performance i'm getting on these virtual machines um i left i left heaven benchmark running overnight while both vms were up and running each one running its own copy of heaven i never this board's been rock solid ever since i got it but even after i changed out let me explain something to you here that i did and i alluded to this in the video when i was talking about it i didn't want to have any problems when i went from the i5 to the i7 processor and i know that proxmox is a little picky about equipment sometimes i've noticed some glitches in the matrix so to speak that's the best way i can describe it and the guys in my discord channel support me on this i've loaded proxmox the same way two times that had different results before so i don't know if that's a matter of version seven not being completely up to snuff yet or it's just some combination of hardware that i've created this uh mess with um but i i made sure when i put the i7 processor in here i started from scratch and that included the virtual machines so what i do is i create a template of a virtual machine or i create a windows 10 virtual machine i sys prep it then i create a template out of it and then i use those templates to create my new vms from and i did notice at one point when i was doing all this stuff with proxmox that i had failed i had made so many changes to the virtual machine to where it finally started throwing up these errors every time it came up evidently at some point i booted i i screwed up the boot sector of the virtual hard drive or something because it would no longer boot and i ended up having to create a new virtual machine from scratch now if that's what you're if you're going to try pass-through that's what i would recommend get to as clean as slate as you can i have spent a good part of the last three weeks learning how to do all this and uh i've learned it from other people so so if i can do it you certainly can do it um and it just gives me more confidence about proxmox but there are those little glitches that i wanted you to be aware of that i ran into one with with proxmox itself and the other one with uh i was able to easily corrupt a virtual machine and i i shouldn't have been able to do that and i yeah did i have a snapshot yeah i had a backup image but even after i restored the backup image and tried to do pass-through because of some combination of whatever comedy of errors it just wouldn't work but it's working now so there you go uh another proxmox video done i hope you uh feel it was worth the the weight and this uh this machine will become a semi-permanent part of monkey joe's playhouse because this will be the machine that jamie edits on uh when he comes in and does video editing he's gonna do some testing on that in the next few days uh once i get him some footage so uh it was a lot of fun doing it um yeah i really enjoyed this uh with the proxmox and i'll make sure i put the i'll put the link to the art i use several articles i jumped around and you know take those articles with a grain of salt because each configuration and combination of hardware could be a little bit different but there you go we hope you found the video entertaining and informative as always please smash that like button if you enjoyed the video leave your comments down in the comments section let's subscribe if you're not already a subscriber and click that notification bell to be notified of new videos when they come out thanks again for coming to see us donate if you're so inclined we accept paypal patreon the youtube join function i want to thank each and every one of you for coming to see me i appreciate it and please don't forget we'll see all of you on the other side [Music] you

Channel: Unkyjoe's Playhouse

Views: 8,662

Keywords: proxmox, vm, linux, gpu passthrough, proxmox video card passthrough

Id: Hj-UFlygUoM

Length: 44min 10sec (2650 seconds)

Published: Tue Mar 22 2022