Eight Gaming PCs in a 1U Server - Cloud Gaming Server Part 16

while I am no stranger to building cloud gaming servers my server rack is starting to get just a little bit crowded so today I'm going to try to cram 8 gaming PCs worth of Hardware into this 1u box because putting it onto a for you box was just too easy [Music] Hey Jeffrey gonna be taking off for the day you need anything else oh hey Rhett before you go I need you to get on and order that craft computing.store domain for me Absolutely I'll hop on a pork bun they got everything we need you're not gonna forget like last time are you hey man what you working on oh hey bro I was just putting the finishing touches on my new logo for my blog craft Computing hey uh could you buy that domain for me man yeah man totally will but uh we're gonna be late for that board meeting oh right I I did not realize what time it was my man let's hit that first don't let your dream project go up in smoke because you didn't buy the domain this is pork bun today they carry over 500 domain extensions from AI to XYZ and your favorite Niche tlds like dot Quest and Dot computer the best part is their domains are priced to sell check out this domain I picked up for Just Two Dollars and 55 cents visit porkfun.com to get your website started today that's porkbun.com and again thanks to pork bun for sponsoring today's video you should grow your hair back out welcome back to craft Computing everyone as always I'm Jeff and yeah I'm up to my old tricks again although the server in this case will be fairly familiar this is the elemental 1u GPU server that I picked up just a couple of months ago in hopes of installing a bunch of AMD Radeon Instinct Cards into but as you might have found out that project didn't exactly go as planned but rather than let the hardware go to waste or just resell it I figured I might as well fill it with Nvidia cards and see if we can solve a couple of the problems that I've had with cloud gaming servers in the past namely in the storage bandwidth department so yes today we will be installing two Tesla M40 12 gigabyte cards as well as a Tesla M60 16 gigabyte dual GPU card but we are also going to load this thing out with solid state storage starting with a pair of 1.92 terabyte Seagate Iron Wolf Enterprise ssds this will be responsible for loading proxmox as well as the operating system disks for all all of my virtual machines but if you've been following the series long enough you know that game storage has been a major issue when trying to run 8 or 12 VMS at the same time so today we are finally throwing some nvme at the problem in my hand is a pair of Western Digital black two terabyte nvme gen 4x4 drives that we'll be installing into the server in hopes of getting game load times to a somewhat reasonable level and thanks to Micro Center for sending these over to me now we're not going to be able to take full advantage of the 8 000 megabyte per second read speeds of these gen 4x4 drives mainly because I only have PCI Express 3.0 in This Server however we will still be dealing with lower latency nand flash as well as better drive controllers inside of here so we should still see a market difference plus we're going to run these into ZFS raid one anyway and I'm hoping that makes all the difference now the elemental server does not have standard m.2 nvme slots so we're going to have to use one of the PCI Express slots to plug our drives in I picked up this card right here which is actually a four slot m.2 adapter to go to a single 16x slot now I would have filled all four Drive slots in this adapter but the only slot available in the rear of the server is an X8 so we can only use two drives instead of all four maybe in a future video I'll put four drives onto this card and see if we can truly eliminate our bottlenecks as for the rest of the hardware that we'll be using today here's a quick refresher this is an elemental 1u server that I picked up off eBay for 279 dollars inside is a super micro 1028 TR motherboard with dual 2011 V3 sockets I've got a pair of Intel Xeon 2898 V3 CPUs which are 16 core and 32 thread each running at 3.6 gigahertz max turbo alongside those is 256 gigabytes of ddr4 ECC registered memory running at 2133 Mega transfers per second and finally to keep everything fed with power we have a pair of 1100 Watt Platinum rated power supplies so with all that out of the way let's go ahead and get this thing together and get to some testing [Music] foreign [Music] foreign [Music] fired up I did quite a bit of work off camera on the software side of things to get it up and rolling I installed proxmox 7.1.7 onto the pair of SATA ssds enabled iommu in both the BIOS and the OS and set up the Nvidia vgpu unlock package using the rust script links to my video tutorial as well as written documentation and sources are all available down in the video description so that's the hypervisor sorted and next I moved on to the nvme storage for that I went ahead and set up a true Nas virtual machine and pass through the pair of Western Digital Black 2 terabyte nvme drives while I probably could have set up the ZFS raid 1 pool in a standalone array in proxmox I prefer the ease of setting up the iSCSI shares inside true Nas if you're interested in how I set up the iSCSI disks I have a link to that video down in the video description as well essentially each virtual gaming machine I plan on running gets its own Z Vol on the nvme pool and in this case I made them all one terabyte in size now you might ask how I plan on having eight one terabyte zval allocations with only two terabyte of storage actually usable on the pool well as the only things that are going to be installed on these drives are games I enable deduplication for the entire pool which will detect identical storage blocks on the array and only keep a single copy with differential data for other instances now this doesn't mean that every virtual machine will have all the same games installed and all the same data even though they're all connected to the same ZFS pool each VM will have a one terabyte disk attached through iSCSI and each VM will need to install a copy of the games that they want however if a second VM installs the same game the drive pool will only need to keep a single copy of that data with differential data for any changes made to each Z Vol like other complicated installs from earlier in this video I also have an entire video on deduplication inside true Nas link is in the usual place and finally we get to the virtual gaming machines themselves between the three gpus I have in the server I'm wanting to run eight gaming virtual machines and I've set them all up with six virtual CPU cores 16 gigabytes of memory and an 80 gigabyte boot drive on the SATA SSD pool now technically speaking I actually have four gpus in This Server the two Tesla m40s with 12 gigabytes of video memory each and the Tesla M60 is actually a dual GPU card and each GPU shows up individually with 8 gigabytes of video memory I'll be splitting each GPU into two partitions and dedicating each partition to a single VM that means a VM assigned to a Tesla M40 will have six gigabytes of vram and a VM with an M60 partition will have 4 gigs unlike GPU memory GPU compute power isn't a hard partition and is actually dynamically shared with the attached virtual machines as each GPU is serving two VMS if one machine is sitting idle the other can take full advantage of the attached GPU that's 3072 Cuda cores and 250 watts of TDP for each of the Tesla m40s and 2004 48 Cuda cores and 150 watts TDP for the Tesla m60s load is automatically balanced between the gpus when multiple virtual machines are accessing compute power but the focus of today's video is going to be on the storage solution and can I actually run eight gaming machines off of just two nvme drives I have no idea so let's give that a shot now alright jumping on over into the proxmox server you can see all of my windows VMS are already up and running now in total we are using 236 gigabytes of the 256 gigabytes that I installed in This Server we're also allocating essentially up to about 60 of the 64 threads although the threads will dynamically shift as far as power requirements go but each of these eight virtual machines is set up with 16 gigabytes of memory six CPU cores that is essentially six threads and a vgpu instance we've also gone ahead and installed the vert I O Network driver so we can get greater than 10 gigabit speed locally inside of This Server moving down to the true now server I've gone ahead and set that up with 32 gigabytes of memory and 12 virtual cores we've also passed through the Western Digital black two terabyte drives with PCI Express pass through jumping on into that true Nas server and clicking on storage you can see I've already set up the two drives into a raid Z1 and then created eight individual z-volves for each of my VMS now each of these Z Vols is attached to a virtual machine through iSCSI and again the tutorial for linking a games library with iSCSI is down in the description I've gone ahead and installed crisis onto seven out of the eight drives that you can see here and the install is about 7.59 gigabytes now some simple math says that 7.59 gigabytes times seven instances of Crisis installed gives us right around 54 gigabytes of used space and that is what true Nas is reporting however that's not entirely accurate now there's no GUI interface that will tell you how well deduplication is working or how effective it is but we can jump into the system shell and get an idea so I'm going to type in Z Pool list and we'll see our Elemental games pool and over here on the far side you can see the dedupe amount as well as the actual allocated space so right now we're using only 11 gigabytes not 54 gigabytes and deduplication is 5.19 X accurate that is it is actually taking five times less space than true NASA's actually reporting so it's not completely linear or entirely perfect but installing eight instances of Crisis onto eight different seed balls does not give us eight different copies of Crisis what we're left with is slightly over double the amount of usable space that we would normally be taking up by a single install but let's go ahead and take a look at those virtual machines in action and for that I'm going to use Moonlight connected to a sunshine service that I've installed on all of these virtual machines so let's start with machine number one so that we'll start with machine one and we'll just put that instance right up there and really quickly I'll try to get the rest of them them loaded up so there are the eight virtual machines pulled up inside of Moonlight and if I pull up my SSH session you can see we've got eight virtual gpus assigned to those eight VMS now it might be a little bit difficult to read so I will have ret zoom in in post but here are the actual vgpu instances themselves as you can see we've got four six gigabyte instances and four four gigabyte instances each of those attached to the gpus listed up here so on top of the 235 Gigabytes of system memory that we've assigned in this machine we're also using 40 gigabytes of video memory as well now as I mentioned I installed crisis on seven out of the eight machines already and I'm gonna go and get machine number eight set up right now so let's go ahead and bring that one on full screen and we'll click on the crisis installer checking out the drives attached to this VM again we have our 80 gigabyte boot volume as well as the one terabyte Z Vol attached through iSCSI and if I open that up you can see there's nothing installed here despite crisis being installed on the other seven VMS on this same Z Pool but we'll go ahead and get crisis installed right now so key colon backslash yes I agree to the EULA and install all right so there is Crisis installed and again we've added right around 7.6 gigabytes or in Windows terms 8.1 gigabytes of space to that Z Vol but let's go ahead and jump on back over to True Nas and let's see what was actually added so if I go down into storage you can see that that number is risen to almost 61 or 62 gigabytes but if we go into the shell again Z Pool list uh we are now using 1.1 gigabytes so that additional insole of Crisis only took 100 additional megabytes out of our Z Pool pretty cool stuff but now for the moment I've all been waiting for can I actually load all eight instances at the same time now I know crisis is not the most demanding title as far as loading assets into memory but loading eight simultaneous random seek reads off of an nvme that's going to be something to watch so I'm going to try to click on these all as quickly as possible all right here we go [Music] all right there's all eight all right so we have all eight instances set up at 1080p High settings and I've already got some pretty interesting numbers not necessarily from the storage it actually didn't take much at all to load the games just to the main menu we'll check on that in just a minute but as far as the resources required to do what you see right here so let me go ahead and open proxmox back up right now we'll go to the summary screen as you can see just to load the crisis menu and stay on the menu screen we're using about 40 to 41 percent of our overall CPU power these are dual 135 watt 16 core CPUs we're using 40 of our available power uh going down to network usage you can see the load is only about 15 megabit per second right now it's it's hardly anything and that's just to keep these assets loaded so this was the game loading and sitting here on the menu screen the gpus are also pretty interesting because they are doing the stream rendering as well as trying to render crisis which even on the menu is a 3D rendering as you can see we're sitting right about 75 to 85 percent utilization on every single card uh and right around 100 watts each although interestingly enough the second Tesla M40 is using about 10 watts more than the first Tesla M40 but temperatures so far are green across the board we're seeing only 45 degrees Celsius on both of the Teslas and right around 55 to 60 degrees on the Tesla m60s and without any further Ado let's go ahead and get this thing started single player new game start so the first four VMS are using the Tesla m40s the second four VMS are using the Tesla m60s all right so we have every single VM loading the game right now that's actually not a terrible wait time uh it's slightly longer than I would expect with an SSD but as you can see they're coming up pretty quick see how closely these sync up [Music] see if I can get to the jump a couple of them are already free Fallen all right there we go starting to hit 100 utilization on the gpus and there we go we've hit 100 on all of them it looks like uh so far so good and as you can see these are all fully playable at 60 frames per second I can click on any one of them and take control that's too much fun each with one of these guys I'm controlling the top Center one right now I don't know if I stay in the water is he gonna die I've never tried staying in the water in crisis before but right now I have like six of them that want to drown and actually our gpus have settled down a little bit sitting right around 78 to 85 percent somewhere right in there uh I do have the frame rate Limited in all of these instances to 60 FPS so not reaching 100 is actually not that bad of a thing uh because we're only going to stream 60fps anyway so why waste CPU cycles and power and heat and wear and tear on your system if you're not going to be able to take advantage of it now Moonlight can stream up to 120 FPS and even Beyond with some unsupported modes but oops one of my guys died apparently you can drown if you stay in the water too long but so far I mean again this is very early results um I'm not seeing any stuttering that I can't just attribute to network if I grab any of these VMS you can see just how incredibly smooth they are there's no stuttering there's no hiccups foreign that's all pretty incredible and what's more is even with all of them loading at the exact same time it wasn't really noticeable versus a dedicated VM with a standalone SSD so far I'm gonna call the nvme drives a win although I'm gonna need to figure out how to do actual load testing or performance testing in real time on these machines so I can actually get some highs and lows without just synthetic testing because I don't think synthetic testing is going to tell me actually what I need to know here but as far as the answer can this run crisis what's the plural of Crisis Christy crises well what that's crisis is in like the the verb what you know there's a crisis happening or noun I guess it would be well crises is like multiple crisises okay but but does that apply to the proper noun of Crisis which is c-r-y-s-i-s maybe it's like uh you know like moose and moose I buy that Yes Virginia you can run eight crises now there's a couple more interesting notes about this GPU server first and foremost are the fans that are used to keep all of this cool if you notice the gpus and the CPUs don't have fans this is a 100 passively cooled server as far as the heatsinks go but that's not to say there aren't fans inside the server each of those fans is a little 40 millimeter Delta fan and I did some quick math and running out 100 those fans will draw 18 watts each which means times 10 fans in the server we're using up to about 180 Watts just with the fans that's a GTX 1070 running at a pretty good tilt we're using it to cool cards that are running crisis like they're guns and I think we'll call this a day and call this project overall success even though I wound up not using the AMD Instinct cards I wanted to in the first place anyway if you're interested in buying any of this equipment that I use in the video today I bought all of it off eBay or at microcenter.com Micro Center is a place where you can get all of your PC parts Under One Roof or in my case if you don't live by one order online at microcenter.com for a limited time new customers can receive 25 off the purchase of any Intel or AMD CPU new customers only and make sure to get your coupon code down in the video description and a huge shout out to Microcenter for sending over some of the parts in today's build if you like this video make sure to hit that thumbs up button or subscribe to craft Computing if you haven't done so already follow me on Twitter at craft Computing for daily Shenanigans like this and if you like the content you see on this channel I want to help support me in what I do consider joining the patreon link is also down in the video description that's gonna do it for me in this one thank you all so much for watching and as always I will see you in the next video cheers guys it's an empty glass beer for today is from one of my patreon mods claw over in Norway this is the nugna Imperial Stout strong dark ale clocking in at nine percent and yes this was shipped all the way from Norway cram the rest of it in this glass or do I share the last two ounces of red thank you it's the constant Battle of morality that I have [Music] you're welcome don't let me tell you he's a bad boss guys makes all the abuse kind of worth it doesn't it it's way more mild for how thick and chewy it is uh and there's [Music] it's a stout but like a lot of the beers that I've gotten from Norway especially the the bigger more robust ones the flavors are completely unusual for what I normally get so this one's going to take a little bit of uh a little bit of work before I can suss everything out of it let's do this first so like I mentioned this is definitely an interesting Stout uh and it's not that it's bad it's just completely different than most other Stouts that I would get here um it's thick it's chewy but there's not really a lot of coffee there's not a lot of roast there's not chocolate I would say the most predominant flavor is almost like a fig um or hazelnuts or something like that uh it's a little bit more on the Nutty side and The Malt is gosh how do I explain the malt in this it's well-rounded but with flavors you're not going to expect out of an imperial stout okay actually there is a little bit of chocolate there's a little bit of chocolate they're right on the very back end and that's really what lingers good beer well-rounded but uh something a little bit unpredictable [Music]

Channel: Craft Computing

Views: 142,441

Keywords:

Id: pIdCV1H1_88

Length: 24min 30sec (1470 seconds)

Published: Fri Sep 23 2022