My Perfect Media Server isn't so perfect anymore...

this is Morpheus my perfect media server in the front bunch of hard drives hot swappable front loading at the back an SFP plus networking port for 10 gig connectivity inside is an i5 8500 CPU with 64 gigs of non ECC memory workload split across containers virtualization storage split across ZFS and merger FS for all of my media server needs however my needs are changing and growing I am starting to do a lot more virtualization stuff for work uh I also want to have more redundant copies of my app data and all sorts of fun stuff and the issue with using a consumer grade CPU in an application such as this is that you end up running out of pcie lanes those are really important when you want to connect things like graphics cards nvme storage and a bunch of other stuff to your system so what we're going to do today is dig into Morpheus and some of the issues that I have behind it and hiding behind me on the bench over there is a little surprise it's the AMD epic 742 CPU based system so I've gone from having 16 pcie lanes over here to 128 behind me and we'll dig into all of that in today's video okay so let's dig in and see what we've got this is the Rosewell lsv 4500u it cost me I can't quite believe it it cost me50 $ about four years ago Black Friday season was good in 2019 this one didn't come with this hot swap Bays that I've added in the front here so uh this was just the bare bones case the lsv 4500u in it over the years I have put in a dual xon system and it was an LGA 2011 and then most recently is this ASRock rack based motherboard the E3 c246 D4 for you these model numbers just roll right off the tongue don't they all right so let's take a little look at what we've got going on inside this case and as I go I'll try and explain why some of this is a problem so we have a a Micro ATX motherboard inside here the ASRock rack motherboard then I've got a nvme card in the 16x slot at the top I've got a 10 gig SFP Plus network adapter in the next slot which is a 4X pcie slot and then in the bottom I've got my HBA card which again is another 8 by pcie gen 3 card now on the face of it this motherboard actually looks like it should be pretty much perfect for this application however the annoying thing is PCI switching and the the number of lanes available on consumer platforms from Intel just isn't really good enough so the CPU itself has 16 pcie Lanes the motherboard chipset which is where the other PCI Lanes come from has I think 2 four off the top of my head but looking at the block diagram of the motherboard itself from ASRock the chipset actually only uses nine pcie Lanes plus a few other things for like USB connectivity and connecting to BMC and that kind of stuff into the chipset this the c246 chipset and so what we end up with is a situation where we just don't have enough pcie Lane to support what's going on for example I have this card from Asus this thing lets me take a single pcie slot so a 16x slot like the one you typically use for your graphics card and add four nvme ssds in VIA that single slot the catch is that it requires something called bicationic I've actually got a Gen 4 One sat on the bench behind me ready to go for the Epic build that's coming up but I've got a 2 tbte um 22 110 so 110 mil long card here a standard 22 was it 2280 yeah 80 mil long card and then this one which goes from the steam deck which is super small but the idea is you take a single 16x slot using PCI bicationic slots gets four PCI Lanes each which by the way is why these ssds are so fast they just have tons of bandwidth available to them so the the idea was I wanted to put this card in the top slot of this motherboard and have 2 by two uh terabytes with a couple of other NVM ssds that I found this one I actually found on serverbuilds this is a Toshiba 2 tbte Data Center Grade enter Enterprise grade uh very high right endurance drive but the first problem I found was that the bif foration on the first slot of this motherboard only goes as granular as 8X 4x4 whereas what I actually wanted was 4X 4X 4x4 so what that would mean is that I could get these two slots here to show up as a single 8 by slot and then these two 4X slots here are individual individually addressable the next problem came when I added in my SFP Plus card last year which meant I had to move my HBA out of the top slot and put it in the bottom one down here as soon as I do that it activates a pcie switch on the motherboard that takes the first slot from being a single 16x slot or a bated slot if you so choose and turns the first slot from a 16 into an eight as well as the bottom one into an eight as well so in in essence what I ended up with was a mechanical 16x slot running as an 8X a 4X and another 8 now now that doesn't quite compute into the number of lanes available on the CPU so the CPU presents 16 Lanes but then the chipset as I say presents a few more so some of these slots are wired directly into the chipset and then they connect back to the CPU over the DMI bus or the interface the DMI interface um and it's it's just not a great situation I've kind of limped along I say limped it's it's a it's a very good system it's been very stable and I actually looked through my notes before making this video haven't actually taken the lid off this server in about a year which should tell you just how stable it is I mean the only thing I've done is changed a couple of hard drives out on Black Friday at the front there but for the most part it's an incredibly reliable platform you know desktop grade CPUs or whatever you know they're great they hardly ever go wrong but the real reason I've stuck with this platform for so long is Intel's quicksync now I made a video about some quicksync benchmarking scripts which I do plan on getting to release releasing a conclusion video about soon but in the meantime the reason I've stuck with this platform for so long is because of Intel's quicksync and that is a graphics card basically built into the CPU which Handles in Hardware all of my video transcoding needs for a budget of power budget of maybe 10 or 15 watts at most when this thing's plugged into the wall with all the hard drive spinning and all the pcie devices lit up and everything just sort of doing a a baseline workload I won't say idle I've got into trouble for that before but just a baseline workload it's pulling about 100ish Watts give or take and so my Benchmark is can I build a more powerful system to replace Morpheus with essentially unlimited pcie bandwidth expansion options and come in somewhere close to that power budget and so that is the Crux of today's video we're going to take this consumer grade platform with its limited pcie connectivity and upgrade to a proper server grade platform with all the PCI Lanes I probably need for dare I say the next decade and so here it is the solution to all life's problems no on this occasion I'm not talking about beer this is an AMD epic based CPU based around their Rome architecture so the Zen 2 version two of these chips this comes from late 2019 and you can see here we've got I think this board was probably designed by a psychopath cuz notice the silk screening on the CPU chip is upside down the way you should probably look at this board is this way up right and this the writing on the CPU is upside down but it doesn't affect the functionality this board is an absolute gem as you can see it's got 1 2 3 four five 16x pcie slots it's got two 8X slots here two nvme slots here for nvme drives six SATA supports with two more down here these six go into an LSI SAS controller I believe and this is a slim SAS or slim sat I think connector I'm going to go into more detail on this board in the next video CU this is just a little teaser I'm not actually doing the build in this video otherwise it would be 7 hours long so what we're going to do is just admire the beauty of this thing for just a moment longer the reason that this Rome platform is so exciting is just it I mean 128 pcie lanes that means each of these slots can be individually bated so I could put one of those mvme cards in each of these slots and get 4 8 12 16 20 ssds just here with two more there and then room for I guess four more between these two slots there's obviously more you can do with PCI than that but you'll notice that one of the things that starts to become a limiting factor is rather than not having enough Lanes it's just not having enough space between the slots themselves so this is a super micro h12 SSL dasi motherboard and it can support up to 2 terabytes of ram which is frankly kind of ridiculous my spec is going to be for 256 gigs of RAM I've bought four sticks here with 64 gigs each uh this should be plenty for now 256 gigs I mean I'm coming up from 64 so it's going to be a nice upgrade the thing to look at is the number of cores and threads on this chip so this is the AMD epic 742 24 cores 48 threads the clock speed slightly slower than the Intel but there's a few niceties on this board as well you know got pcie Gen 4 versus gen 3 obviously I've got a lot more cores to work with so I'm hoping that a lot of the containerized workloads that I I put through this thing and virtualization I put through this thing will benefit greatly from that multie architecture being able to distribute the workload more evenly across the CES but I think that will probably do us for today I going to go away and now start building the server that this goes in I've got a sliger case down there which you'll see in the next video as well and then the final video in this kind of miniseries is going to be the HL from 45 drives versus The Rose will 4500u which you saw earlier versus the sliger case that is well it's it's sat over there ready to go waiting I haven't touched it yet so I can't give you first impressions uh so stick around subscribe to the channel if you haven't already and until next time I've been Alex from ktz systems

Channel: KTZ Systems

Views: 34,263

Keywords:

Id: B2cjMSVRC-c

Length: 11min 40sec (700 seconds)

Published: Sun Feb 25 2024