An Epyc Homelab Monster: the Perfect Media Server mega upgrade

this should be a good one I'm almost on first name terms with my UPS guy after this week lots of new stuff has arrived we've got an AMD epic 742 CPU a super micro h12 SS l-i motherboard a sliger CX 4712 Nas chassis a a 4u rack mount Nas shassy uh an Intel Arc Pro A40 unobtanium graphics card that one of my podcast listeners sold me uh we've got a u.2 I've got a couple of these 2 tbte intel u.2 nvme ssds uh three 4 tbte crucial mx500 ssds for video editing over the network on ZFS a bunch of 3 and 1/2 in spinning hard drives a pair of 20 terb hard drives but in the server just off camera Morpheus my old server there are maybe seven eight I can't quite remember 7 or eight maybe more hard drives in there ready to go 256 gigs of ddr4 ECC memory clocked at 32 mega mega transfers clocked at 3200 megga transfers per second this is Samsung memory and then for the boot drives it's about as simple as it gets really I've got a pair of one tbte SATA 2 and 1/2 in ssds and I've deliberately gone for different brands I was doing my res Arch over the weekend and found that these 870 Evo drives from Samsung have a little bit of a problem with their firmware or something never quite sure what to believe when you're researching things like that on the internet but you never know so I've kind of split the difference I've gone for one Samsung one TB drive over here coupled with one 1 tbte crucial 2 and 1/2 in SSD and they're going to run on the two orange SATA Dom ports down the bottom here on the Epic motherboard now this is a a voyage into the unknown for me I've never used any sliger products before but they did send this case over for review no money changed hands they just sent it over and said here take a look let us know what you think so here we go we're going to do just that and boy are we going to put a beefy system into it today so what have we got here with this case we've got 10 slot loading uh 3 and 1/2 in Drive Bays on the front this lovely sort of magnetic RI you know front panel that just comes right off I've asked them through email I haven't had a reply yet but I did only send it this morning um can you get some kind of a filter for this front mesh because you probably see you know it's it's quite porous so any any dust in the air is just going to get sucked right into the system there are a couple of cases to compare this against in my mind you've got the hl5 from 45 drives which I reviewed a month or two ago along with the Rosewell lsv 4500u which is just out of camera over there and what's interesting about this case is it comes in at about a $400 price point for that you get a full ATX actually it supports eatx motherboards uh full for you height so you can use ATX size power supplies like this seic Prime gold I've got over here and you don't get a pre-wired back plane but what you do get in the Box are these kind of are these sat to SAS adapters so in the front here you can fit 10 3 and 1/2 in hard drives they can either SAS or SATA depending on what you want to do and then down the side over here just about see that in my bcam down the side over here you can fit four more 2 and 1 half in hard drives but where it starts to get really interesting is you've got a pair of 5 and a/4 inch bays at the front here as well like 2 CD ROM drive size so I've bought this guy this is an icy do case and it's a 6 in1 2 and 1/2 in SATA uh hard Drive cage and it goes in one 5 and 1/4 in Drive Bay and it just has little spring-loaded Clips on the front here which pop out and you got a little Drive sled that your SSD will fit into and then it has pass throughs uh SATA on the back with typical sta power and a couple of probably noisy fans because they look small and cheap this is just going to slide into the front here and give me space to mount six 2 and 1 half in ssds that I can hot swap out in and out the front as well as space for four more internally so I'll put my boot ssds at the front over there and then over here I'm going to put data ones so this could potentially be useful for like offloading footage or something okay use a ninja uh what's it called a ninja 5 plus to do all the recordings for for this Channel and that's using a Satur SSD so maybe I could offload footage that way I probably won't but I could so things actually start to get quite interesting when we look ins inside the case at the fan wall in the middle in my Rosewell case I've completely removed that middle fan wall and just rely on the drive caddies the four four three and a half inch Drive caddies to pull pull the air in from the front and then I've got a couple of 80 mil fans at the back to suck air all the way through and the middle of the chass is kind of No Man's Land But Here There is not space for front fans and that's because it's a front loading trailers system for the hard drives and there isn't really space on the sort of I'm going to call it a back plane but I'm going to use inverted commas as I do it because it's sort of a pass through back plane so it's still hot swappable drives but you'll directly connect to cables to the back plane yourself manually rather than the 45 drives approach or even the uh Rosewell approach where it's got a PCB available whereas this is literally just those those adapters screwed into the metal I I didn't I kind of found this out by mistake I was messing about last night just looking at this thing before I pressed record and I discovered that the motherboard tray is removable I have never had a case before that lets me do this and I think it's the coolest feature in the world Jeremy Clarkson would pleased in the world now if I pull this out I can I can literally take out the entire on once I clip a zip tie there we are zip tie clipped I can take out this entire tray so oops and this this is this is solid like some some sheet metal can be really thin and flimsy but I don't know if that comes across on camera but that is solid thick steel and that ain't going nowhere so this this tray as I say supports uh ATX all the way up to e8x motherboards and it really is going to make my life installing all of this stuff much much easier I think it's one of those things it's going to be useful for the the next few few minutes whilst I install the motherboard and things like that and then after that who knows but uh I guess time will be the judge on that one that's a quick overview of the slagger case itself it's probably about time I put some of this Hardware inside the case right so let's get on with it shall we this CPU Cooler is the one that came with the motherboard and CPU and RAM combo that I bought on eBay I wanted to just save anybody that buys this exact cooler just a few minutes of time I thought to get in here to to do up some of these screws there are four screws to connect it into the socket right when you when you're actually mounting this thing and to get to a couple of them you have to take the fan off and I started you know trying to reach my screwdriver in from the side like this trying to it was just it was just really annoying and then at the the top here you actually can't physically get your screwdriver in Well turns out I'm stupid all you need to do is undo these two screws on the top and then just simply Slide the whole fan assembly right off and it comes into two pieces and you get full access to all four screws to put it into the socket i' say I'm about halfway through the build now the system is booted for the first time proxmox is installed and now I'm looking to install of the all of the storage and things like that next up is this pair are these pair of ud2 Intel nvme ssds I've got one here that I've had since midco at some point uh p4500 and one that I bought last week a 4510 a p4510 Intel SSD and what's cool about these is because I can now bate my eight times PCI slot I'll come on to the bation stuff in a little bit but essentially it means I can divide one of my pcie slots up into two each of these things needs four pcie lanes to operate properly so I simply take this card slot two of these ssds into it enable verification in the Bios and I'll be off to the races right so with all of that testing done I think it's time to actually put this tray back in the case now tuck these cables out the way this is the first time I've ever actually been able to build a computer outside of the computer case and then just shove it in I guess this is what the blade guys must feel like you know all nice and smug look at me with my blade server I can just shove it into a slot that's kind of what this is on a much less technically advanced level so it's nothing like that at all is it that was easy I was expecting it to be be really fiddly especially with all the extra weight and stuff like that so let's put the screws back in where' I put the screws kidding I've got a little magnetic screw pad over here to my left this uh this little screw pad by the way you go look on the bcam this little screw pad came with a a wow stick from I I think recommended from random Frank P of all YouTubers the screwdriver itself is terrible honestly but this little magnetic pad for screws is fabulous I use it all the time for it's just it's just sticky enough that you can pick things up and move them around and they they'll mostly stay put was a great demo something fell off wasn't it obviously it depends how Ferris your screws are but uh I I love this little mat here so in the side of the case back here to screw in the sort of back wall and actually bring some of the rigidity of the chassis back again there's a couple of screws to put in on either side you can see it sort of flaps about a little bit here because it's not screwed in soon as I put this screw in it'll lock it all down lovely there we go that'll do the trick okay so it's probably time to start thinking about Cable Management next so in here um I'll do all the PCI PCI devices last cuz they'll be easiest but it's stuff like rooting all of the SATA cables from front to back around the fan wall into the back plane area that I really want to start thinking about next um I've installed the 5 and 1/4 in um SATA SSD caddies into here obviously in is are they tooless oh that's nice and then there's just a couple of screws required on there there are some very spefic specific sized screws here to go into the bottom of these toolless mostly toolless 2 and 1 half in Drive Bay CES for the icy dock so a pair of screws goes in the front of these so I need to make sure I label these then once the screws are installed it should just be a case of pushing it in and then blocking it in place yeah love that as easy as that to install a two and half in SSD yes please okay with a bit of basic organization taken care of it really is time to start rooting cables now so my two boot ssds the Dom ports these two orange ones I don't really want to have to run these cables all the way over here if I can avoid it that's quite a long run actually what I've done in the rosew case is I just have a pair of you can actually tuck you can actually tuck in the rosee you can ssds and things down the back of the power supply there's just enough room I've I've run mine like that for the Last 5 Years and had no issues whatsoever before someone tells me they got to overheat back there they've been they've been just fine I found my little baggie of SATA cables I love things like freezer bags for organizing cables this par Ikea make of all people Ikea make the best freezer bags besides the the traditional zip blocks uh although I think I actually think these probably are a little more solid this bag has been with me since London and it's still in great condition uh but the reason I like these bags is because you can just throw all the cables inside them you don't have to really tidy them that well and then you just literally go and grab your bag of SATA cables and bring them over to where you want to go something else I've picked up over the years as well is to if you can always try and use these locking SATA cables sometimes the ones that come with motherboards don't have the little locking adapters on even actually I'm noticing my little slim SATA sff to8 SATA breakout the these ones are just friction fit and I have had on a couple of occasions the friction fit just work just a little bit loose so particularly on things like a boot drive if you can try and use the Locking ones all right so I've just finishing up putting the two ssds the two boot ssds in this front corner down here and you know the access is it's tight but it's doable I think if you're going to install all four you might want to consider taking out this fan wall bracket in the middle which is held held in place with there four screws two on each side from underneath not the easiest installation it would be kind of nice if it was like a toaster maybe maybe they could sell a back plane it would be like a toaster and I just insert them like a slot I mean there's room you know if you do a similar layout to the icy dock on the front there's room for six or something down there you know but in general it's a little fiddly to get in that front corner but totally [Music] workable [Music] [Music] a [Music] and just like that the build is done I cannot tell you how nice it is to be treated like an adult by the bios in this thing let me explain what I mean I went to enable bicationic cards in this system and some of them need to have the pcie slots bif what that means as I said earlier is you take one 16x slot and you divide it up into four so for example I have an Asus m.2 card which takes 160x slot and lets me put four m.2 nvme ssds in it to enable that to show up to the operating system as four individual drives I need to turn on bif fication in the Bios to make it show up as four * 4X PCI slots in my Intel days I've had dual zons with an LGA 2011 platform and bif fication was kind of okay but it was a bit of a crapshoot to be honest with my consumer grade Intel CPU the I5 8500 that's been in my media server today Morpheus that only has 16 PCI Lanes on the CPU and 20 or 24 on the chipset and the bation in there doesn't expose the granularity that I needed to do what I needed to do in this it was trivial it was it was genuinely boring which is let me tell you it's the best kind of way to be in this situation so I was like right going to look look up in the motherboard manual which PCI slot or the silk screening on the motherboard itself right CPU pcie slot six I need that slot for example to be 2x4 slots for my to u.2 nvme ssds so I go into the BIOS and enable bicationic slot and divide it up into two 4X slots reboot boom just works no problem whatsoever so here I am in the Bios and I want to enable PCI by forication so if I jump into the pcie menu scroll all the way down look at this it exposes to me individual controls for each slot now here I I'm not entirely sure which slot that I is I want to bate so I I grab the motherboard manual and it shows me PCA slot to that's the one I put my two Intel nvme u.2 drives in and if I select this option here it gives me the ability to turn a single by8 slot into two 4x4 slots and what you're seeing here is the magic of ipmi I'm connected through a web browser seeing my full bios and boot process happening right in front of me now the last thing I want to do is verify that those two nvme ssds appear as I would hope so I do an Fisk minus L and look at that both of the pcie devices have been bipoc correctly and they just showed up I mean that's how easy it should be are you watching Intel another example PCI pass through this is what got me interested in Linux in the first place 10 or more years ago when I was a poor student and Retail worker and couldn't afford to buy two computers PC pass through is notoriously picky when it comes to bios settings Hardware compatibility and all the rest of it so I thought right I'm going to set myself a challenge can I spin up a nyos svm pass through the Nvidia RTX A4000 that's in there to that nixos SPM and get it working within 20 minutes yes yes I could Theo mmu was enabled out of the box I didn't have to do anything there the io mmu groups were perfect each individual group just everything had its own individual group no interrupt remapping or ACS Downstream patches or anything like that was required it just worked and when I say it's boring that's because I've spent the last decade with it being incredibly unboringed to get it working in this epic based system everything everything when it comes to pcie devices is is just it's a Dream It's a real it's a top tier platform and if you're in any doubt as to whether the Epic Rome platform is going to work for you in your use case and you've got a couple of gpus a couple of nvme ssds I can wholeheartedly recommend this platform for your use case so let's talk power draw for a moment that was one of my biggest concerns going into building this system I'm delighted to say that at idol just the CPU and the motherboard itself with obviously the RAM and the boot ssds 50 Watts 45 50 Watts or something like that which is a lot less than I was expecting I'm coming from my dual zon days where I could expect 100 150 plus just at idle next I added in the 10 gig connect X3 SFP plus Nick that I'll be using to get 10 gig networking going in this system it was about another 10 watts then I added in my Ark Pro A40 single slot GPU another 10 watts give or take at idle the next thing that I added in was yes the Nvidia GPU the A4000 now this is really interesting at idle with no drivers or any kind of power management enabled the A4000 Drew about 40 to 45 Watts doing absolutely nothing and that meant the whole back area of the case there got really kind of warm so I had to install another noct Tu fan just at the back there just to get some more air flow going through I've got three 120 mil fans here right in the middle but I wasn't quite able to get enough air flow with just those alone however I'm pleased to report you know that nixos svm I mentioned a moment ago as soon as I enabled the PCI power management for the Nvidia cards in NX OS the idle power usage of that Nvidia GPU dropped from 42 watts to eight Watts yes that's right eight Watts at Idol Oh What A Dream the next thing to add in with the two u.2 nvme PCI ssds along with the um Asus hyper 16x 4 nvme m.2 SSD cards so two pcie cards were added those added about 30 Watts give or take to the system total and then at the front the business end 10 spinning hard drives those added about 60 wats give or take and so if you've been keeping score at this point we're at approximately 175 Watts I've been using this kilowatt here to measure the power I I don't know how accurate it is but all the measurements were taken with this device here and The Benchmark I'd set myself was to get as close to to 100 Watts as I could which was never going to happen with all of this Hardware in this case the reason I set 100 watts is because my previous server has an igpu so I don't need a PCI graphics card in there to do quicksync transcoding it had just the same number of hard drives but it had way less pcie devices in there so that thing Morpheus idled at approximately 100 Watts so I was think oh come on how close can I get to that with with this server here 175 Watts given the fact that this is a 24 core 48 thread monster system with 256 GB of ECC memory in it not to mention the 10 hard drives the three 2 and 1/2 in ssds at the front the what is it four m.2 ssds another two u.2 ssds two gpus 10 gig networking and an HBA card as well frankly the fact that this comes in at 175 Watts it just blows my absolute little tiny brain into pieces this server is it's I cannot really explain to you how pleased I am at the outcome of this the power usage is way better than I expected the usability of all the PCI stuff is way better and easier and more reliable than I expected the only downside perhaps is the cost it's not crazy for what you get but it's also not what you'd call cheap so I paid $1,700 for this CPU and motherboard combo from from a chinesee eBay seller called TM I'll put a link to him in the in the description down below for an eBay listing which came with the CPU Cooler that you see in here as well I ordered of a Chinese New Year and the shipping times frankly were were totally reasonable considering was about 2 weeks I've had Buddy's order from the same seller though and it takes a matter of days it's airf Freight shipped out and even when my buddy had an issue with some of the surface mount components being physically damaged on his motherboard the seller just cross shipped him a new one couple of days no problem it's not often I bother recommending an eBay seller to people but this one in particular is worth your attention now a couple of other things to think about the probably one of the biggest issues I've found so far are the two u.2 drives those are sandwiched in between a GPU and the Asus 4X m.2 pcie card those u.2 discs get pretty warm actually pretty toasty we're talking sort of 60 65 Celsius underload so what I'm thinking of about doing is taking one of the 5 and a/4 inch Bays here at the front of the case and ordering some kind of a cage or chassis I think I see do make one and then I can just frontload the U2 discs in there and get some cool fresh air in the front straight over those U2 drives so that that really congested pcie area at the back there has just a little less heat to deal with finally I want to touch on an area that has been a problem for me with super micro motherboards for years and that is fan control it can be really tricky to get get them to play nicely with nocta fans specifically because they spin so slowly so what happens normally is the noctua fans idle at about 400 450 RPM The BMC can't comprehend that a fan would spin so slowly and then it goes there must be a critical failure so it ramps the fans all the way up to 100% And then gradually brings them back down and the cycle repeats so I found a script which I'll put a link to in the description down below python script which monitors the CPU temperature and uses raw ipmi tools to Monitor and control the fan speed and keep the thresholds exactly where you want them so I found a nice sweet spot is roughly 20 to 25% something like that that's not too loud when I'm in here in my room you know tinkering away in the basement I'll probably just turn them up a bit because it's a basement that's kind of the point I don't care but if you're living with this inside a house you will hear it even with not to a fans the just the sheer amount of pcie devices back there dictate that you're going to have to have some decent air flow pushing through the case so how did the case do well I'm going to do a full video review of the case because I've got some thoughts on it overall it's pretty good actually for the $400 price point the the build quality is fantastic it knocks the Rosewell out of the park there are literally a couple of sharp edges nothing too crazy and probably my biggest complaint with it is how the back play here I I kind of I'm reluctant to call it a back plane even I mentioned that earlier in the video it's just those pass through SATA SAS connectors literally screwed directly into the metal at the back here which means you have to Cable up the drives individually yourself frankly there isn't much room in here I can just about get my fist in there and then I I actually move the fan wall back a slot there's two there's two positions the fanall and occupy I moved it back a little bit just to give myself just a little bit more room um that's probably the biggest drawback of this case one thing that sliger could do to fix that is to modify the included SATA power cables to be custom length I mean they know what the spacing is right I know it's going to add a little bit more cost or even have a a button that says you know connect me to cable mod who will do the exact spacing on the 10 power connectors that I want here something like that would would actually go a long way to making this quite a lot better the other thing that I also mentioned earlier is Maybe the SSD Mount here if it was vertical or something else you get a bit creative with this space here there is a lot of space to work with but overall really quite a fantastic case I will be working on some other videos over the next few weeks on things like how I'm going to be using NYX OS for testing might be putting perfect media server on top of nyos no promises but I am I am working on flake for that exact purpose right now and there's going to be a bunch of stuff with like self-hosted AI like the O llama stuff machine learning this box is going to unlock a lot of potential for me and I've got some data to move around I've got a couple of spare discs that I've kind of rotated in and out uh throughout this whole process but I am really just delighted it's done first of all because it probably took me about 8 nine 10 hours yesterday including the filming I suppose so that I mean I guess it could have been a little bit less if I wasn't filming but still the biggest issue was just putting the cables in here that really did take me quite a long time to get everything lined up just as I wanted I'll also be giving some more thoughts on the self-hosted podcast at self-hosted doow and until next time I've been Alex from ktz systems

Channel: KTZ Systems

Views: 26,874

Keywords:

Id: 91dp5l44X8A

Length: 29min 35sec (1775 seconds)

Published: Fri Mar 01 2024