Shiny New 128 Core DevOps Build Server For Greg Kroah-Hartman! AMD Epyc ftw

uh yeah yeah okay um wow only 10 compiles an hour well i think that's something we could probably deal with yeah sure all right i'll plan on it we've got a build to do [Music] we've been called into work today for none other than mr greg crowe hartman number two on the linux kernel what does he need it doesn't need a workstation he's still rocking his ultra high-end ripper system 32 cores of monstrous insanity but also not super loud i get it i work on stuff too and i want lots of horsepower and it's quiet and powerful it's really hard to do so now he needs more horsepower not in a workstation class but in a server class see they need a build server we did another video talking about kc bench and kc bench rate it's a kernel compile benchmarks but it's not just compiling the kernel you also have to compile the drivers and everything else and you would think you know maybe the workflow for that can be improved you could rearrange the header files and get some more performance out of it yeah that may help for the future but don't forget we've got long-term kernels that have to be supported so what do we do we do what most companies do we throw hardware at it so much hardware so this this is the smallest possible build if you want a high-end system like this this is 128 cores you already know what you're getting into 128 epic cores this is pretty much the smallest dual epic that you can build dual 280 millimeter radiators heck we even threw in an rx 6900 xt this could just as easily be an mi 50 or some other instinct card for machine learning could put up to a terabyte of memory in this this is the salvo studios case for this kind of nonsense yes this is a custom loop we got our pump and reservoir on the back i did a separate video on this as awesome as this is this is not for someone like mr gkh because this is not a zero maintenance for 10 years system so what do we do we do a build first up the processors this this is a care package from amd the angels have heard our prayers this is twenty thousand dollars of processors 77 63 128 cores in this box for mr gkh sponsored by amd thank you very much amd so that's the most important thing now why the 7763s why does it have to be the 64 cores is that just the flex no we did the video on that you can check out the kc bench stuff even the 7713 64 core cpus which are the lower clock rate 64 core epic cpus could outrun 75 f3 cpus now the 75 f3 cpus those are 32 cores but they clock up to 4 gigahertz they are insanely fast for any kind of mixed workload those cpus are unstoppable for anything else on the planet but for this kind of a job more cores went out over clock speed and i'm surprised by just how much these 7763 cpus for the kernel 5.15 benchmark they're going to be doing around 30 kernels per hour with the full module config no cash everything else like that the 75 f3 could do 22 23 but for this system for the absolute maximum number of kernels per hour 30. that's moving up from 10 on his 32 core system so it's not a linear scaling either moving from 32 to 128 cores we don't really have a linear scaling to the number of kernels per hour that we can expect but we also found that adding a second socket does help us in a much more linear way than adding more cores per socket because we tested 32 core in one socket 64 cores in one socket 32 cores per socket in two sockets and finally 64 cores and two sockets this is the best hardware for the build so what's next there's pretty much only one case choice the fractal torrent why because it's got giant fans this is the mz72hb0 the box is almost not labeled in fact the box actually isn't labeled there's a sticker this is super nice eco-friendly this motherboard is the unstoppable fire breathing motherboard that can supply 280 watts per socket to this thing without overheating so this is a server motherboard it is designed for a server chassis the torrent is not a server chassis but it also has to be quiet this is the best fit for retro converting a server motherboard into a desktop into a tower server without really giving up a lot assuming that you don't need a lot of two and a half inch drives you don't need a lot of uh of other connectivity other than just a place to stick a motherboard i'll also mention the lienlee dynamic xl this is a runner-up because it could hold two power supplies but it's not really possible to configure the power supplies in a redundant fashion so for the power supply we're going with an fsp group power supply fsp group is a huge company they make all kinds of server power supplies they do everything you know all aces in terms of power supplies they have an atx sized power supply that actually has two power supply modules that's what we're going to be using for the power supply on this build and for the coolers we're doing something a little unusual the freezer 4u sp3 now it's easy to imagine at some point in the future mr gkh may transfer this to a rack mount case these fans are compatible with rack mount 4u configurations so that's really awesome but these these coolers will also keep our 280 watt cpus under control which is nice now you might be wondering why not noctua well i do have nacho in the system behind me which is basically identical same motherboard and same case that's why i was so impressed with this case you might think the horizontal cpu configuration is part of the reason why i went with arctic that's not really the whole thing this push-pull configuration with two fans the nachos are gonna do a little bit better for cpu cooling for the testing that i've done this room's a little warm even with the side on now if you're gonna do this with noxua you're gonna have to use a 120 and a 140. two 140 side by side perfectly in line like that it was a little tight and the airflow over the vrm's is a little bit more impeded with the smaller one in the front the 120 again a push-pull configuration that seems like there's enough turbulence that the vrms get the airflow that they need big reason is that 4u mountability these will not fit in a 4u case but gkh could transplant his system into that 4u rack mount system and that'll work a little better with the arctic configuration strictly speaking the performance of the arctic coolers is on par with these noctua coolers in this configuration so no complaints there really now for the memory do you not have any idea how hard it is to get memory in the current climate sponsored by level one because that's all we could do sk hynix i'm pretty sure that i'm going to use this sk hynix hmaa4gr7 aj r8n these are 32 gig dimms this is only going to give him 512 gigabytes of ram i mean if you're building a system like this don't you at least want a terabyte i mean come on but because nobody wants only 512 gigabytes of ram these are actually pretty affordable so i'm happy about that yes we're using eight dimms per socket because we've got 64 cores we need as much memory bandwidth as we can possibly get and yes those are dual rank dimms and they are 3200 so this is gonna really really sing now full disclosure arctic did send us one of these but i used it in another build so both of these are also courtesy level 1 techs look at that cooler it's all heat pipe all heat pipe all the time with dual fans comes pre-installed plenty of clearance for ram it's not like our rams gonna have wings but you do want airflow around your ram and the way that these are engineered this really works also note the orientation it's gonna work really well with our torrent this version of the torrent no tempered glass it's just metal the fcc will be so happy you might be thinking this case is a little pricey five fans and two of them have more volume plastic than three other fans so it's like you've got seven fans our fsp group power supply it needs about 700 millimeters of clearance but guess what the torrents got that and at the back depending on what mr gkh's storage needs are how they evolve over time he could run a raid one of mechanical spinning rust because spinning rust is so unreliable you should really at least run raid one but we've also got a ton of two and a half inch bays back here that yes we'll be more than happy to fit our storage which brings me to the storage part keoksia is hooking up mr gkh so they're going to provide some u.2 storage for the server i'm going to use some breakout cables or possibly even a pcb so probably something like this this is two u.2s on a single pcie carrier it's really tough to get pci express 4 carriers like this and actually strictly speaking i don't think the carriers are pci express 4 because some of the ones that i've got they'll work at pci express 4 speeds and the upper slots on the motherboard the ones that are closer to the cpus but they will not work in the ones that are farther from the cpus which is a signal integrity issue that the pcb really doesn't do much to ensure pci express 4 signal integrity other than physically locating the connectors as close as possible to the cart edge connector so that's something but i'll definitely make sure it's reliable before sending it out now again before you've even ordered the first part i know that the system is amazing and you just want to rip your wallet open right now and order this without thinking about it but i can't stress enough you should not do that you should know exactly what problem it is that you're trying to solve before you ever spend the first dollar sometimes it's hard you might have to you know co-opt some time on a friend's machine or reach out to me and say hey i'm trying to do this job maybe it'll make for an interesting video let's figure out this academic or this research problem rent some time in the cloud and see what works and what doesn't i mean if you look at the other videos in the lead up to this we tried many many many different system configurations and ultimately the 7763 system it stomped them all so here we are to build day this is not a problem that amazon is going to solve this is not a problem that bare metal in the cloud is going to solve heck we even tried disc to cc distributed c compiling just because i was curious it's like well two sockets is good four has got to be better right it wasn't worth the headache i do love gigabytes eco-friendly motherboard boxes look at that oh it's so beautiful and look at that big tall vrm heatsinks because 280 watts per socket 10 gigabit lan plus ipmi onboard vga we've got gen 4 breakout plus onboard m.2 man this thing is sweet now i'm working my anti-static workstation so make sure you have an anti-static workstation too maybe one of the gamers nexus mod mats those are nice all you get in the box is a couple sata cables which are not using the rear i o shield and this this is a motherboard installation manual for people that know what they're doing yo dawg here's the connectors good luck by the way we're on facebook like and subscribe [Music] [Music] now on this gigabyte motherboard these cpu sockets are not offset at all they're perfectly in line so one heatsink is gonna feed the other i'm a little worried about that but we'll do some testing and make sure that it's okay i'm also not going to mount the coolers just yet it's sometimes a good practice to mount your coolers before you install your motherboard but there's so much cooler here i'm not going to do that and they're relatively close to the board might make it a little harder to screw in some of the screws so i'm just gonna go ahead and mount this in the uh the fractal torrent case and then we'll install those fans so let's get the case i've relocated our standoffs now the fractal case doesn't have one here at the front where this motherboard has one so you can see the hole here on the back this is a normal position a normal atx position this screw and it has the atx screw in the line with that the problem is that our dims are in the way so that's moved down very slightly to be in line with this screw here there is an extra standoff in the box so if you only have one or two standoffs that don't line up and everything else does i'll show you a trick sometimes you can screw one standoff into the other one upside down and then this will provide mechanical support it doesn't fasten the motherboard to the case but if you press down on the motherboard it will provide some rigidity and once again we check all our screws there's nine remember counting the one that's uh a little arbitrary in the last year we've had four people come to the forum that have had problems with their motherboard that were problems with their build that were mysterious and strange and ultimately it was because there was a standoff that wasn't supposed to be somewhere shorting something out so it happens count one two three four five six the special one seven eight nine that's how many we had seems like we're in good shape warning please peel off before you use good advice what i really like about the design is this the heat pipes are in direct contact with stuff the io dies got a couple of heat pipes and there's just there's just heat pipes everywhere so thermal paste got my two tubs of mx-5 now notice that these have air flow marked on them and this is you know this is there's an asymmetrical design here arctic is thought of things if you're like me and the orientation is not what you need you can flip which side's what one thing you gotta be really careful about is when you're tightening this with your screwdriver if you slip you're gonna stab some ram and that's going to be a 200 mistake our airflow arrows are pointing the same way even though our sockets are rotated on this motherboard and there's no offset so i'm gonna have an adventure getting both of these fans in place i may not be able to get both fans in place we'll try it another nice feature of these arctic fans is that it's a pigtail header so you can have both fans feeding off of one header if you need to do that and these server motherboards typically do not have a lot of headers so it's a nice touch now without the other fan in place on the back here the vrm is going to get more airflow so i'm going to leave that off for now there is enough room for it to fit with zero gap there would be no gap whatsoever now the last and most important piece of this build is the power supply something very special i reached out to fsp group because this caught my eye and i said hey this is for a good cause will you uh give us power supply and i said okay so this is the twins pro 900 watt power supply twins pro it's a standard atx form factor but with the modular hot swap now it's not just one 900 watt power supply it's dual 900 watts but that doesn't mean that you can run 1800 watts off your your system it means that you've got you know primary and standby so one of your power supplies could could die and it doesn't matter this is just overkill i'm sure that mr gkh could run out and buy a power supply if something happens to the power supply but i like redundancy i would like to know that this thing will run maintenance free for like 10 years knock on wood and that's sort of what i want to give them and fsp group is a very well known name in server power supplies a very uh a-list when it comes to server power supplies so i'm pretty excited about this product that is a little bit longer than normal power supplies as i explained in the full video you can i did a sort of dedicated video on this power supply you should check that out but uh yeah yeah it's uh it's a little long it's a little big but that's okay well we got the system set up and i'm in this room not really a lot of sound treatment so you can hear just how loud it is now it's not whisper quiet and definitely the loudest thing in a system are those tiny power supply fans in each 900 watt module but as you can hear the power supply smartly ramps now it is whisper quiet when it's not doing anything but right now it's compiling a kernel load average is like 200 300 something like that and it's this quiet for that much horsepower drawing almost 700 watts at the wall that ain't bad the hottest hotspot as reported by lm sensors is about 70 degrees c i would like that to be a little bit better it does feel like a blast furnace coming out the back of the case but this is really pretty good for tower coolers and sustained load well with this system complete there's really only one thing left to do it's just like in the uptime funk music video i'm calling greg crow hartman yeah it's it's ready oh i think you're gonna be really excited let's do a chat soon i'm gonna get this in a box out the door and on its way well this is exciting for both of us one because i get to live vicariously through a kernel developer with uh fire breathing beastly horsepower and another because he's going to have fire breathing beastly horsepower thanks again to amd for providing the cpus gigabyte for the motherboard fsp group for the power supplies we're using arctic coolers the arctic tr4 coolers it's got a nice configuration in this case uh i level one picked up the memory and the uh the case and some other stuff because hey that's what our community wants us to do and while this is level one i'm signing out you can find me in the level one forums [Music] oh it's the boiler snake come to give it a good send off uh that's boiler snake for what you're referring to as linux is actually gnu linux or as it's taken to calling it recently gnu plus linux

Channel: Level1Techs

Views: 111,001

Keywords: technology, science, design, ux, computers, hardware, software, programming, level1, l1, level one

Id: zTav7r38y-Y

Length: 19min 24sec (1164 seconds)

Published: Thu Apr 07 2022