Full Noctua Fans & New Motherboard - Server Re-Build

today it is time for server maintenance and this is going to be a very in-depth one so if you haven't seen any of my previous server videos i would encourage you to watch them to see how this thing has evolved but i'm going to be getting into every single part of it today so if i'm honest you really don't need to uh because i need to take out every single part of this thing today because we're going to be upgrading a lot of key components um first off we're going to be putting in a couple more sticks of ram that'll max it out at 128 gigs but we're not putting the ram in that motherboard we're putting it in this motherboard which is an identical super micro oh i want to say x9 drd i f um don't quote me on that and uh the reason is the one that is in there now was sent to me some time ago but it came with shipping damage uh and the onboard ipmi which is the remote management interface was broken and it's caused a few problems this board though is identical and should have a functional ipmi and was also sent to me by the person who sent the original board cody and i'm extremely thankful for that because that's going to be really nice to have that working he also sent in the two additional sticks of memory that will max it out and that is a very critical upgrade that we're going to be doing today but that's just half of what we're doing because the other half is 200 worth of knockshower fans because i don't know if you can tell this thing is loud and i want to see if i can make it not loud because i have to replace the fans and if i'm going to do that i might as well really do it so today we're going to try and make the server completely functional and totally silent [Music] now i'm really curious how the fans are going to work out in this thing because i have this connected to my main desktop over 10 gigabit but i have two ports and i'm only using one of them if this is quiet enough that i can tolerate being next to it while editing i would really like to bond both ports together and have a 20 gigabit connection to the server i can't really use that because i don't have any 20 gigabit drives but it would be nice to know that the bandwidth to multiple drive pools because i have a lot of drives in this thing wouldn't be bottlenecked in any way by the network interface so i want to try doing that but also it would just be nice to not have to have this crammed into the server closet that i normally have it in because that really is inconvenient to do any kind of maintenance on this thing so if i can move it into somewhere more accessible in my desk that would be really handy now as part of that i want to compare the volume of the server before and after the fan replacement and you might think that i could use something like a volume measurement app on my phone except i can't you cannot do that with anything other than an iphone and it's not because apple has some kind of fancy audio processing it's because android phones all have different microphones and there's no way to calibrate the software to the exact microphone that's in them so it's just not possible it could provide potentially some kind of relative measurement but i can do the same thing by just plotting the waveform unedited at the bottom of the video here so that's what i'm going to do i'm going to leave my microphone which i have slightly off frame set to the same volume level when i measure the server and i'm going to place it in multiple spots around the chassis just to get a baseline reading of how it is now then once we've done all of the hardware replacements i'll put the microphone back in the same position at the same volume setting and will compare that should give me just as much information as one of these inaccurate apps will but it's going to be a little bit easier to show because i'll be able to just plot the waveform so that is how we're going to do that now you likely noticed in here i have a gpu and that's not because i'm doing bitcoin mining or something i'm running a gui because i render videos on my server at least i used to uh the version of resolve i'm using is no longer compatible with this setup for some reason so one of the things i want to do today is try and fix that so this has just been sitting in here idling basically recently and i really want to get it back into action but uh we'll be reinstalling the entire operating system today to make that happen so that's going to be a whole other fun thing now the reason i have to replace the fans is because the bearings in this one have gone bad if i try and spin it here it will not continue on but the other ones that are right next to it uh from the same time will it's really hard to spend there's a big metal grate here so if this fan is bad all three of these fans are likely going to go as well as the other two in the rear that came with this case when i bought it this is a norco chassis and you can't really get this anymore it seems uh so i'm lucky i got it when i did it was used and uh i don't know how long it was in use so i don't really know what the lifetime is of those fans so i figure one went bad the other two probably gonna go bad soon so i might as well replace the lot now unfortunately in order to realistically remove those fans i have to remove this divider which has holes in it that all of the cables are running through which means i have to undo every single bit of wiring in this thing in order to get those out i could get to the screws from the back but i still have to remove the two cpu coolers the power supply and the graphics card to do that and if i'm going to all that effort i may as well remove it anyway and i still have to remove the motherboard so realistically there's no reason not to just remove it to do this because i have to get so far into there anyway that it's not gonna save me any work so i do need to make some notes here as i go along because the drives are all connected to i think we have the motherboard and then two different hbas in here and i need to make sure that the drives stay connected to the hbas that they are connected to because at least one of these has a really dumb thing where when you connect new drives to it you have to initialize them quote-unquote which wipes them so i need to make sure that i connect all the drives back to all the same hbas so that everything is happy and i don't lose any data so that's going to be fun back here we can also see uh two other fans these are 80 millimeter fans that i'm going to replace these haven't shown any signs of going bad but just look at them that giant hub motor and the thickness yeah those are really loud i also want to attempt to replace these fans now these are really new these came on the cpu coolers when i bought them when i first got this server and i'm going to be just kind of slapping the noxua fans that i bought onto these uh the front one will mount pretty cleanly but the one on the back is not going to be great i kind of wish that i had bought the noxua coolers that are available for this thing i don't know if i knew that they were available and just wasn't willing to pay for it when i first put it together but i kind of wish i did now it doesn't matter though they'll mount cleanly to the front it's just the back that's going to be an issue elsewise i think these might actually be better coolers um the actual heat sinks it's just that the noxua fans are more in line with what i would really like to have in this thing in the long run so oh i think it's time to start tearing this thing apart um oh this is not going to be fun but it will be nice to completely rerun all the cables in here because it is a little bit of a mess as a refresher from the other videos this graphics card is connected through this riser cable this had to be done because this is the only 16x slot on the whole board so i didn't really have a lot of options and obviously the card itself is longer than the space that it has there so it couldn't physically fit because of the cpus so i'm running this extension over to the part of the case that the motherboard doesn't actually use and then i have it mounted on some standoffs back here to raise it up above the motherboard it barely fits but it does and even though it's right up against the side of the case it has never had any cooling issues at all i don't know if it's because the case mostly flows through or what but it's been totally solid so as weird and horrible as this setup looks it's been flawless so i can't really say much more about it than that um but now i'm gonna go ahead and pull it all apart i'm probably going to rearrange some of the cards here especially the 10 gigabit card i would like to put here so it's in the open airflow area because that's the card that gets the hottest and then it has some heatsinks poking out and then it won't be up against another card it'll have more area against the riser cable so i am not as concerned about uh which cards go back to where because that shouldn't matter but i'm going to leave all of the cables connected so i know which cables go to which parts of the um bays up front and then i don't have to try and match those back to the hbas later i might go ahead and label them though just to be on the safe side once they're out but i don't think i'm going to need to worry about that so let's go ahead and start getting all of this out now and now finally oh come on there we go okay so this is my problematic motherboard um you can sort of see in this area that the serial port is bent inwards and then this vinyl tape is actually blocking a usb firmware programming header um that was sheared off and on the new one which i'll bring over in a moment we can see that it's actually a really robust connector so that took some effort to break off um i couldn't really tell any more damage i think maybe the vga port yeah the vga port is knocked back as well but that's really hard to see so this thing did have some unfortunate uh damage in shipping but thankfully the new one is in much better condition the only minor thing i can see is probably not shipping damage there's a little bit of a break on one of the sata ports but that's nothing so let me get that one over here as well and then we can start transferring parts over all right i'm uh hoping here this video didn't just take an unfortunate turn um when i was setting up the server i was trying to find a mouse and a keyboard to plug into it and i noticed that the board didn't have a ps2 and thought oh i thought it did okay and so i went grab usb peripherals well i'd looked at this board and saw yeah no ps2 but this one does this is an x9 drdif and this is an d x9drd f uh so i'm guessing these are different revisions of the board and now i need to go do some research to make sure that i'm not gonna run into any problems here most likely not but i just want to make sure that there's uh nothing waiting for me here that might be an issue like cpu compatibility or needing to update the bios or something so all right it's probably fine everything looks like identical um the only real difference i'm seeing is that some of the capacitors are from a different series so i think i'm going to be fine um but that's unexpected so yeah let me go do some research probably all good just want to make sure okay well as far as i can tell from research there is absolutely no difference they just stopped putting the ps2 port on at some point they're both labeled revision 1.02 up there so they should be fully compatible i don't understand why they dropped the ps2 port but whatever that's fine uh so i'm gonna go ahead and start working on migrating the parts over to here um and then we'll test fire it and i have the power supply over here still i'm not gonna put it back in the chassis for that the vga port should be functional on this one it's not on that one because the that's one of the casualties of the unfortunate shipping damage so uh hopefully i'll put the cpus in there and the ram and it will just fire right up so let's go ahead and get started on that all right i'm gonna go get some wipes and clean those up and then uh transfer them all right that should be good to go go get myself clean and then transfer okay i've got the new sockets ready to go i took a look at all the pins just to make sure that they're good and i'm pretty sure everything's fine so let's go ahead and start the transfer process these are xeon e5 26 90 cpus sir socket 2011. these are technically socket 2011 narrow but that only changes what cooler mount options you have moved over i always like to check my fingers make sure i don't have thermal paste on them after touching them when they are previously installed chips just to be sure okay there we go i do not have blinking slots for this so this one will be going into this bag now and uh that should be it now let's go ahead and get the ram populated now uh it's gonna be really easy to know which slots to populate because it's all of them now there we go that's 128 gigs of ram that is awesome okay coolers um a little conflicted about this so i bought four of the noxua fans to put on here all right yep that's good um i'm not sure how i can mount them to the back though so i want to explore that a little bit here before i mount them because while they're out and movable will be the easiest time these boxes are ridiculous so let me play around with that for a bit okay it took a bit of trial and error but i did come up with a way and it was using uh twist ties so i got both of these mounted like this so first off the front one obviously just snaps onto there comes off a little easier than i would like but it's probably gonna be fine this thing never moves so it's not really concerned uh the backside fan there's just no way to mount it cleanly so i ran some longer twist ties through the fins at the top and then to the nearby mounting point and then i needed to make sure that it mounted flush along the bottom and i couldn't get through here because these are kind of crushed on the bottom i don't know why but the fins on the bottom are too close together so it wouldn't work so i ran from the heat pipe to the outside one so i didn't add extra depth there so it should form a good seal along here and it does it on both and uh yeah that's pretty much it oh cool i have to put these on while it's on the board awesome [Laughter] okay i was not expecting that well that's just lovely okay i guess the next step is to get these mounted on then since i now have to do that to put the fans on uh so i'm gonna do one more cleaning since i've had my hands all over these to make sure there's no oils or anything and then i will do a thermal paste application and install the coolers from here things started to take a frustrating turn i lost track of how many times i actually mounted and remounted the coolers because of different issues so we're just gonna fly through here to the first problem i ran into so oh there uh adding the extra fans on the back of the coolers made the one near the cards too long to fit any of them anymore but i could at least flip the cooler around and get just enough free space to make it work okay there it's fixed there's enough room there now for cards so oh man let's go ahead and get it set up so i can test posting it that's a good sound ah yes but that's a sound [Music] the motherboard was giving a thermal warning as soon as the system was turning on that shouldn't be possible with any heat sinks on the cpus but i decided to redo the thermal paste one more time just to be sure but it still gave me the warning beep so i figured it might be the chipset heating up without air flowing over the board so i grabbed the fan divider and got to work on getting the new industrial noxua fans installed i put them in the best possible way to reduce noise by using the rubber isolation mounts i had left over from the cpu fans it was a bit more work but it can make a huge difference not having them solidly mounted to the chassis for reverberation after getting that set up and trying it again i had the same problem and tried to investigate it further with the system on what is the deal with that beep that is weird okay so i have it booted off of a flash drive right now just to test and make sure that it's not cpu temps that's making it cry um but it's fine there it goes again nothing nothing's wrong why all right so you can hear that thing off beeping in the background i'm logged into the uh remote admin um i'm running core uh lm sensors on it you can actually see what the temperatures are here um the cpu is 38 degrees c and 44c so it's running fine but the weird thing is if i go over to the sensor readings the board itself can't get the cpu temps that's why it's complaining so much so i'm working with the person who sent me this right now to figure out what on earth is going on here because this is super weird some explanation is needed here the fans on this board are not controlled by the chipset or operating system they are controlled by the ipmi system which is basically an entire middle management computer built into the motherboard for better or worse it watches the cpu temperatures directly and adjusts the fans based on different profiles this control was more direct than i thought it was before i started this project and led to some unexpected issues later but for now i needed to figure out how to get ipmi to see the cpu temps so it knew the computer wasn't melting all the time after getting some more help from cody who sent the board i had an idea on how to fix it okay apparently i need to go into maintenance um factory default and then restore here all right let's see what happens oh fan speed just ramped down that's so quiet now oh man i know you can't see it uh i can start recording i'm gonna do that okay so you guys have a camera view i'm actually not sitting at the computer i'm at my desk okay they're ramping back down all right let's try again here okay [Music] excellent uh we're going to continue uh the fan stopped over there i i think it needs it's in some kind of bad pid loop and it mostly was it defaulted to the optimal fan profile which changes the speed more and once i was able to set it to one of the other options it stabilized out the cpu temps were now also readable by ipmi and the thermal warnings were gone this finally gave me a chance to play around with remote management features that i hadn't been able to use on the previous board okay back over here because this is actually working now uh no more horrendous loud beeps uh i'm pretty much ready to be able to put this back into the server chassis but i thought i'd show this off because it's finally working this is uh one of the the main reason that um cody who sent me the motherboard was so kind to send me another one ipmi which is super micros remote management interface is now working and there's an app for my phone that allows me to go ahead and control everything from here which is so nice there's also a web interface that's what i was just showing you with the screen capture um now i gotta remember how to use this uh there we go so uh what i can do is i can control the fans from here for example i can set it to crazy uh high end actually let's set it to full here that'll ramp up more perfect as i mentioned this board does not have a dynamic fan uh control so it doesn't manage like the fan speed compared to the cpu temps it's just not how server board operates they want like a specific amount of cooling or not uh i think it's possible from linux to set more granular controls of the fans um using some magic bites to specific registers or whatever it's been a while since i've looked into it but i'm pretty sure you can uh and i know there's been some projects trying to do it i haven't looked into them too much so i'm not sure but it's probably going to be worth it to try it because the low speed on these fans is really great that didn't work out like i thought it would but i'll come back to that later i was still excited to see the remote management working so well that i forgot to address the next steps i'll be taking here now that the board had proven to work and i had fixed the issues with it i was ready to put the rest of the fans in the case and start reassembling it i had high hopes when i first took the system apart that it would go back together more cleanly but there really isn't any way to well route the cables in this chassis just getting them out of the way of the air is as good as it gets for the next few days after this though i ran into more problems relating to the fans that threw a pretty large wrench into my initial plans for this project ah coffee is a delicious narrative device for showing the passage of time i mean hello it's been a few days and obviously the server is working now that took a little bit of effort now the fans so here's the thing it's on and it's not that loud but it's slightly warm and i have to decide what the trade-off is going to be so at this moment it's running um everything is acceptable but on the high end of acceptable and it could not be so this server is weird when it comes to fans it does not have a configurable fan profile and the way that it regulates the fan speed in relation to the temperature is kind of an on or off situation so this server has i'm gonna just simplify it to say has three fan profiles um they should have more and it should be a little more complicated but this board is kind of strange i'll get into that in a moment so this is the standard fan profile now as i mentioned ipmi actually works which is the remote management interface so i'm going to change this stuff from my phone over an app which is awesome um and we can see how it works here so this is standard which is about 30 speed at idle and then when it gets hot it'll ramp up the fans to i think 100 if it needs to there is a intermediary i'm gonna say mode called heavy io speed which should split the fans between 30 and 75 percent but doesn't again i'll get into that um and here's what that sounds like now for a server running at a higher fan setting that's actually pretty good i don't think i could just sit here having a conversation without having my ears bleed and trying to run away from this screaming so it's really not that bad but this is kind of what it takes to keep it at temperatures i would like although not quite um i'll get into that in a moment there is one other fan speed that's just full blast and i'll go ahead and set it to it now no real change so here's the thing this uh full speed here is more than enough it'll actually drop the temps really fast in there um and that would be fine and i might end up running it at that if i give up and stuff it back in the server closet which is probably what's gonna happen um but it should be possible because you set the fan speeds more intelligently i just can't figure out how so the server or super micro boards in general are supposed to have um at least for this series which is x8 x9 and x10 uh two fan zones one for the cpu and one for peripherals um the peripheral fans are supposed to be or is it the cpu fans are supposed to be designated with letters and the peripheral fans are supposed to be designated with numbers this board only has numbered fan headers there is no fan header a and b in here so everything is just in one zone now this is important because if i had been able to get different zones working i might have been able to move the fans around using extensions and splitters and stuff and got the fans that are running the hard drives or cooling the hard drives which are the parts that are heating overheating the most to run a little bit higher with one of the different settings um but not everything else it may not have been as loud but because everything is on one zone uh my options are 30 75 and 100 and the difference between 75 100 is just non-existent so really it's 30 which is too low or 100 i would really really like to be able to test 50 but i can't now there is supposedly a way of adjusting the fan speed over ipmi tool which is a linux command line program that you can run to send settings to ipmi on the board some people have figured out a way of sending raw fan speed controls to ipmi however they do it by zone and again as far as i can tell this board and x9drd if has one zone and i can't get any of the commands i can find online to work for it so i haven't been able to set a manual fan speed on here if i could figure out how to do that if one of you knows how to do that on an x9d or dif please leave a comment because that might change how i can use the server as it is now it's gonna have to go back in the server closet because it's just too loud at 75 and there's nothing in between that i can use so it's going to have to be that now what i'm saying that it's slightly overheated when i run it at 30 percent on standard the hard drives get to 35 to 40 c which is like right there at that point where you don't really want them to be but not too hot and then the cpus are between 40 and 45 c i would say it's a bigger spread because there's 16 cores and high variability but around 40 to 45 which is i've had haswell chips that idled at that speed but this can be lower so it's just annoying gpu just rock solid i just checked the temps it was at 38c at 30 percent it's got its own fans it manages its own temperatures it's doing whatever it needs to to keep cool so it's not a problem it's just those really the hard drives that i don't want running too hot because this is primarily a storage server and i don't want to lose data so yeah fans are complicated so the fans were a mixed bag in the end but the motherboard swap has been undeniably better since all of the ipmi features are now working the last thing i wanted to check out was getting my editing software reinstalled and running on the server which actually went smoother than i'd ever experienced before on linux which was also a little annoying because i've been having a bug with linux resolve on my desktop that causes it to freeze the ui when i do some things and now i know that's some sort of hardware incompatibility that i'm going to have to get help from blackmagic with but for the first time in a long time now i can use my server for remote rendering again now that it's updated to resolve 17. that new version also introduced a completely new proxy feature that works even better with how i built my server so this has been a huge improvement in my editing setup overall with that there are only a few more loose ends i need to cover so final thoughts on the server upgrades the motherboard replacement went absolutely perfectly uh i have all ipmi features it has a functional vga output the vga remote capture for the kvm over the web interface works perfectly it's just my fault that i have another gpu in there and i'll eventually figure out the xor configuration files to use this as i want so that's just a time thing the fan upgrade i mean again i had to replace the fans because the one was bad so it just was a requirement i didn't have to spend as much as i did on full on noxuas that were super quiet only to find that they weren't as quiet as i'd hoped so i'm going to call that one a bit of a wash um because i needed new fans and these are quieter matter of fact let's go ahead and do a comparison between the original fans that were i don't actually know what they were set to i'm going to assume standard um and then these fans in both standard and full so you can see what the volumes are like because even at its loudest here it's still much quieter than it was before [Music] so yeah even though it's not totally silent and it's not going to be able to be next to my desk while i'm editing as nice as that would be because then it would be really short cable runs and everything would just be a lot easier to work with and then i would have easy access to the gui uh it's gonna be fine in the server closet and it's gonna be way quieter than it was before just i you may not even have to like close the doors and all the stuff i have like foam padding in front of the doors try and make it as quiet as possible i can probably reduce a lot of that down now so that is going to be nice but it's not quiet enough to keep around where people are all the time all right well if you enjoyed this video you may want to subscribe i don't know where the next server video is going to be because it's kind of done now but i will eventually be doing a full linux editing workflow video i just have to get that one hanging problem fixed with resolve which is going to require installing centos and getting official blackmagic support so that's gonna be fun but once that's done there'll be a video on that if you want to support the channel i am on patreon but for now that is it and i will see you next time you

Channel: Tech Tangents

Views: 20,657

Keywords:

Id: wB12N2HjENg

Length: 36min 54sec (2214 seconds)

Published: Fri Apr 23 2021