I'll make my own heatsink - Just add Blackjack

for those of you who like my cooling experiment videos this is going to be one for the ages [Music] welcome back to craft computing everyone as always i'm jeff and obviously this is going to be a little bit of a different video starting with this this is the tesla m60 it is a dual gpu card that was graciously donated to the channel a couple of months ago okay about a year ago but it has one major issue with using it and that is that it has these weird janky custom water blocks on it that i don't trust to put into a server now the tesla m60 originally shipped with a passive heatsink just like all other tesla gpus and i really don't want to just track one of those down because i'm going to be left with the same exact cooling problems that all of my other tesla gpus have so i figured why don't i swap out the custom water blocks for something a little bit more streamlined this right here is the ig cooling is-30 and it is a super low profile cpu cooler with four heat pipes and a 92 millimeter fan if you're new to the channel welcome and i'm going to answer your first question right now no this is not going to be done with zip ties while i am a fan of some of the zip tie engineering that happens in this community i tend to over engineer my solutions so today we are going to use my brand new cnc machine to make some custom aluminum brackets to mount the heatsink onto the tesla m60 but wait that's not all because that's not all that needs cooled on the video card i also have these heatsinks right here that i'm hoping to machine down to fit into the center section to cool down the vrms but first things first i need a trip to the hardware store so i can get all the various bits and bobs that i'll need to actually pull this off so let's get started well i was gonna vlog the trip but then i realized home depot probably wouldn't like me filming inside the store and i didn't want to deal with that on top of all the anti-masters that are around my area so i decided to just go to the hardware store and buy the stuff that i needed so first and foremost is this aluminum flat stock this is uh half inch by 1 16 inch thick and should be absolutely perfect to make the brackets for the is-30 heat sink now this is the underside of that is-30 and you can see there's a couple holes on each side and those are normally used to attach to the brackets that actually bolt to the motherboard uh i'm gonna use this aluminum stock to create my own bracket and get the proper spacing to mount onto the tesla m60 now unfortunately half inch is just a little bit too wide so i will have to mill down a little bit of the side of this but that shouldn't be a problem for my cnc machine the second thing i needed was just some quarter inch ply stock so these are two foot by two foot sheets that i'm going to cut down and create myself both a spoil board and a jig to mount my aluminum stock to so we can repeatedly mill out these aluminum bars because i need to make eight of these i think yes eight because there's two brackets per cooler and there's two coolers per gpu eight now i've already done a whole bunch of pre-planning for this project and the first thing i made was this this is actually the bracket that i need to make on the cnc machine obviously not out of petg but out of aluminum so the spacing is correct i have mounted it to the heatsink and i have also mounted this to the pcb on the m60 and it is a perfect fit so that's what we're going to recreate out of aluminum to do that repeatedly i made myself this little 3d printed jig that i'm going to fasten down onto the piece of plywood so we can put stock in the same orientation every single time and hopefully get pretty repeatable results and with that let's go ahead and get to milling [Music] my [Music] all right after all that work i have made the world's crappiest boat paddle i guess uh this is the jig that's gonna go onto the cnc machine to make sure that i get repeatable results for cutting out my brackets without trying this in the machine yet i will say i am pretty happy with it thus far uh these two screws down here have threaded inserts that are super glued onto the back and hold the aluminum stock very very flush to the board also this board is cut so it is 100 square with the cnc machine itself what i'm going to do is mill out the top section right here and then all i have to do is break off the little bit of remnants that will be left slide the aluminum stock forward and then tighten it back up and i should be able to cut multiple brackets in a row without needing to realign the machine in between takes i guess all that's left to do is see if this works one more thing that i didn't mention about the jig that i made is the 3d printed part has this little square cut out of it so i can actually home my machine based on where i want it to cut out the bracket also the top of it has enough slack so i can clamp down the top of the part as well as using these two screws on the bottom side so again hoping hoping hoping this is actually going to work all right now let's get it on the machine [Music] so it was definitely a valiant first effort and in fact uh the spacing on those turned out just about perfect uh this little dot right here is just a registration point that's not supposed to be drilled all the way through that's just so i know which side points in towards the heatsink a couple of things i did wrong number one my z-axis is not quite homed in the way it should be so it didn't quite poke through on any of these although they are maybe like five thou from actually poking through uh in fact three of these actually have like bumps on the bottom side where it tried to start poking through uh so i just need to set my z a little bit lower uh and i also homed this slightly further inside than i wanted to uh it was actually going to start cutting into the support on the opposite side and that's not what i wanted i wanted this to cut completely in the aluminum within the support bracket so i think i'm gonna have to just make a couple of quick tweaks and we'll try again [Music] all right well dang it attempt number two and we are so much closer you can see we broke through three of the four and uh actually the fourth hole there is a little bit of a pinhole of light there so pretty much i can take like a punch and knock that out uh the part did break loose pretty much how i wanted it to but unfortunately i don't have enough pressure on the top of it to hold it in place when the bottom support is removed uh so i'm gonna have to figure that out i might even just while the part is being machined i put a screw right through the middle of one of the larger holes and see if that'll hold it in place uh because once the holes are drilled there's plenty of room for me to do whatever i want on the outside of the part or on the inside of the part rather so i think that's what i'm going to do moving forward and we'll see how that goes a quick trip over to my grinder and i took care of all the leftover flashing from the failed part and what i got was a working part uh you can see that now this will mount right up to the bottom of that heatsink hold on there we go so i've got the two mounting holes for the heatsink itself and then i've got two holes that i can mount to the tesla m60 so i need to figure out how i can hold this down a little bit better but the part is good so let's go and make seven more of these time for a new plan i keep getting stuff that looks a lot like this and well i think it's a limitation both of the machine that i have as well as the cad software that i have uh i'm not able to generate tool paths and change a lot of parameters uh what i'm using right now is meshcam 8 which is a fairly expensive program it's about 250 dollars but you plug in an sto file and then it generates tool paths for you but you can't modify orders or anything like that you just have to accept what it gives you and what it's giving me right now is number one not quite working as well as i would need and number two my machine is causing a lot of chatter and so all i'm getting is these really rough cut edges and it's breaking away before i wanted it to in the program so new plan i have made myself this with the 3d printer this is a little template which i can put over the top of one of these bars which i can center punch out and then drill the proper holes on my drill press and i know half of you are probably screaming he has a drill press right there why isn't he using that the answer is because i didn't want to i wanted to try out something on my new cnc machine and unfortunately this was just a swing and a miss so now we're going to punch these all manually and drill them out and hopefully still wind up with a fairly good part and secondly now it's time to start uh diving into the larger heatsinks that are gonna cool the vrm and those will be on the cnc machine so i guess we're starting over now i have cut all eight of my brackets and now it is off to the grinder and yes i understand the irony of using zip ties in this project [Music] a little bit of time on the drill press and i have all eight brackets cut and obviously it was a lot faster than using the cnc but i'm not going to get that machine finished look that i kind of wanted but these brackets are going to be between a heatsink and a pcb i'm not going to see them so i really don't care how they look anymore but now we move on to the part of the project that i've kind of been dreading and that is creating a custom heat sink out of an existing aluminum heat sink because you can't buy custom shapes of heat sinks in quantities of one or two so i'm gonna have to take this which is the closest thing that i could find which is a 40 millimeter by 100 millimeter by 18 millimeter tall aluminum heatsink and turn it into a heatsink that will fit properly onto the top of the tesla m60 now the heatsink is going to go right here in the middle between the two cpu coolers to cool off the vrm and it will be touching all of the vrms that are right down in here however there's this massive choke right in the middle that is taking up a fair amount of room on the pcb and i have to carve a space around that so i can mount the heatsink down and be able to touch the rest of the components number two is the only way that i can mount this heatsink to the pcb is drilling these three holes out and then threading them and screwing in from the bottom so i only have three of these heatsinks and i need to do two of these cards so i only get one practice shot of this it's not that these are outrageously expensive i just don't want to buy any more of them so i pretty much get one shot of this here's hoping [Music] um [Music] all right there we go there's the bottom of it straight off of the mill and i have to say that is not a bad finish at all now obviously this is still just a 300 cnc machine it's not the finest surface in the world but that is definitely not bad but the moment of truth is will this fit onto the tesla m60 so here we go okay slots over the choke let's check our holes you know on two out of three i am dead on the center hole is about a half mil up and a half mil right but i think it's close enough i'm gonna be able to use it that is a success and now for the part that i really haven't been looking forward to and that's modifying the top section that is the fins uh these fins are still too tall on either side of this heatsink to fit underneath the low profile cpu coolers that i'm using so i have to cut off about 15 millimeters worth of fins on each side of this heatsink now as you can probably see i tried to do this with the mill already and none of my bits wanted to do that there's just too much chatter because there's not enough material here so uh i'm gonna have to do this the old-fashioned way it is the next day and after a very long night of doing some more work off camera i finally have a heat sink that i think will fit onto the tesla m60 but let's go ahead and recap where we're at really quick because i know yesterday i got a little bit rambly at times starting with the mounting brackets that i made yesterday you'll notice i added a couple little standoffs here these are m 2.5 standoffs that are four millimeters tall which makes up for the difference in height between the pcb itself and the gpu die meaning i should have the proper standoff height next up was to turn this heatsink into something that looks a little bit closer to this so it would fit in between the two air coolers i also hollowed out an area on the back so it would fit over the choke and added some screw holes so we could actually mount it to the top of the pcb and i think it just might fit so at this point i don't think there's anything left to do but start putting this together [Music] and there we go that is one fully actively cooled tesla m60 dual gpu compute card but it doesn't quite look like a video card does it i have one more surprise at the very least we need to put the i o shield back on so we can actually mount this into a pc properly but i figured what's a video card without a shroud so i custom 3d printed this two-tone nvidia shroud that will also cover the rear power connector and keep everything safe [Music] how sweet is that and finally we are ready to actually test this thing and see if all of that work is actually able to keep this card cool but more importantly now that the power tools are all put away i can finally have a beer now i know the question on everyone's mind for probably this entire video has been why am i going through so much effort to cool the tesla m60s because i gave them a fairly unfavorable review about a year ago there are a couple reasons that this card is enticing to me while no they did not perform up to my expectations when using official nvidia vgpu drivers and licensing i knew there was probably still some meat on the hook here you see the main problem that i ran into was a power limit and how nvidia encoding was handled so while the actual rendering performance was actually pretty good on these cards the encoding stream that i would use to access via parsec or some other method didn't have enough power or resources available to it so the end experience to a remote client was pretty much unusable but what if i use vgpu methods that are not necessarily sanctioned by nvidia like telling the virtual machine that it's actually running a quadro m6000 that is where things get a little bit more interesting last month when i consolidated all of my virtual machines down to a single giant epic server i was planning on using a gtx 1080 for vgpu i wanted plex to be able to have hardware encoding as well as run a couple virtual gaming machines and have my editor with a dedicated vm to log into to edit my footage remotely with the gtx 1080 only having eight gigabytes of video memory on board i ran into the same issue that i did with previous vgpu experiments that is you cannot mix and match vgpu profiles with different amounts of video memory so if i have a single eight gigabyte profile that obviously leaves zero left for anything else but if i wanted to give my video editor four gigabytes of video memory that means i can only have another single 4 gigabyte profile active now i don't know if you know this but plex really doesn't need 4 gigabytes of video memory for nvenc that means that any extra overhead i have after assigning 4 gigs to my video editor is pretty much going to be wasted so what i really need is more than one video card in this system wait isn't the tesla m60 a dual gpu card with 2048 cuda cores and 8 gigabytes of video memory per gpu yes yes it is so now here's the new plan i can give my video editor a dedicated virtual machine with 8 gigabytes of video memory and 2048 cuda cores that he doesn't have to share i can also split the other gpu into four different parts with two gigabytes of video memory each one for plex and three for some 720p virtual gaming machines but all of that is moot if the cooling mod that i did doesn't actually perform so let's go ahead and test that out jumping on over to my proxmox server i have set up two brand new windows 10 virtual machines each of which has an 8 gigabyte vgpu profile one for each of my gpu cores in the tesla m60 first off let's go ahead and see where my idle temps are sitting at uh looks to be 33 and 35 degrees celsius so at least i know i have enough contact with the dye that it's not overheating just from being powered on so far so good go and start up these two virtual machines and we'll get to some testing each of these vms essentially has four cores and eight threads for my epic 7601 as well as 12 gigabytes of memory using parsec i'm just going to fire up heaven benchmark on both of these and see what the max temperature gets to and let's go ahead and tessellate to the extreme and might as well go to ultra quality 1080p full screen and run all right there's the first one running go ahead and exit out of that and we'll fire up the second one and do the same thing ultra quality and extreme tessellation 1080p full screen there we go we have both machines rendering so let's go ahead and see what our temperatures are doing and yes i do know what the watch command is i've been doing this for a while now now in this test i don't have any kind of frame rate cap running i'm letting the gpus boost to as high as they want to go and you can see both of them are running at around 99 utilization which is absolutely perfect when i run these in actual gaming vms i usually cap the fps at 60 because anything else is just wasted and actually takes power away from being able to run other virtual machines but so far so good we're sitting at around 70 degrees celsius on both of our gpu cores drawing around 140 watts each and i'm just gonna let this run until those temps max out so i'll see you in about 10 minutes or so but for you i guess it's gonna be and the testing is all done including a couple benchmarks that i ran inside of heaven just to verify that it was running at the right speed so as you can see we leveled out just a couple degrees above where we were sitting earlier with one of the gpus at 77 and the other sitting between 74 and 75 both of which still at 100 utilization and at 1177 megahertz now i will say i am cheating ever so slightly as this gpu is currently installed into my air conditioned server rack which means the ambient temperature is around 60 degrees fahrenheit or around 15 and a half degrees celsius so definitely not a normal use case however being able to sit below 80 degrees is still very very impressive now i don't think i would be able to run these cards at their full 150 watt tdp in the dead of summer when my rack does reach upwards of about 85 degrees celsius even with the air conditioner on however these cards are only drawing about 135 to 140 watts during this test so i think if i actually limited that down to 120 i wouldn't be sacrificing too much performance in lieu of being able to run these in 85 degree ambient temps so definitely not half bad for a result when i essentially ghetto mounted two 25 cpu coolers onto the top of this gpu now again i'm a little disappointed i couldn't get my cnc to quite cut out the brackets that i wanted instead i had to cut them out in a drill press and they wound up a little less clean but again i was able to mount the coolers so i'll chalk that up as a win what i can say is machining this center heatsink down and getting it to fit was an absolute pain and one of the more difficult things i've done on this channel uh now i will say i am very impressed with myself getting my cnc spacing correct on the first try including the pocket for the choke so this heatsink would actually mount down to the rest of the components but none of the tools that i had at my disposal wanted to cut through these aluminum fins very much less unsupported and so i had to take a hacksaw and spend about an hour on the heatsink cutting them down to size and then filing down the rest so it would fit under the two air coolers but it's kind of hard to argue with the results and i think this is where we're gonna end this video this was a heck of a lot of fun i can't remember the last time i number one spent more time on a project but number two had this much fun out in my garage being able to just work with my hands again now i know this project was pretty niche and pretty much only applies to me and my use case mainly because my tesla m60s came with these epoxied copper water blocks that i really didn't trust long term inside my server rack so i needed to come up with some solution but you can take some of these concepts and apply them to cards that you might own however do be cognizant of the tdp especially on the is-130 coolers they're only rated to between 100 and 125 watts even the tdp on the tesla m60 is 150 watts per gpu this is not a mod you're going to want to run on the tesla m40 for example which has a 275 watt tdp on the single die that's on that card however the tesla k80 might be another viable use case but the spacing is going to be a little bit different than what i used here in this video overall i have to say i'm pretty satisfied with the results here i now have two gpus inside of my server with eight gigabytes of video memory each that i can divvy out to various services and that makes me pretty happy if you like this video you know what to do and subscribe to craft computing if you haven't done so already follow me on twitter craft computing for daily shenanigans like this and if you like the content you see on this channel and want to help support me on what i do head on over to craftycomputing.store pick yourself up a pint glass hoodie or t-shirt and if you want to chat with myself or any of the other hosts from talking heads think about joining my patreon link is also down in the video description thank you all so much for watching this one and as always i will see you in the next video cheers guys [Music] today's beer is from breakside brewing in collaboration with barley brown's beer this is the wonder jack ipa i'm not sure what barley browns brought to the mix but i do know this is based on the breakside wanderlust which is a classic oregon ipa boy even from this distance i'm getting like orange peel and a little bit of gosh what is that it's almost gin like in the botanicals seriously yeah orange maybe a little like honeydew melon ah crisp very very crisp and look how clear that beer is i mean that's just crazy [Music]

Channel: Craft Computing

Views: 34,116

Keywords:

Id: qCakMV0vLjg

Length: 27min 41sec (1661 seconds)

Published: Tue Oct 05 2021