This is the million dollar PC, a very expensive collection of
servers that are linked together with an incredible piece of software called Weka that allows them
to act like one giant storage server with the ability to read speeds measured in the one
hundreds of gigabytes per second. Watch this. And... Damn, boy! Wait a second, that's a
little faster than it used to be. That's because the million-dollar PC has glowed up a little bit. We've scaled from six to nine storage machines, giving us a whopping 2.2 petabytes of raw, Kioxia-powered PCIe Gen
4 storage. It's almost as much storage as we have on hard drives in our mother
vault, but orders of magnitude faster, which means it's also really power-
hungry, which means it's also really hot. Even with three tons of air conditioning in here, it's still 40 flipping degrees, which is terrible for hardware, and terrible for any humans who need to work in here. Now, we could add more air conditioning, but that would be even
more expensive than this already is. You know what's not expensive? The Not One. But six APC tower coolers that we inherited for free from a local
data center. That's right, folks. We are doing a whole server room water
cooling, and this time, I swear we are going to do it right. So, where are the plumbers? Well, we're not really doing it right today. We'll do it right
later. That's fine, right? We're just going to DIY. That is not what I had in mind when I sold this to our sponsor. Build Redux. They make it easy to get into the world of PC gaming. Just pick your games, see the performance, and the pros at Build
Redux will build the PC for you. Visit Build Redux at the link down below. Now under normal circumstances, you would run chilled water in the unit in your data center and use
that chilled water to cool the air, but chilling the water costs money, and that's what we're trying to
avoid. Right. So, since we also spend money heating our warehouse, and we have so
many of these things kicking around anyway, we were thinking, why not cool the server room and heat the warehouse in the winter, turning this one into less of a cooling tower and more of a heating tower. Yeah, it was designed for that, right? I mean, no. This should work great. Like in theory, this is a mint plan. In practice, well, I guess we're about to find out. Yeah, in practice, we have a giant cooling tower on a forklift on a not storage rated mezzanine. Water's not heavy. Three, two, one. Oh, okay, yeah, that feels great. Where do we want it exactly? In theory, it blows hot out that side. Mm-hmm. So probably pointing that way. Sure, that might not be intuitive because this is obviously the front of the unit, so you would think that's where it draws air in. But in a data center deployment, these cooling towers sit in between the servers and servers will typically pull cold air in from the cold aisle in front of them. And then heat up the air and blow it out the back. So by having these towers in between your servers, what happens is that hot air comes out, gets sucked into the cooling tower, and then gets blown through this radiator in the middle, blown out the front, and sucked back into the server. Let's put it in the corner pointing that way. Hey, Jake. Yeah, what? I figured out what one of the things we thought might be a temp probe before is. What?
It must have gotten jostled loose at some point, but these guys right here, they have little floats on them. Oh, they're like the water level. They're for an excess condensation meter. Yeah, when you're running chilled water through them, they can start to condensate pretty aggressively. We're not doing that, so it should be fine. Yeah, I saw the drip trays before, and I also saw that there was rust everywhere in the bottom of the units, but I didn't realize that that was by design. As cool as it is, though, that this unit is more intact, it also means that we haven't modified the fittings yet, so. Yeah. Well, you did such a great job last time. So, you wanna have a chance to try to do such a great job too. Oh, okay. Boom, uno reverse. See you later. Before we solder on those fittings, I wanna give you guys a bit of a background. The tubing inside the unit is one inch, which is a pretty standard size, which is nice. Except what I found out, getting one inch tubing, it's not that common. And also, all of the fittings are very expensive. And since this is like our DIY approach. Even though my original plan was to use garden hoses. I found out garden hoses are really expensive. So we might as well use the stuff we're gonna use in the long term. And since we're gonna have to take it down anyways, we're gonna use shark bites. And shark bites are really cool. They're basically like you push the tubing in and that's all you do. It's now a fixed, it's warrantied for 25 years. It's sick. They're just really, really expensive. That smells delicious. It was taking too long. A one and a two. You gotta make sure it's clean before you solder. Otherwise the solder doesn't stick. At least that's what plumbing TikTok told me. Naturally, this is gonna be what you'd call a closed loop system, meaning it's filled with a certain amount of water and closed off, but only to a point. When you have radiant heating in your house, for instance, there is some amount of evaporative loss. So you fill the coolant up and over time, slowly, you lose some of it. The way that they get around that is by using a check valve that's basically just hooked up to your house's water pressure, which automatically, as some evaporates, a little bit comes in to
fill. I didn't really want to do that because I want to do it today, and
I'm not a plumber. But fortunately, our mechanical room is right below us, and it has
a hose bib. So we were like, why don't we just hook the hose bib up to the
tank? But then if you just turn it on, the tank's going to overflow and
explode and whatever. It's like, how can we fill a tank? I could, like, design a custom system. Or I could just buy a toilet fill valve. We have this tank from McMaster Car, which we're going to add fittings to with these fancy bulkhead fittings. But in the meantime, this has got to come out the bottom, which means we need to drill a hole. It just knew the job was done. I bet you this stuff is tasty. Just apply a flux sparingly. Oh, oops. We're putting the threaded ones on here. We'll solder those and then twist the female shark bites on. And then the tubing can just go like that, in theory. Aw, she's mint, bud. It occurs to me we never tested this thing. I mean, we have spare parts. It would probably be faster just to pull another one up here, right, Jamie? Oh yeah, absolutely. Luckily, there just so happens to be the exact plug we need, right here. 20 amp, 120 volt. We got some adapters and extension cords from Infinite Cables, thanks guys. Pluggy plug. Hey! You hear that? That's the condensation pump running at the start. That's not too loud, right? That's great. It's actually not that heavy when there's nothing in it. As he struggles to move it. What's the worst that could happen? I drop the thing? Mint! Let's go over this one more time. Pump right below the reservoir, that way it gets fed directly. We had a problem trying to prime it at Linus' house with the pool, trying to avoid that. Then it will go down to the cooler, come all the way back up, blah, blah, blah, into our heater, what I'm calling it, and then out the heater into a filter before the reservoir, so we catch all the gunk before the reservoir, reservoir back to the pump, and the vicious cycle continues. Cool. Are we expecting a lot of gunk? No. I feel like we should've also filtered the water coming in. We'll put a valve down here so that when the water comes in well at the start We can close this and the water will go into the reservoir which
will make it easier to bleed then once the system is running We just open this valve and most of the water will just go directly
into the pump and kind of ignore the reservoir when it's Coming back up here that way We don't have like the full force of the pump just like gushing
into the reservoir constantly making a bunch of noise I wish we had like straps this makes this much easier unless we use
the LTT cable management things to this metal pipe Oh hell yeah the reason that we're putting valves here and actually at every
piece is A so we can service a piece individually you shut off the Valves and then we can take it out and swap it or change it or
replace it Whatever we're doing and B because it's like I said before this is all gonna get redone Properly and I want it to be easy for him to just take apart Whoa, it's like it's designed for it. It's mint mail a tea And a valve. Yeah, this isn't gonna leak. No, it's gonna be great.
They're shark bites Sick. Before we had this grand idea of water cooling the server
room we tried to mitigate this problem by adding a duct and a fan which was
pretty easy to do because right where we are right here is about 10 feet from the
server room and there's just a hallway so we added like this insulated duct
between and a big fan and it did help but ultimately once it gets up to
equilibrium it wasn't enough. Hey, it's through. Oh, that wasn't even that hard. Huh, huh? Mint. We forgot about the filter. I realize Alex is putting this all together. I'm like, "Oh yeah, where are we putting the filter?" Damn it, right where Alex is working. Yep. Whoa. Brother. Brother. Did they sell me the... Mother F****. Oh my God. Why didn't I check this? No filter it is. Since the long-term plan is to rip the rads out of these giant
units and just stick it right on the back of the rack so it captures the heat directly, I don't care
that much about where this goes since it's probably only going to be here for a couple
months later. So we're thinking we just leave it up against the back here. That way the hot air
comes out, gets sucked in on the back, and shoots out cool air in the front. And because of that,
we're thinking we just leave the back door off, and then I can just pull this tubing kind of right
in there. Beautiful. Look at that. That's mint. Good job, Alex. I'm excited to see if it
leaks or not. The great thing is we won't know until it's far too late on the bottom side. Yeah. Shut
it off. Here's the problem we're running into. This tank is thick down the middle where the seam is,
and when this is sitting in here, it looks good like that, except when you tighten it down... That extra material pushes it out on an angle, and now the seals can't seal. I walk back into this project and I'm immediately told, "Well, it leaked." Whoa,
whoa, whoa. Be careful. There's fittings on the bottom. Well, we have a fix for the leak, and it's
glue. No, not that. Not that. It's the, the UHU. This stuff is like $40 a tube. It does not
leak. It's a nice non-hardening, so it stays kind of plasticky and goopy solution. Like a silicone.
Yeah. I wasn't able to do it exactly how I was hoping because it's still kind of cockeyed, but I
don't think it's going to leak. I put, I put a lot in there, but I have a job for you. Cool. I
spent some of your money and got the wrong thing, so we have to do it again. If I had a dollar for every
time he said that, I would be able to afford all of these mistakes. What are we building? A
filter housing. They sold me the wrong size of PVC, even though I was like, "I just need to connect
these two things," and they were like, "Okay, we're going to give you one inch PVC," because it's
one inch. Well, it's supposed to be inch and a quarter. Cool. So I have 20 feet of one inch PVC that I have no use for. And then when we went to the store, they were like, "Okay, we can
tell you this. It'll probably be long enough." So you have to cut the threads off. And then cut it in half. And hopefully that is like enough. Oh man, that's gonna be, you have one shot at this. Don't screw it up. And this is my saw? Yeah, look, this is what the professionals told us to buy. They might've told us the wrong thing last time, but here we are again. Surely that wouldn't happen twice. No. Got the clamps. Not really what this is for, but it's working. I cut off part of the threads. I sanded down the little nubbins. Now they fit in here, which is why I was doing that, by the way. Really? Oh, wait. Oh, balls. We have to cut this down the middle, too. Okay, well, I have to go do a conference call, so. Yeah, that's what I mean. You only got one thing done, and you didn't even get it done. I don't know if you're getting your minutely bonus. Minutely bonus. Bloop, bloop, bloop, bloop, bloop, bloop, bloop, bloop. Look at that. Worky. Oh, nice. Yeah. I didn't know that this glue set up in 10 minutes. There's your filter. Perfect. Damn it. I looked in the manual for the cooling tower things and I did randomly stumble upon them mentioning a filter and they said a 20 mesh. And I think this is a hundred. So it's a lot finer. Should be fine. As long as our pump has enough power for it, it doesn't really matter. I mean, it's not that fine. Yeah. And also it's still like meant for one inch tubing. So it should be fine. Now we just have to find out where we can put it. Now we just have to pose with it. Like not everyone's a poser. Can you walk me through what's going on here a little bit because I missed the part where you guys drilled holes in the building. We've got the cooling tower over there. We've got the tubes coming off of that. And we already have the tubes in the server room already hooked up. We had previously cut a hole for our ducting solution. So we just stuffed the tubes down the duct. That goes through the hallway there into the server room. Oh, wow. And these awesome cable management things, the magnet is so strong that it just, it's kind of perfect. In terms of pump, this is not our end game. This is like a cast iron pump and we're using OxiGuard tubing, which it limits the amount of oxygen. It's probably fine. Look, your house has had probably the same cast iron pump in it for like 20 years. And it's fine, but it isn't the end game. I picked up a stainless version of the one step up from this pump. This one I think does around 20 gallons per minute. And the big boy that we're getting does around 40. Wow. Which is. A lot of gallons per minute. A lot of gallons per minute. Okay, are we ready to hook this up? In theory, yeah. I mean. Let's do it. We tried before, so let's try again. Hold on, before you do that, that tube is plugged into the wrong thing on the top of that. Oh, did you guys do anything right on the first attempt or? That is a excellent question. And we decided to not mount the filter up there. He always chirps me for doing things like kind of the easy way and then fixing it later. And he's literally just like purposely doing that right now. Well, we're just not gonna fix it later. It's fine. I'm gonna fix it later. What? No, don't waste time on that. It's fine. I'm not the one that wasted time. Don't move it. It doesn't matter. You told me it doesn't matter. I'm gonna send you a video of me moving it later. No, that is gonna be a huge waste of time 'cause you'll have to disconnect the tubes in order to- You could have just fixed it right now. Let's put the tubes in, come on. I'm coming back here with a rotary tool. I'm actually going to shave off the tops of these screws. Let's give ourselves a little bit of breathing room. That's a lot of breathing room. That's fine. I like to breathe a lot. We know. I plan to keep doing it. Easy peasy, by the way, I found an easier way to do this. Just take the white thing out first. Oh, okay. Okay, this is bothering me. Do you hear that right now? Is everyone hearing that? I don't know what you're talking about. You hear what he just said? It's bothering me. Editor's note, cut out this entire storyline. Okay. It's on the wall. Filtered. What? I dropped one of the little shark bite white thingies and it fell down the hole. Hey Linus! Yo! You suck! Oh my God, Jake, did you turn the water on? No. Why is it wet? I don't know! Where's the water shut off? Is this valve broken? I was dumping water down the hall. Oh my God, you suck. Do you want me to turn the water on? Okay, that actually does pour down into an electrical box. So you really shouldn't do that. Oh, what, really? Yeah, I honestly have no idea where the shark bite thing went. Ta-da! What? Oh, she's on. No, it's in, it's in. Somebody needs to go in the server room and watch. Okay. In case that leaks, because if that starts leaking, it's gonna be a while before it all stops leaking. Dick? Ah, a dick, I get it. No, stop, you guys. No, he's A, his first name is A. My login in high school was A.Dick. Okay, and then you want it one click apparently, right? 7/8 inch ball cock. Let's turn this thing on. Yeah, go for it, see you later. Okay, it's on. I got water coming, that's good. Do it, full send, let's go. I can't believe this tank is not leaking. Are you ready, Mr. Dick? I am ready. It's coming. Whoa. It self-fills. How's it looking? How do you do this at my house? This is awesome. No idea, it's actually great. You know, V2, right? Is it fine? Yeah. Like everything's fine down there? Jake, we have a leak. Oh God. Where? On the return fitting. No, we got it, we got it. I tell you guys, standing right at the back of the air output of these servers, it's exhausting. Alex, Alex, Alex. He may be a dick, but he's our dick. This is the world's slowest leak at this point. Ready, one more, and nice. Note that every fitting I did is not leaking. This is the one you did. I didn't do that one. Yeah, you did, you stuck the tote. I don't know what you're talking about. We managed to fix our leaks. I replaced the fittings down here on both sides. I think we just used too much Teflon tape the first time we hooked it up, and it kind of just like bunged up the threads. And then upstairs, I just cut the peck, sanded it down, stuck it back in, and everything's good. No more leaks. Now I've got the million dollar PC hooked up and running again, and I want to test something before Linus gets here. One of the big things you want to do in a setup like this, especially if you were doing this in a data center, is contain the hot air. So we're going to make ourselves a little bit of a hot aisle with... some cardboard. Right now it says 31-ish degrees on the intake.
Maybe it was 31.1, 31.2. Let's see if it goes up any higher by just adding the cardboard. I
want to just kind of proof of concept before I spend the time doing a better job of
this. Obviously, still a giant gap here, but let's see if it makes a difference. Oh,
31.1. 31.3, 31.4, it's going up, .5, .6. Okay, maybe it's only like half a degree, but half a degree is important because the whole game here is temperature differential. The larger we can make the difference in temperature on this side to the temperature out in the warehouse, the better this system will work. Hey, that's not bad. We have our hot aisle, which is going through the cooler, and the
rest of the server room is our cold aisle. It's not a perfect science, but it's better than when it was open. Oh, oh God. Look, I wanted optimal circumstances for our very scientific test. It's definitely hot. How do you like it? I have so many questions. That's great. It's hot in here, hey? It's really hot in here. But would you believe me if I told you the air conditioning's off? Really? It's 32 degrees in here. Don't get me wrong. It is hot, but the AC is off off. And this has been running for like hours. That's... Actually kind of cool. So look at the UPS, we're drawing 4.95 kilowatts, except no, we're not. Half of these servers are hooked up to the mains power. So we're probably drawing more like, probably like eight to 10 kilowatts. However, the second we turn the AC on, aren't we going to be taking cold air that's been heated up a little and running it through our heat exchanger, potentially cooling the warehouse? Look, I don't know if it's gonna get that cold. I don't know if you've stood behind these things at all. They're hot. It's quite hot. The exit temperature of these servers is probably like 40 degrees, even with air conditioned air coming through it. But you are right, it will be colder. That being said. Before, when these were actually running, with just the air
conditioning, it was 40 degrees in here. I think we should turn that off and see how hot it gets in here. Okay. Because it's been 32 for over an hour. This is a cool software called Checkmk, it's a monitoring tool
that's open source, very awesome. We've hooked up the systems with SNMP, which is a monitoring protocol, and you can see... the coolant coming in is 28 degrees. That's coming from the
warehouse. Yes. After it's been cooled with air that's around 21 degrees, and then leaving here it's 32
degrees. That's a four degree difference. That's a... lot of heat. It doesn't sound like a crazy number. But at the fluid
flow rate that we've got, that's a freak ton of heat. Yeah, clearly we have a problem of it not quite getting cooled over there as much. But I had a brilliant idea. We have six of the bloody things. Let's just put another up there. We could. Why not? Or, hear me out, we bury a coil in the parking lot. Oh, I'm totally down for that. Give me the approval. I will find a
contractor today. That is not, that's above my pay grade these days. Oh, I gotta ask Taren? Yes. Hi. Money, free cooling. Linus said it was a good idea. Anyway, yeah,
we could throw another one up there for sure. Like pretty easily. We just actually need two T's. That's it.
Interestingly too, the return versus supply air is a three degree difference right now. So
the air coming into the back is almost 40 degrees, 37. And then the air coming out of the front is 33.6. They are still going up a bit, but like I said, it's been at 32 for an hour. I imagine if we left this overnight, we probably come back and it might be like 34 or 35 in here. But considering we're not air conditioning this at all. Let's see what happens if we turn it off. Oh God. That is hot. I don't think the air conditioning is gonna make that much of a difference in terms of the air coming out of here. The metal on the back of this rack is up to 46 degrees. The air coming out of the back of this power supply is as high as 60 degrees. Oh, 76. It's 33.2 now. This may not be safe for this hardware for much longer. It's 33.3 now. It's gone up a full degree in like three minutes. It's getting warmer, 34.1. I'm getting quite uncomfortable. I think what I wanna see now is if the air conditioner can actually work cooperatively with the heat exchange. To get it down to like normal temperature in here? Yeah. Okay. All right. Let's do it. So our baseline is when we started this project, it was hitting about 40 degrees in here with just the air conditioner running. Right. If we can beat that, it means that somehow through the magic of thermodynamics, they are working together. 34 and a half, plug the goddamn thing back in. Oh. Sweet relief. So our peak was 42 with the AC off and now it's come down to 37, 37.8. And up here is equally as mint. You like my sound deadening? Yeah, I mean, it'll do something. Not much in this configuration. That's what I was going for is something. Something. Long-term we get sheet of plywood and attach the rock wall on the inside of it to kind of make a baffle. If I stand here, it's pretty loud. But if I go behind this thing, it's a pretty substantial difference. We just have to create kind of a sound blocking maze that air can move through, but blocks sound from reaching people below us. And it's also at like 80% fan right now. If we put a second one up here and run them like half the speed, it should be tolerable. All right. Wait, we should look at the filter. Oh. Is there shmoo in there? Not too bad. And we're back. So, not as good as it was before. No, it says 1.800 watts on the readout, which-- Oh no. Which is a lot given the amount of power that we're putting into it. What, one pump and some fans? Huh? 22 degrees in here right now. Supply versus return air is still like a delta of more than two degrees. That's solid. And interestingly, the cooling temperature is lower, but what I realized, look at this. The fan speed is lower. Oh! What I also realized after looking at this is that there's a valve. I remember when you guys did the Fred River cooling, there's a
manual valve, right? That's like a bypass valve. And you guys were like, "Oh, you're supposed to close it so it
doesn't bypass." Actually, you're supposed to leave it open because there's a mixing
valve in the unit. For your application, actually, probably just shut, it doesn't
matter. But the thing can dynamically adjust how much water goes through
the radiator and how much gets bypassed. I think to maintain a delta, an ideal temperature. Yeah, it's totally fine in here now. It's working. I'm surprised it works this well. Well, the ducting was key. But the other big thing is I think that we were just on the edge of this air conditioning unit's capacity. So just taking the edge off, taking another thousand couple thousand watts out of here. And what's cool is because we've got it hooked up directly to the exhaust from this rack. Adding more to this rack is going to increase that inlet air temperature, which increases the efficiency of the thermal transfer of our heat exchanger. Mission accomplished. Yeah, I'm actually kind of shocked. I mean, two degrees cut off the exhaust temp doesn't sound like a lot. But that's like already outside of the server and we're talking two
degrees and a lot of air. Oh, I can't wait for part two. It's gonna be so much better when
this is directly on the rack. I'm thinking we put two on the mezzanine for like winter time. But we could also put one like outside something. We could have
valves and we could decide where we want to dump heat from this room. We could automate them pretty easily too. We could have electric
valves and a couple temp sensors and the raspi. So we can send the heat outside when it's hot in the warehouse and
send it inside when we want to heat the warehouse. And interestingly, we could put it in that mechanical room and
just, like, cut a hole in the wall and put a big thing. Yep, it's our mechanical room. Yeah, what's the landlord going to do about it? Screw that guy. And screw this segue to our sponsor. Squarespace. Creating your own website doesn't have to be difficult. And luckily with Squarespace, it isn't. Their all-in-one platform makes it easy to get your website up and running quickly, and designing with their fluid engine site builder is easy. It's so easy. Start with a template and customize the details to your heart's
desire with intuitive drag and drop elements on both desktop and mobile. You can also use their
asset library to manage all your files from one central hub and use them across the entire
Squarespace platform. Plus, with Squarespace's analytic insights, you can see what's working
well and what needs a little TLC. You know, and if you need help, Squarespace has got your back with
helpful guides and a 24/7 support team. They do not sleep. Head to squarespace.com/ltt and get 10% off today. If you guys enjoyed this video, why not check out the time we did
this at home with radiators from PT Cruisers. Oh, yeah. Yeah, this would have worked a lot better. Yeah, but
those work pretty good. You could just take one of these home. No, they work great. I'm not changing anything. Oh, yeah. Okay ducting I could duct to the PT Cruiser radiators, but that's all I would change Oh my god We're getting that Elegoo like one meter by one meter by one meter 3D printer and we have a 3D scanner They could 3D scan the back of your rack and make like a duct
shroud look He's actually like considering it right now. You're thinking your
board, but we could do it in here Oh, yeah, totally. We could take the AC straight to the front of
this rack. We want some over here, too This'll be okay. I guess if the ambient is low enough, it's
probably fine. It'll be fine. Yeah, okay.
