Data Center Tour & Technical Deep Dive into the Power, Data and Cooling Infrastructure!

now normally you don't get to film inside of data centers but I was able to get access to an abandoned one a few months ago that I did a video and you'll find that linked down below now that video got the attention of a much much larger modern data center my friends over at Deft well newly acquired friends because they watched that video and in that video I had asked hey can I film at your data center let me know reach out well Deft reached out and said you sure can we'll even give you a guided tour we're going to cover the power system we're going to cover the cooling systems we're going to cover the centrifugal UPS system and they gave me an absolute brain dump of knowledge that I was able to film record and assemble in this video cuz I knew my audience love it this is in no way sponsored this is just nerds being nerds completely geeking out about all the systems there I had so much fun on this tour and I so happy I could bring you along now there's a lot to cover so let's get [Music] started [Music] now our tour begins in the centrifugal UPS room because that was just fascinating to me that there's a spinning UPS instead of batteries but that room is actually the loudest room record in throughout this entire video the audio is still good you can still understand it and the first question that wasn't on camera that I asked is how long does it take to switch from a total loss of Municipal Power to these UPS's running before the generator start and well that's where we're going to start our tour well you said 16 seconds that's so 16 seconds is the amount of time it takes for one of the generators to go from off to 60 mph in a set so think of a semi- rig doing a/4 Mile Strip time it's doing it about 16 seconds wow so that's what's needed to be able to get the facility from not having power fed by the utility to being able to be self-sufficient on its own can range from anywhere from about 250 KW up to about 1.3 megaw in size okay and what happens is there's a large rotating drum inside there that's spitting magnetically levitated so that is able to have very little friction in a hermetically sealed container and what happens is when the power goes out the resistance in the motor windings slow that container down as it's generating power and it slowly spits down when the generators kick on and everything's hunky dory those same motor winds are now a motor and will spit it back up and get it going how long do how much maintenance has to they're pretty maintenance free I'm assuming they're pretty maintenance free but there is still a fairly rigorous schedule to it the floor here had to be laser leveled to make sure everything was nice and flat and then these things have to be calibrated maintained and then if there's an issue your on-site team needs to be certified to be able to do the maintenance on there again hermetically sealed is a little bit of a challenge to do on site but that's why you have redundant numbers of these in a data center so what we're looking at is the diesel generator backbone to a data center so power goes out this is ultimately what kicks on to be able to provide Power for the building they range in size from anywhere from about 500 KW up to about 2 and 1 half 3 megawatts a piece in this case this unit has four turbo uh turbos on it allowing it to generate a massive amount of horsepower but what's really cool about it is the level of redundancy that's built into this you have dual starter Motors so that if the starter motor has an issue there's a secondary one ready to go you have fuel filtration systems next to the generators ready to go you'll notice a raise barrier to be able to deal with fuel containment in casee there is an issue and then usually you would end up having what we call a day tank which is a small fuel storage onsite ready to go next to generators that's able to be able to provide that fuel immediately if they were to kick on in this situation you have uh the cold air will come in from this side of the building goes through get sucked through the radiator and then out the louvers on the back side that will all automatically open up at the point that these things are requested to turn on and again these things go from sitting still not on to full 60 MPH ear bleeding Rush uh in about 16 seconds if these come on run that way yes exit the room you'll see there are headphones for covering uh the ear protection uh by the door there but uh all automatic with uh like a giant essentially exactly that is really cool and then you have like you got the Dual filter system here we have just cool seeing these um being in Chicago is just on I was just on that I did that OT tour and it's got those giant diesel engines you know very similar sort of concept you've got redundant uh sorry filters for the radiator system oil system everything is designed to be easy to service and redundant so that no single component can take this unit offline and that is just an absolute massive Improvement to be able to have when you're working on a setup like this yeah and you have the it's kind of I see all the isolation to keep this from vibrating cuz vibration is the enemy enemy of data centers you got % so full suspension tray on these things that are load rated and uh everything has a little bit of a flex coupling so that it can shift a little bit and vibrate without causing like even where it connects to the exhaust where the everything so it cuz these things do have some movement when they start they're just so massive exactly and what you have here is the diesel engine so this is very recognizable but this back half is the component that is the actual electromagnet generator so it engages the clutch system this turns a whole bunch of windings in size and then this kick out uh usually typically 480 is 600 Vols some will go higher than that into a power distribution system and then there's going to be a regulator system on here that helps to maintain the RPMs so that the waveform of the power is nice and consistent yes cuz you do not uh connect a generator at off sync correct you want it in sync and you want the waveform to be 60 HZ cuz that's what the power is in the US so your generator needs to match that and if the generator revs down a little bit your hurt cycle is going to compress or expand depending on that yeah that keeping everything in sync is one of those things you don't you don't think about when you're just doing a switch at home or something small but as you scale it it's like a it's a two waves crashing of high voltage is going to shutter these things in a way that's actually very scary very much so uh so in a server when you deal with redundant power supplies what they're doing is bringing in two power supplies that convert AC to DC and then they connect the DC bus DC is a lot easier to do that with it's easier to balance but when you're dealing with AC it's a lot more of a precise dance that you have to do and a lot more equipment involved to make sure it's done right the first time yeah that phasing is just so it's it's over it's overlooked sometimes by people who are maybe not be familiar with it um but there's a few books on it they regarding this where they talk about how they have attacked facilities and lock just just the way they put them out of phase was enough to damage them definitely so when you're dealing with a building block of the size so some facilities range anywhere from a couple megawatts up to dozens of megawatts and when you're dealing with power grids of that size one generator isn't big enough to be able to handle what you need so you have busing or paralleling equipment that's done and if that first generator attaches to a bus or a paralleling piece of equipment and has a bad wave form for the sine wave then it can possibly set the standard and so when the second generator latches on it says hey I can't match that standard it's going to say nope not going to latch on I'm going to fail and you'll end up only having one of maybe your seven generators attach can't run an entire data center with 17th the critical power generation yeah and one thing I'll note it does not smell like diesel in here it doesn't stink there's no uh you could eat off these things yeah the everything in here is it may be hard to tell but it's really clean for being a diesel room that's got louvers that pull air in and everything else it's actually something I'm just kind of noticing especially after doing that little ubot tour you SM as would someone say burnt crayons everywhere if you've been around just in industrial diesel equipment and being in Detroit I go to a lot of manufacturing facilities so also cool to see these uh St Detroit Diesel on they've been fantastic these ones have been in operation for a number of years no trouble and they're just still kicking doing great very cool this area is the pump R for phase one and what you see is the Redundant pumps for cooling from this side of the chiller plant to the cooling Loops inside each of the rooms so the coolant goes through pumps out routes through uh it goes up and then actually goes down routes underneath goes into every single room so most of the rooms have their own independent chain to keep that kind of diversity going and keep it independent but it has to run coolant all the way through the perimeter of the room because each of the uh computer real Air Handlers they don't have compressors on them so they're only just big radiators that are taking that hot air pulling it through the radiator cooling it down with the coolant and then they push that coolant right back this way so these pumps are pushing all the way to almost that back half of the building and backed and what they do is cold water goes in hot water comes out and that hot water comes over to the centrifugal chillers here which then exchanges that heat and concentrates it to the evaporation system and the uh cooling towers so for an idea of scale your normal residential air conditioning unit for a house is usually anywhere from 3 to 5 tons each one of these green units here is 1,50 there's a redundant number of them here so that we can have one take over if we're at full load but we're not usually at full load so that makes it so that we maybe have two or three of them on depending on how much heat generation is on and they rotate the load across them to kind of balance the hours otherwise you would wear out half of the units and the other half would be almost brand new the cool ins side we have a million gallon water tank over in the corner there there and another million over on that side for the other phase those are like cold batteries so think of it this way it costs a lot of money during the day to cool things down right poers usually had a peak but at night it's a lot cheaper so what they can do is at night when the heat generation still static but the efficiency is super good they can cool down a million gallons of water drop it in temperature and then during the peak times of the day not use the centrifugal chillers as much but use the cold water to bring into the system to help kind of cool things down so they use that water as a heat transfer to warm it up instead of running all the other stuff there got it so now you're taking this million gallons of water you chill down and you're slowly heating it up during the most expensive times of the day allowing you to lower your power consumption the thermal efficiency is really interesting cuz there's there's a lot of thought put into it it's not just hey run until it's cool it's how do we we get the most efficient way to do this totally this half fiber coming into the area fiber cross connects we have a redundant side over there diversity so that if again you take a chainsaw in the middle of the rack here you're not going to take out both paths so we have equipment powered equipment on this side to be able to handle things like waves to other facilities so that's dark fiber where we're lighting it up and we can slice it up to as many 10 gig connections as we want across that fiber and I say 10 gig but really there's 40 and 100 Gig variants this sense 2 and we do that all across the board uh we've actually recently had a request for terabit per second requests going across so we're at those sort of scales where people are requesting that sort of connectivity terabit nice terabit per second yep so distribution to customer cabinet so this is stuff where each of these panels contains 144 fiber strands you need two strands for communication so we can handle a really nice plethora of customers and we've slowly been filling them up and adding on additional ones as customers needs have changed and this back haul you can facilitate all the way back to like on Prem s you can light up dark fiber from an office building within the area and be like hey here's where my data my servers aren't in the server room theying your data center exactly and yet it's coming across locally as if you were sitting on site with them yeah we see that very commonly done with Office Buildings for large organizations where you might want that between your primary data center your redundant or Dr Data Center and your office building and we can set up those Rings help manage it and make sure that you're going through diverse carriers too cuz sometimes carriers will go through the same manhole or same conduit and the back will be able to take both of them out in some scenarios copper distribution so anything from uh IP kbm so that's your keyboard video Mouse over a network connection really handy when you need to get into the bios of the server and it doesn't have the ability to do it built in we have those sort of services available then over here we start running into kind of copper and what's interesting is we dealt a lot with copper originally 14 years ago and it's shifted a lot to fiber so you can see as the infrastructure has changed out it's just all fiber it's fiber now we're able to reuse these bundles because we have a little bit of extra slack down there so when we do equipment upgrades we're not redoing everything from scratch but these cables all run to existing panels so when a technician is doing a new service turnup they're only working in a panel where they're connecting what's needed never are they working on the live devices I I think something you mentioned with IP KVM yes that fact that people don't realize there is Enterprise equipment that does not necessarily have ipmi 100% yeah this is the thing that happens and and when you run into a problem with it you know the other option is extend a keyboard and a crash cart to the server and sit in front of it so these things are really great solutions for else to be able to sit there and say Let's help you out with something yeah we actually have our own version of that using a Raspberry Pi which is uh the Raspberry Pi IP KVM service yes we've deployed a number of them because we can go and put it in their cabinet connect it and devote it to them and then when they're done with it we retrieve it so we have that as a service for our customers that is really cool so our juner switch equipment we're a juniper shop we're agnostic when it comes to customer equipment but for us our preference is junifer this is the third iteration of equipment that's been in these racks here so we've been able to go from Cisco to Juniper and from one version of one model to another version of a different model all live well the customers don't see any downtime and the reason is redundancy I have my distribution one my secondary right there I can upgrade one while everything else is being handled by the other unit that kind of active active functionality where I know I can flip over in case I got to do firmware updates software updates or anything else along this blue means it's an ethernet connection green means it's like a Serial or IP KVM or uh you know some sort of non PB IP connection and then when we started building out we had four of these racks devoted but we built out one and then as we grew we built them out and what happened is we needed a little bit more density we suddenly had a room that added another 100 cabinets to this room on the build plan that was never on the original design but because we do organic growth we were able to change our technology to a slightly more dense system that allowed us easier maintenance and management so instead of punch down on the back I now have a module that I can crimp terminate and click in very s ilar to the Keystone system that You' see and residential wiring but this will recertify at 10 gig without any problem day in and day out I've got connections 15 years old that will recertify the same way they did on day one modular is the way to go that's the only way to build stuff and we have racks that have a higher density than this uh per you so you can get 48 Connections in a one U size a little hard to work on so when you're working with an organization that cares this much and is growing things organically this much so this is what happens when you start to grow like it gets a little crazy take picture the back take picture people love this this is this is the internet's favorite part of the internet totally this is 15 years of continual development and growth so this isn't a day one and Dawn the very first racks were these ones then it extended over here for our customers our equipment on the bottom customers on the top and it just kept kind of growing that way conduits coming in from the underside there are fiber connections to the entrances to the building so point of entry or meet me room sort of connections so we have some connections there and we have some connections over there but inside that and you're doing it all right I mean this is the the little things like the the sleeves so have a picture yeah the sleeves yep the cold side is cool but it's the hot side where you see the difference in people's experience and that's where we really come along to help our customers out 42in rais floor so so this is a loadbearing glass tile that we had uh acquired to be able to do this very thing you can see the color coordinating of the power cables for the cabinets but those large conduits are actually Distributing power from the room distribution units to our power panels and because who doesn't like interactive technology I'm heating up a sensor with my hand here and it's going to open up the lers there and you can feel how much static pressure there is is it's opening now oh yeah so this is the part that you don't really get a chance to kind of you have to experience this so that is a 5 ton tile right there so in theory that has 5 tons of cooling do you know how the term ton came along for cooling that I don't know it's a fun story and it's quick so it used to be related to how much ice you needed to cool a room so back in the days when that was the way you'd order for your you know mid western movie theater I need three tons of ice to be put in and they'd have a fan that would be pushing that air across the ice to cool a room down so if you needed three tons of cooling that was the amount of ice You' order to cool a room nowadays we know a ton of cooling to be roughly 11 to 12,000 BTU I say roughly it's 12,000 BTU but when you do calculations in a room we like a little head room so we calculate with 11,000 BTU so you have rough about 4 Kow to a ton of cooling so right there you just experience 20 Kow of cooling that is neat people don't really understand what it means until they get to experience it a little bit this is a 20 ton sorry 40 ton air handler uh so what it has is a big radiator coil here variable speed motors for the fans on the underside and it's just baffled up to get the hottest of hot air because the hotter the air you can get into this thing the more efficient the thermal transfer is to the cooling Loop more heat it offloads the colder the air can be on the other side there and that's solved the problem that solves the problem and you'll notice redundant power connections on these they're automatic but if we had to go to the bypass or the utility maintenance grid we the ability manually switch it when I said the power grid in this building is a little different than everywhere else so there's something called STS static transfer switch in the data center and the purpose of that device is to take power from one input and switch it to the other there's two Styles older fashioned Styles which is mechanical where it takes the connection breaks it and then makes a new one you have an interruption power goes out servers turn off then there's like the ssds of the market solid state static transfer switches which is what these are they have the ability to switch power faster than your power supply can notice so right now these units have two inputs feeding them and it can choose hey I have a primary input and a secondary if the primary input has any problem problem it switches to the secondary because it can do that I can take power from here run it to one of my power panels and you have dual fed capability feeding each panel and you have two power feeds going to your cabinet so now we have the switching at the room that's expensive to do because these things cost a lot of money so traditional data centers will usually put this in the back of house next to the UPS so they put the stack transfer switch on the generator SL UPS side and the utility as the inputs so utility goes dark it then switches over to the generator turning on and the UPS rides it as the output until the generator turnes on says I'm good and it latches on to the generator side that creates a pathway for the STS to use and it starts charging the UPS back up so UPS static transfer switch generator utility and that's the common building block wow this is what 1.3 megaw of power distribution looks like often or not you have very large breaker panels that are Distributing power so the power comes into the room breaks out into a larger than residential size distribution panel and then those go to each of the stss which then go turn to each of the power distribution units 1.3 megawatt that's just 1.3 so that's that is a small City yeah and this facility has 36 megaw of capacity in it that's we have facilities that we're in that have up to 300 megaw of power too that's wild it is really really crazy when we're dealing with collocation not everyone needs a full refrigerator size cabinet for their servers so we offer half cabinets and quarter cab compartments I think they on the other side there so that we can kind of write size sometimes you just need your redundant equipment or some offsite so we can help alleviate some of the cost by putting two customers in the same physical space so again the quarter cabin idea so the idea is that you have a fully isolated secure panel area where you're able to kind of have your space but you don't need a massive amount so your offsite your redundant I it's worth noting too these are not your generic all the same cabinet Keys correct yeah so everyone has different needs and to help facilitate that so this is a combination lock we have a key to be able to get in for our side for our technicians but there's a code that the customer has and they can use and they can give out we can change the code at will for them so that they can kind of rotate as needed we do offer fingerprint based and card access based biometric systems for the cabinet level as sometimes they actually have a requirement says I need a card reader or I need a fingerprint reader but they don't need a cage to be able to do that right now the next and final stop on our tour is the partch room there are several technicians working in there and it's cool seeing all the work they do on behalf of people that have things in the Colo so obviously you can put stuff in the Colo and maybe go there yourself but what if you went there and forgot a cable or what if a drive goes bad and you just don't have time to drive out to the Colo they have spare drives they have spare cables they have a lot of them the reason why we have white yellow blue purple and green network cables isn't because we really couldn't make up our mind as to what color we want to standardize on it's because maybe you have your primary switch and your secondary switch and you want to make sure visually at a glance when you walk up to your cabinet every server has both connections yeah so if you bring your equipment here you have the SP parts that will match your layout exactly that's that's a big difference and if you have your own standard we can help you maintain it if you don't have a standard we can use our best practices to help you with that we also can carry Cat 5 cat 6 but Dack cables when you start getting into the 10 gig 25 gig and 40 gig things maybe you don't have a 5 m 10 gig Dack and you came on site to put up your new storage array and oh I don't have the cable or the connection to do what I need well this is where we're able to help you out because for us it's a lot easier to have it on site and ready to go I I think it's fun too because you don't realize until you sent some guy a 2hour drive three-hour drive to the day Center you're like oh he forgot the deck cable well like you ship something off like the manufacturer's like here you go you just spent $40,000 on this new storage array and it's great oh and they didn't include the cables right or the rack mounting kit so by having the parts on site here we're able to help you out and maybe even save you that 2hour drive because we're here 24x7 so why don't we rack it up for you absolutely so ssds they wear out they go bad and they're specific to size the most common failure now and the difference between oh that's a 800 gig versus a 960 gig can be the difference between you having a good day and a bad day yeah very much so so for us we stock all the common sizes of both Enterprise grade rotational drives to solid state drives to pcie memory based systems so that we can plug it in and help it out we have to maintain systems ourselves so we need these spares for our own internal use but why not be able to make it available to our customers but the fact that you have all this too sometimes you have a half height PCI slot and a full length card uh yeah and Lord knows you've uh had to connect a storage array and like which one did you need the mini SAS or get hundreds of power distribution units with their variations between manufacturers and feature sets sometimes when you're holding a cold spare for someone you know what you need has to fit this way where the outlets are going this way or maybe you need something that's a little wider if you have one in production and it fails on you you have to have a spare the problem is well it depends did you have something that ran 208 volts at 30 amps or were you running something that is you know 40 or 50 amps is it a Twist lock plug that looks like that and then if you come over to this side or is it one of these ones where we start getting into kind of the fun 50 amp level uh heaven forbid you are at the 60 amp level which gets to something around that size oh there's the beefy one and I think we have one more fun one here I should have a 60 in receptacle floating around under here and this kind of comes down to the spares ah here we go so if you took the Metra at all yeah that is the twist lock system you start running at at 60 amps when you're dealing with multiple phase power same style connector used on the uh train systems yeah neat so now the Dilemma is great all these choices which one's the right one well that depends the problem is you have all these different form factors because you have all those different connectors there and different requirements which means when we have to ship something to a different country that has a different Power standard what do you choose well it's easy to make a mistake here when you have all these options here and they're all very similar to each other so what we have over here is really cool product from Eaton and they call it the upu so what you have is a pdu that has a connector here a multi-pin connector that depending on how the cable's wired up changes how the pdu works so this thing can be a 20 amp 208 volt singlephase power distribution unit that can handle about 3 Kow of power or I change the cable out and it can be a monster that can handle 17 Kow of power oh man all without having to change out the chassis very cool and we'll show a little later on the uh reason not having to change out the pdu is so important because these things get buried inside the cabinet I like the little green test lights totally it's just cool so these are brand new now imagine if I had this chassis I could buy a couple and have them at every data center we have across the world sitting there waiting you order a 30 amp circuit that's 208 power great I send out a cable FedEx envelope overnight easy International shipment you're ready to go oh you had a problem with one of these units in your cabinet great I unplug the cable pull out the box put a new box in done the most common scenario we run into though is hey I started buying services from you 5 years ago when my power needs were this but now I've grown to the point where I need something more what does it take to upgrade well it's a two-person job of installing additional power cir down the floor then removing one of these things physically while it's live putting a new one in migrating all the power plugs and then doing that again and when you uh have to do that twice on a live environment it takes a lot of time and it's highly stressful or I just change out the cable and you're done that's just simple it's truly insane how elegant the solution is for being able to handle both Us North America South America and international European asia-pacific power needs all with the same cable a big thank you again to Deft for this tour allowing me access to their data center this was just a lot of fun really enjoyed it hope you enjoyed it as well leave your thoughts and comments down below let me know what you liked or didn't like like And subscribe to see more content on this channel and of course sign up for my forums forums. laoren systems.com to discuss this topic or any other topic you've seen on my channel is a great way to engage me and [Music] thanks

Channel: Lawrence Systems

Views: 177,885

Keywords: LawrenceSystems, data center, data center tour, data center power, data center power system, data center power management, data center power consumption, data center power supply, data center power redundancy, data center cooling, data center cooling system, data center cooling system design, data center cooling tower, data center cooling techniques, data center construction, data center networking, data centers explained

Id: gsN_CJJDy_o

Length: 29min 52sec (1792 seconds)

Published: Mon Jan 29 2024