Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein (Germany)

[Music] hi and welcome back to a new video you can see we are in the middle of nowhere it's called falkenstein and i'm not sure if you're familiar with this but at least for me being from germany and like the good old teamspeak days and when you were still picking servers for your games sometimes some game servers were showing the location called falkenstein and that is in the eastern part of germany and we're here we are at our corporation partner hetzner which is a data center provider and i'm yeah very curious to see how everything here works out we will get a tour today to see how they're building servers how everything is set up in regards of like energy management which is also very important nowadays considering the climate change and everything they're also using some uh consumer hardware like amd ryzen cpus in servers which will also hopefully give us some very interesting insights in regard of what happens if you use amd ryzen cpus for a very long time for a use where they're maybe not intended to so all of that could be interesting and i also noticed that they're just building a new entire building from what i can count it's like one two three four five six seven eight and this is like the ninth building so it's pretty massive all right let's just go inside and check how things look what i was actually most interested in is to see how they manage cooling and also power supply of such a huge facility because obviously if they have several thousand pcs or servers running it's a very high energy and power supply required which also takes up a lot of resources but hetzner is known to pay attention to those details and also try to use as much screen energy as possible has a slight touch of entering a prison with this massive door and like the barbed wire on top apart from the data center in falkenstein which we are visiting today hudson has the second one in nuremberg in the south of germany and also one more in helsinki in finland and what i find really good about hetzer is that they're paying attention to use 100 green energy and especially in finland they managed to only use wind energy so we are standing in front of the building number one if you remember from the intro that is the building like on the total left side and that is the first building they established around 2008 and we will now enter the building and check out probably one of the servers out of 250 000 servers that's quite a lot so as i just pointed out this is the first data center which was established around 2008 and if you just look at this this is all midi towers but that is mainly because headset just wants to keep them running as low as they're working as long as they're working their policy is to make sure that they're using energy and resources as well as possible and in this regard if the hardware is still working why would you just throw it away right if you still have customers that can use them they can just keep using these systems and it's also quite entertaining if you just look at the systems they have labels on the back for example if you just look at this label it points out i7 377t uh using four times eight gigabyte of memory and like an asus motherboard which is also just a consumer grade and if we just move one pc to the right then we can find an i7 and 920 which is also quite old already so that is very very interesting so the cooling concept is also quite interesting because it's actually underneath me uh like one meter below me they have some air ventilation channels and they go to the site and those pcs are like yeah you always have two rows and the fronts are facing each other and you always have the cold in a cold air intake uh through the front so they have the ventilation going through the bottom right here then it will go up and they will have the cold air intake uh from the front and then one of the technicians from headset just highlighted that i should not be worried about the messy cable management and i was just looking at this i was like this looks like perfectly organized i mean they should they should just look at my place at home it looks completely messy compared to this but according to their standards this is very messy and not organized and i'm very curious to see one of the new servers we are now above those midi towers and just look at them down there those are 14 000 midi tower pcs obviously this is not 2021 standards anymore we will get to a nowadays server soon but it's just a comparison and also for you to see the evolution they had at hadzner when it just comes to the rack design and everything but cooling wise um now you can see if you check it from above you can see the like cold air intake which is this channel between the midi towers in the center and the cold air intake is about 20 degrees celsius and that's also a mixed air because they're mixing the cold air from outside which is about currently 15 to 16 degrees celsius they're mixing it with the warm air that is already inside the building so they're not even using active ac because that's maybe a misconcept conception that i had i always thought that data centers with this size they always have to rely on active ac cooling but that's not the case they only use the ac on few days per year because i mean it's germany germany isn't the hottest climate there are only a few days where you have much more than 30 degrees celsius so the most time of the year they can just rely on the cold air from outside and then they're mixing it with the ambient air in here to always make sure that the temperature in here is always 20 degrees celsius well at least down there because up here due to the natural convection it is much warmer it's probably like 30 degrees celsius up here and that's also the reason why they have this angled roof because they rely on the convection for the cooling or well for the air exhaust right through the roof the warm air can exit and i can totally feel that and that's i mean this kind of reminds me a bit of mission impossible if i was some cruise i would probably climb in there but yeah security levels are very high here we just entered a different part of the area from hetzner and this building right here was established in 2019 roughly and just looking at this you can see this is completely different no more midi towers and it is warm in here it is really really warm that is because we are like at the hot air outtake like hot air exhaust area so if we just walk up here we can feel that the hot air is coming out here and yeah i mean let's just go upstairs again and to get a proper impression okay so we just went upstairs again in the different data center area which is a lot newer and just looking down it kind of feels like it's it's a bit more empty than the other one which could give you a wrong impression one of those rows are 1400 servers and what is quite cool in this case that we can look through from the top so the top is closed off with some acrylic but if we take a look through the two rows of the servers you can see the mesh on the floor and that is the cold air exhaust well that's the cold air delivery for the servers it will go up between the servers and then push through and at the same time it's not only getting pushed through from the air conditioning or from the cooling concept itself because the air conditioning is not switched on as i already explained but the servers are also actively cool so they have depending on the server configuration they have pull configuration push configuration push pull because there are also a lot of epic servers down there and epic cpus depending on the cpu maybe 280 watts tdp yeah they require cooling and that is also why this kind of looks empty because the area down there where you have all those green blinking leds that is a cloud area and the cloud area has a very high power density which means that it requires a lot of energy at the same time they have to make sure that they can cover everything with the ups and yeah that is the reason why it might look empty but it's definitely not everything which you can see down there is also not the normal 19-inch rack standard because they are building their own servers they're building their own racks their own cases their own cooling concept everything is completely built in-house and developed in-house which means on first look it doesn't look like a standard but it also gives them much more flexibility running 250 000 servers also requires a lot of storage i mean that's what the servers are for and sometimes some of those hdds fail which you can see they're all dead and they have to be professionally recycled and obviously also get destroyed because otherwise you cannot make sure that the data would be completely erased obviously 250 000 servers have to be built somewhere and that's why we just entered the production area just looking at this is already quite impressive just trace of cpus for example 9900 case so case cues 8700 without k those are for the ax 61 series that's the service we also advertised on my channel ryzen 9 3900 one tray another tree underneath i i would like to touch things but i tend to break things that's why i'm not going to do it and we have a lot of cooling equipment the typical server heatsinks thermal paste yeah so that also shows the cooling concept of hat center again because they're usually relying on a pool configuration for the cooling itself just pulling out the hot air that's why you also have those like air channels depending on the specific server like for example px62 it's a cooling channel for asus wsc 246 okay so hopefully we can now go to the production area and actually see how they're building some of those servers as i just pointed out hetzner also i mean they have their own main boards those main boards are specifically built from asus in this case for hatsner and just looking at the skew it's a pro ws565ace never heard about this before i also never heard about the pro 565 chipset which is built on this mainboard and this chipset it it's not like it's spanning overclocking but it just doesn't allow it which is essentially the same but um it also makes sense that in the server you're not allowed to overclock the cpu because it's i mean they're building this for very specific like heat dissipation and cooling concept and if you're running in this case it's a 5950x if you're changing the power dissipation of the cpu run changing the voltage and everything it will have a huge impact on lifetime of everything of the vrms of your cpu itself and yeah they have to make sure that everything runs reliable and runs on a very long lifetime that's why they're using this very specific chipset but let's just go over and check out the build process process itself the build process of the server itself just as the components is also very minimalistic you only use what you need and that's why the case itself for example is just a few sheets of metal and by the way the case itself is also 100 made in germany and for the psu for example it's also custom made to use for shorter cables to only use the resources they really need and also to make cable management much easier apart from that it's almost identical to any kind of desktop build you know just using much less components just using as much as you really need so this would be our server ready to go and as you can see everything on this is purely cooling focused and that's also why they have this very specific layout with a different cooling configuration or cooling direction as what you would have on a desktop pc because the exhaust is on the rear on the right side with this cooling channel on top and it will just suck the air through the heatsink and also at the same time make sure that it's also cooling the memory sticks on the side and also tries to cool the vrm's at the same time from the same direction so everything is very efficient to cool the server with just a single fan for exhaust and at the same time cool every single relevant component you will remember a huge pile of hdds where we just set on and somewhere you have to determine if those hdds actually are damaged or if they're still in a good working condition because sometimes the customer might report that they're seeing some kind of issue within the hdd and hetzer has to verify if that's true or not and that's what's happening right here that's the hdd testing area just such an enormous amount of hdds and you have those status leds like right on top here number one to seven they're blinking in a slow pattern which means that they're in the testing condition right now if we switch over here so number five is permanently on which means that it passed the test but if we go down there you can see for example this hdd with this fast blinking pattern means that this is actually damaged and this one is actually also damaged so those eventually will be replaced even though hetzer pointed out that they're working on this and in in their mind this is still messy and to me this is like like everything is perfectly organized but they're still working on making it even more perfect the organization that's why they started like 3d printing some hdd holders for their testing equipment adding the leds right in the testing holder itself and then have this like more organized because in their mind this is not organized but to me this looks like perfect this is the same area where we've been before from above so this is the area where we have about 700 servers on this side 700 servers on that side so they call this the cold aisle basically you have the the cold air coming out from the floor through this mesh right here and then the cold air goes all the way up through the surface back out here the temperature is quite okay it's about 20 degrees celsius if we go up back outside it's about 28 degrees celsius and at that point i also want to highlight that's actually a huge honor and i just want to thank hetzer that we can be here that they have the trust to put me in here because i'm well known for breaking things and i could just rip apart i don't know things here and damage parts which i obviously don't want to do so yeah absolutely amazing that they were so open and show everything to us and gave us the opportunity to show everything also to you guys out there right here it's again the ax 101 the server which where we just uh showed and watched the building process right here with the two ssds on the side above we have an ax 161 which is featuring an epic 7502 so single socket server epic above we have an ax41 nvme that's the one which we were advertising quite often in our pre-roll videos ryzen 5 3600 and that is compared to the midi towers which we saw right at the beginning that is the most recent evolution of direct design and also like also very recent hardware in the event where you have a loss of power then certainly you have to do something that your servers are still running so you have those battery rooms it's just it's just a room full of batteries uh one of those cells has two volts so you can just hook up a ton of them um in a row just 120 for example then you have 240 volts which is still dc so you have to convert it to ac i guess not sure how exactly they're doing that um or you just run it with 12 volt directly to the server which is something you wouldn't do because it's running on 230 volt yeah it wouldn't make much sense anyway so this is enough power to power the area which you just saw one data center that's the area we saw from above for 15 minutes uh yeah 15 minutes but that should usually never happen because you always have your emergency power generator and which is running on diesel that should run within seconds so this will never be out of power usually since cooling is the most relevant part of the server itself and the entire cooling just relies on a single fan they're also doing in-house fan testing and the in-house fan testing is in an area where the ambient temperature is about 40 degrees celsius which is slightly above what you would have in the server itself so it's tougher for the fans and those fans are running 24 7 for long durability testing so if one of those fans or multiple will fail they know there might be an issue with these fans and they can replace it or could replace it in the server before they are failing and it will also have an impact on future purchase decisions this is one of the massive fans which are pushing the air through the floor that is really massive and especially if they're rotating fast you can imagine it's also quite loud we're taking a quick look inside the test lab of hetzner because obviously they have to verify different kind of setups if new for example new cpu generations are entering the market they have to check cpu mainboard memory configurations but also if there is like an external request for example a customer wants to have 100 servers with a specific configuration they don't usually do then they have to make sure it's verified everything is working for example the left they have a specific test sub setup that's only testing hdds or ssds and if we just continue to the right there are multiple different setups and this one is just running right now with some thermal testing load so they have a thermal imaging camera right here and this thermal imaging camera is tracking the area down here of the vrms so it's checking if the vrms are overheating during let's say prime95 load for example we also found this very interesting unit uh the first look you will just think that this is a typical server because of the like greenish pcb and everything but let's just go to the right where we have the same type of mainboard but it's just not populated the base is b550 but it's again the pro 56595 chipset whatever so it's the same chipset which is basically blocking overclocking but the special feature is that you can see it's a rotated socket so evga is not the only mainboard company which is able to rotate the socket you can see this one is from astro rack and the reason why the socket is rotated it's just a pure cooling aspect if we move again to the left so talking about the cooling concept again the cold air will just enter the server from the right side and because the socket is rotated the cooling can make use or utilize the full length of the cooler at the same time cool the brm's on the right side cool the vrm's which are sitting underneath the tunnel on the left side and also efficiently cool the memory sticks so the only reason why they had this specific mainboard made because of the rotated socket which allows better cooling for their very specific use the last also interesting feature inside the test lab is this thermal testing chamber where they can increase the temperature inside the chamber there's just some heating elements sitting inside so they for example can simulate an ambient temperature of about 45 degrees celsius which is something you would never have in the data center but it could theoretically happen that your ac is maybe breaking down i don't know and in that event you want to make sure that your pc or your server doesn't crash it might throttle because the cpu is overheating but it shouldn't crash and that's what they're testing right here with their servers and then we also found this interesting thing it's a 3d 3d printed part fan testing so they're also testing the fans we already saw the fan like relia reliability testing area they're also checking the fans for example is the advertised air volume on the fan exactly what they're also delivering because they also have to double check if it's like 50 square meters per hour or if it's maybe just 25 even though hetzner usually doesn't do water cooled systems or water cooled servers we still found this piece right here and this is for testing psus because i mean what else would you use than an amd threaded were overclocked that can consume a ton of power and in case you want to test if your psu can handle 600 or 700 watt load then you can just use your amd 30 per system all right so that's the end of the video series from visiting hetzner i hope you enjoyed the video as much as we did and thanks for tuning in see you next time bye you

Channel: der8auer EN

Views: 1,082,965

Keywords:

Id: 5eo8nz_niiM

Length: 20min 29sec (1229 seconds)

Published: Wed Oct 13 2021

Reddit Comments

[u/r3dk0w]

I know a lot of self-hosters host at home, but Hetzner is a very common external hosting option and this video gives you a behind-the-scenes view into what goes on in the datacenter.

[u/llobotommy]

Falkenstein’s monster

[u/planetpranav]

good video, thanks for sharing, please consider sharing this on r/hetzner too

[u/zxcase]

Quite interesting to see the place where a lot of stuff is running (except for Nürnberg ofc)

[u/diskowmoskow]

Mid tower section is damn interesting, thought they wouldn’t be efficient.

[u/[deleted]]

You hosted your own data center?

[u/CaptainSur]

One of the homes of spam heaven. Along with OVH (who is much worse). I think I may have 50% of the ips home at this datacenter blacklisted! lol

That is not to say Hetzner is not a quality operator. It is not generally their fault their products are abused although their sales methodology does lead to an easy abuse conduit.

[u/CAP_NET_ADMIN]

I recently had contacted Hetzner support and can only say great things about them. Especially moving from OVH it's refreshing to see professional people working there.

[u/bradbeckett]

I have clients that host all their services here. The energy crisis in Germany is very concerning though. Somebody needs to replicate their model in the US.