Building a Power Efficient Home Server!

Video Statistics and Information

Video
Captions Word Cloud
Reddit Comments
Captions
Are your utility bills driving you into debt? Is your mum kicking you out of the basement because your server consumes more than all Chinese mining farms combined? Is Radiation Safety Commitee knocking at your door because your home server looks like a juicy piece of enriched plutonium from space? Well, fret not, because in this video I’ll show you how to build a super power efficient home server/NAS, no matter your use case. Whether you need a low power mini PC or a beefy Proxmox host I’m gonna tell you how to get the best performance per watt and make sure your homelab hobby doesn’t stick out like a sore thumb on your power bills. So strap in, and let’s get started. This is my current home server. I primarily use it for video editing over the 10 gig connection and media storage. It draws around 23 watts from the wall on average, and with current energy prices in Germany this machine costs me 5 euros and 37 cents per month to run 24/7. Now, considering the fact that a lot of r/homelab users run machines that draw 120+ watts at idle, I think 23 watts is pretty good. Keep in mind that this isn’t an Intel NUC or a similar small form factor compuer I’m running a full-blown rackmount server with four 3.5 inch hard drives, two NVMe drives and a 10 gigabit networking card. So, why would you even spend time optimizing the power consumption of your server in the first place? Well, If you live in a country with expensive electricity, the difference between running a 50W home server and running a 100W home server will definitely be noticeable on your utilities bill. And considering the looming energy crisis in Europe, this might become very relevant for a lot of people in the near future. A power efficient server will also stay cool and quiet so if you live in a country with hot climate or don’t like the fan noise, this might also be the way to go. Whatever your personal reason is, designing your home server to be as power efficient as possible is a great way to save money And I don’t know, personally, I kind of like the idea of squeezing the most out of my system while making it consume as little juice as posisble. Some people might find my power efficiency obsession stupid and pointless especially if you live in a country with cheap electricity or just don’t care about your utility bill. Well, good news, I don’t force you to become a power saving freak like me and you’re free to ignore my advice or skip this video altogether. But if at least one video finds the information in this video useful, that means my work hasn’t been in vain. Building a power efficient server begins with, well, picking power efficient components. And that’s not as simple as buying a low TDP CPU and pairing it with a server motherboard First things first, if you want to save on your utility bill, don’t buy old hardware By old I mean pretty much anything older than Haswell or the 4th series of Intel CPUs. Older systems might be cheaper, but your savings will probably go towards paying the increased utility bill Even though older CPUs might have the same nominal TDP than the newer ones, they’re much less efficient at idle and might not support the power efficient C-States that the newer CPUs support. At the same time, newer doesn’t necessarily mean “more power efficient”, And recently we’ve seen the PC components like CPUs and graphic cards Become more and more power hungry for the sake of ‘performance’. Pretty much any Intel CPU that is 6th gen or newer will be pretty efficient at idle, And even though the newer CPUs might be slightly more efficient, You will have to shell out more money for a newer motherboard and a more expensive CPU So if you’re okay with the performance of something like 6th or 7th gen CPUs, don’t chase the newer 12th and 13th CPUs purely because of the power efficiency But what about AMD? Well, when it comes to Intel vs. AMD, Intel systems tend to win When it comes to budget-ish systems, Since they tend to consume less than equivalent AMD Ryzen machines. Ryzen CPUs also still seem to suffer from the idle freeze bug on Linux, Which can only be fixed reilably by disabling the power efficient C-states, Which is kind of counter-intuitive That being said, you can still build a pretty power efficient system with a Ryzen CPU, And some users report as little as 7W power consumption with a 4350G. One more thing I want to mention is that the TDP spec means absolutely nothing for the real-world power consumption in a home server. TDP only describes the power consumption under load, And in a lot of cases, even that figure doesn’t match what you’d see in real-life use. Despite consuming upwards of 100W under load, Many modern processors can still enter a power-efficient idle state In which they sip less than 1W. That also applies to the T-Series Intel CPUs These are pretty much the same chips as the non-T models, just capped to a smaller TDP. This has no effect on the idle power consumption, so don’t pay more money for a T-series CPU because of the supposed power efficiency Since your server will most likely be idling most of the time, at least compared to a desktop PC, the idle draw is exactly the figure we’re interested in and it can vary wildly depending on your motherboard, PCIe devices, power supply, and so on Unfortunately, very few manufacturers and reviewers publish the idle power consumption figures, but luckily for us, there’s a whole community focused on building power efficient computers, and it can be found on this German forum called Hardwareluxx The forum members even maintain a database of the most power efficient builds Sorted by consumption Which can be a great help if you’re looking for power efficient components So let’s dive in As you can see, the most power efficient systems, ones that consume as little as 1 to 4 watts, are Intel NUCs, laptop motherboards and basically ultra small form factor PCs. Obviously, those computers are power efficient for a reason they don’t have the most beefy CPUs in them And the number of features and ports is also very limited You’ll rarely find more than one SATA or M.2 slot, and PCIe is also usually out of the question. That’s the price you’ll have to pay for ultimate power efficiency, And if you’re okay with not having those features Then these options could be for you These machines should still have plenty of power for Virtualization, Docker, Kubernetes, Proxmox, Or even running a media server or a Home Assistant Instance But they’re probably not the best choice for a NAS since they lack expansion If you want your home server to be a bit more capable, The next option is a desktop CPU and a miniITX motherboard MiniITX motherboards tend to have less ports and features than their ATX counterparts, and because of that they usually consume less power You might think that the difference between a miniITX motherboard and a full ATX mobo with the same CPU is negligible, Aaaand you’d be wrong Most modern CPUs are pretty efficient at idle no matter the motherboard, contributing as little as 1W of the overall power consumption. However, if a motherboard isn’t really optimized for power efficiency or has a bunch of onboard devices like multiple Ethernet ports, Audio, serial, iKVM module and so on those devices can add as much as 10 to 15W to your power draw, and disabling them in BIOS doesn’t always help. However, not all miniITX motherboards are made equal, and some models just aren’t very well optimized for power efficiency, These miniITX motherboards from ASrock and Fujitsu on the other hand seem to be especially good when it comes to power efficiency So if you’re okay with only having one PCIe slot, a miniITX motherboard is definitely an option that you should consider Some server and workstation motherboards also support PCIe bifurcation Which lets you connect multiple devices to one PCIe slot using a bifurcated riser like this one. However, a bifurcated riser can cost as much (or even more) than the motherboard itself so it’s only worth it if you want to build a small form factor server in the first place But what if you need multiple PCIe slots? I have a 10 gigabit card and an NVMe adapter that I need to plug into my motherboard, and the Asrock mobo that I used before only had one PCIe slot, and no NVMe. Well, let’s once again take a look at the spreadsheet. As you might be able to tell, the most efficient motherboards aside from miniITX and ultra-small-form factor PCs are from… Fujitsu This build in particular is very impressive –  an Intel Core i7-9700K and a Fujitsu D3643-H combo only pulls 5,8 watts from the wall I already had a CPU handy, so I started looking into Skylake motherboards from Fujitsu, and I found this D3402 motherboard on eBay. With 5 SATA ports and 2 full-size PCIe x16 slots, it fit my needs perfectly. Now if you don’t have any parts handy and want to build a new PC from scratch, I would recommend going for a newer Fujitsu motherboard that supports at least 8th generation Intel CPUs Or at least I would, if you could actually buy them At the moment, both 8th and 9th gen Fujitsu motherboards are out of stock on pretty much all online marketplaces New and used. But when they do crop up, they’re usually not super expensive, so keep your eyes peeled Meanwhile, you can take a look at these 6th and 7th gen motherboards They seem to be easier to find on eBay and local marketplaces, and are also very power efficient. After rebuilding my system and replacing the MiniITX Asrock motherboard with the Fujitsu one, I booted the machine up, waited for the drives the spin down, and looked at the power consumption It pretty much stayed the same, which was kind of disappointing, but at the same time it didn’t increase, despite me switching to a bigger motherboard. On the other hand, similar builds in the spreadsheet pull less than 10W. So how is it that my system still pulls 20? Well, let me introduce you to Package C-States. Package C-States are power efficient semi-idle states that the system can go into when there isn’t much going on The system is still technically ‘awake’ in this state and can do background tasks, but consumes much less power There are usually 8 to 10 Package C-States on modern systems, and the bigger the number, the bigger the power savings C1 corresponds to high system activity, whereas C8 is kind of a daydream The easiest way to see which C-state your system is in Is by using an command-line utility called `powertop` You can install it using your favorite package manager, and after doing that let’s open it and go to the “Idle states” tab. Keep in mind that we’re looking at Package C-States on the left, not the CPU C-States. As you can see, my system only goes down to C3, which is not ideal. What could be the culprit? Well, either the OS and software configuration, or pretty much any connected device that might just have crappy firmware The former is very easy to verify by booting up a fresh Ubuntu LiveUSB, installing powertop, executing `powertop --auto-tune` and then running powertop again and taking a look at the measurements. If it stays the same, the issue is with the hardware. In my case, the culprit was my 10 gig networking card, Mellanox ConnectX-3 It’s pretty old and I bought it for cheap And because of its age, it doesn’t really support proper PCIe power management. As a result, with the card installed, the system wouldn’t go into deep idle Unfortunately, the “cheapest” SFP+ card that supports ASPM is Mellanox ConnectX-4, which starts at €170 on the used market Not really worth the investment Another common culprit are cheap NVMe drives The companies making budget drives don’t usually put a lot of money in firmware and power optimizations, so with a lot of them, you won’t get proper ASPM support, once again resulting in no deep idle I got two cheap SSDs, one from Sabrent and one from Crucial and the former does not support ASPM and won’t let the system go lower than C3 As you can see, with the Mellanox card and the SSD disconnected, the system now goes as low as C8, which results in 8W idle, even with four hard drives connected to it in standby mode. Obviously, at this point it’s not really worth it, since I would like to keep my 10 gig networking and my SSD so what are some other options for reducing power consumption? Well, the thing is, even with all the right hardware and a platform that can go down to C8 idle, You probably won’t see the power consumption go as low as 8W with a conventional ATX power supply And that’s because even power supplies with Gold or Platinum rating have relatively poor efficiency at lower loads Which is understandable If you’re building a conventional desktop PC, You probably aren’t planning to just let it sit at idle And do nothing the whole day If you turn it on, you probably want to game on it, watch YouTube, do some work, and so on Because of that, a lot of people who run low-power systems Use alternative power supplies called PicoPSUs paired with laptop power bricks Those PSUs are much more efficient at lower loads, and as an added bonus, they’re also completely fanless Thanks to their size, they also work very well in small form factor systems. Obviously, using a PicoPSU only makes sense if your idle power consumption is lower than 50W and doesn’t peak higher than 200W. Otherwise the savings will be negligible. Another downside of the PicoPSU is the number of power connectors. You’ll usually have an EPS 4-pin connector, an ATX connector, one Molex and one SATA connector. And that’s it. However, you can use splitters to connect more devices, and I’m currently using a Molex Y-splitter for the SATA backplane On my Supermicro case. It has four WD Pro 7200 RPM hard drives installed in it, And the entire system pulls around 120W while booting up, Which is well within the 160W spec of my particular model. For systems with more than four hard drives or a dedicated GPU, I suggest this 550W Corsair PSU instead. The 2021 model in particular has shown to be almost as power efficient at low loads as a PicoPSU, And with way more power connectors, it’s basically the best of both worlds. If you’re building a NAS, you’re obviously gonna need hard drives. And here you have a choice between 5400 and 7200RPM drives. The 7200RPM drives are usually slightly faster than the 5400 ones, but also run louder and hotter; and consume more power. Western Digital, who by the way, sponsored this video, has both 5400 and 7200 RPM CMR drives in their Red line up WD Red Plus and WD Red Pro. The WD Red Plus drives consume as little as 3W in idle, and when spinned down, this number goes to 0,4W. The 7200RPM Red Pro drives, apart from being faster, can also be used in server enclosures with up to 24 hard drive bays, thanks to their anti-vibration technology. Both WD Red Plus and Pro are CMR drives, No matter which capacity you go for, and are a great choice for a RAID or ZFS array. Thanks again Western Digital for sponsoring this video, and now let’s go back to the topic of power efficiency If you’re trying to build a power efficient, cool and quiet server, I would definitely recommend going with 5400RPM drives. You probably won’t notice the speed difference compared to 7200RPM drives Unless you run a RAID or ZFS array with dozens of drives, But what you will notice is slower spin up times, Higher temperatures and higher power consumption So unless you absolutely know that you need 7200RPM drives, just go with 5400RPM. However, even 5400RPM drives will still consume power while spinning It’s not much – a single hard drive will draw around 3 to 8 watts, but it obviously scales with the number of hard drives that you have in your system. So no matter how well optimized the rest of your build is, If you have let’s say 6 or 8 hard drives spinning all the time, You will see a sharp increase in power consumption and noise Now my personal solution to this is to spin down the drives after a certain period of inactivity However, spindown is kind of a controversial topic Some people say that spinning your drives up and down Causes additional wear and tear, And can possibly reduce your drive’s lifespan. But some people argue that this is not an issue anymore with newer hard drives Sminlal on Tom's Hardware forums says: This is not as big a problem as it once was In the old days when you spun down a drive, The heads would lose their air cushion and come to rest on the platter as it was slowing down That caused wear Most modern drives use loading ramps which let the heads be retracted beyond the edge of the disk So that there's no physical contact between them And the platters when the drive is spun down As a result of this the drive specs that I've seen Show ratings on the order of 300,000 stop/start cycles That's a 10-year lifespan if the drives are stopped and started Every 20 minutes during every hour of every day, 365 days a year Spinning down a drive brings its power consumption from 3-8 watts Down to less than half a watt It also completely eliminates the noise, and provided you don’t spin your drives up and down every 5 minutes, Shouldn’t have any appreciable impact on your drives’ lifespan. I found that spinning the drives down after 30 minutes of inactivity works pretty well for my case However, spinning down the drives also means that they’ll have to spin back up Before you can access the data stored on them. This might not be a big deal with 5400RPM drives, But 7200RPM drives take longer to spin up, And if you’re running something like Unraid or MergerFS, The drives will spin up sequentially instead of all at the same time, Which will result in a longer wait before you can access the files stored on those drives But there is a way you can eat you cake and have it too That is to say, have near-immediate access to frequently used data and minimize the hard drive power draw and noise. Normally, if you want to have a fast NAS With instant access times, You have to either create a RAID array with dozens of drives and keep them all powered on all the time… Or build an SSD-only NAS with no spinning rust whatsoever. The former requires a beefy server motherboard and multiple HBA cards, And will also have a massive impact on your power consumption …And the latter will obviously cost an arm and a leg, Since SSDs are much more expensive when it comes to the raw dollar per gigabyte value. However, by adding a relatively small SSD array to your NAS, You can make use of a technique called "Tiered caching". Tiered caching is the process of combining fast and slow drives into one array To speed up the file operations You basically tell all applications and services To write to the SSD array first, And once it gets filled up or you haven’t accessed the data for a while, the files get moved to the slow drives Unraid, one of the popular NAS operating systems, actually includes a utility called ‘Mover’ that does exactly that You can set any of your shares to use the cache, and then the Mover will take care of moving the files From the cache array to the slow spinning disk array regularly If you’re not using Unraid, there’s also a Python script Written By a Github user elmuz that basically accomplishes the same thing You can schedule it with cron and customize some parameters to fit your workflow, such as the target percentage of drive space and so on. The SSD array you’ll use for caching can be as small, or as big as you can afford, but in this situation, bigger is better. If you attempt to write a file that is bigger than the whole SSD in one go, You’ll probably encounter an input output error. Apart from that, the bigger your array is, The less you’ll have to spin up your hard drives Which means more power savings and longer drive lifespan. For my NAS, I currently use a RAID0 array of two 1TB consumer grade NVMe SSDs, And it has been plenty for my use case I am however planning to upgrade it to an array of four 2TB WD Red SSDs soon though, so make sure to subscribe if you don’t want to miss that video So that’s basically all the tech tips I’ve got for you today, I hope you enjoyed this video
Info
Channel: Wolfgang's Channel
Views: 612,433
Rating: undefined out of 5
Keywords:
Id: MucGkPUMjNo
Channel Id: undefined
Length: 19min 46sec (1186 seconds)
Published: Wed Oct 12 2022
Related Videos
Note
Please note that this website is currently a work in progress! Lots of interesting data and statistics to come.