$5,000 Raspberry Pi server

this is absolutely ridiculous in the best possible way this is a 48 terabyte raspberry pi nest and it cost more than 5 000 bucks these ssds are overkill for raspberry pi not to mention the insane 8 terabyte nvme drive hiding out on the bottom to be clear having 48 terabytes of ssd storage isn't ridiculous but strapping that onto a raspberry pi most definitely is and i'll show you why but you might be wondering at this point jeff how can you afford all those ssds well let me introduce you to this video's sponsor lambda remember that video where i used a raspberry pi to detect red shirt jeff's face and it barely managed three frames per second well lambdas like strapping rocket boosters to your ai and machine learning workflows they have specialized hardware and cloud gpu compute resources built for machine learning their servers make things like pi torch and tensorflow work way faster than anything else on the market and even if you don't want to buy one of their insane servers you can use their gpu cloud check them out at lambdalabs.com stupid can be fun and that's what this channel is all about i figure since everyone's going to point it out in the comments anyways i'll give a quick rundown on why this whole build is insane then give the reasons i did it anyways first here are some reasons not to build the snaz these drives are expensive at 88 bucks per terabyte for the evo ssds compared to about 30 bucks per terabyte with 3.5 inch drives and i could use cheaper drives with extension cables but then it gets messy and loud anyways these ssds are way overkill since the raspberry pi can only put through a little over 400 megabytes per second total on its entire pci express bus so no matter how many drives we slap in here they're gonna bottleneck especially that fast nvme drive on the bottom the pi also doesn't have ecc ram or well technically it does from a certain point of view and i won't get into what it is or why it's important for enterprise-grade storage servers but this pie definitely doesn't have what it takes to be a top-tier storage server but it is fun and in the course of making this video i learned a lot but is all that learning worth five thousand bucks probably not but for this video it's go big or go home so first let's look closer at the rads at taco it's this main board that connects a compute module 4 to a variety of i o through this pci express switch chip the compute module 4 is my favorite pi because it gives me access to a pci express gen 2 lane i can plug in almost any pci express device and well a lot of them work with a little coaxing and maybe a kernel recompile i have a shirt for that on redshirtjeff.com the official compute module i o board only has a single slot but pci express similar to ethernet can have switches which lets you plug in multiple devices and the taco switch lets the compute module 4 interface with 2.5 gigabit ethernet an m.2 nvme slot and 5 sata 3 ports as a final bonus the production board will have an extra m.2 slot for things like wi-fi 6 or machine learning accelerators the board also has a few headers for a fan and i o a 12 volt barrel plug for power a power button hdmi usb 2.0 and an extra gigabit port so it's no slouch when it comes to io before i put it together i put on these metal standoffs just to hold the board up off my desktop rads will have a case for this soon but i couldn't get one in time since they're still working on it i popped a four gigabyte light compute modulon then i screwed in this 8 terabyte sabran nvme ssd why 8 terabytes well thank lambda for that and this ssd is actually dual sided it has memory chips on both sides of the board and it still fits next i flipped over the board and installed five 8 terabyte samsung 870 qvo ssds again why 8 terabytes well two reasons one sponsor two i wanted a quiet low latency nas and ssds are the way to go these eight terabyte ssds are actually slightly cheaper per terabyte than a similar spec 2 terabyte ssd and there aren't any 2.5 inch mechanical disks even close to 8 terabytes and yes there are even bigger enterprise ssds but a channel my size i'd have to win the lottery to afford them if you work for linus tech tips and want to ship me a few of the old drives from wanak comment below i put in a micro sd card with 64-bit ios and was ready to plug in the taco to be honest i was a bit nervous there were five thousand dollars of hard drives attached and i'm about to plug in a 96 watt power adapter that's right this isn't your father's pi this thing needs a lot of power i have a link to the power supply i used in the description the taco booted but i also wanted to make sure it stayed cool so i plugged this 5 volt noxio fan into the taco's usb port even without active cooling in open air the controller chips only reached around 45 to 55 degrees celsius so i'm not too worried but i did run a stress test on the pi and it reached 79.8 degrees celsius but it never actually throttled in my tests if you want the taco to stay cool i'd recommend a heat sink and a fan i logged in and checked what i could see on the pci express bus ls pci tells you everything about the pci devices on your system and it'll even tell you if there's a kernel module loaded for them raspberry pi os doesn't include a lot of the drivers you'd find in a typical linux distribution so it'll probably need some drivers and looking at the devices the pisces i can see there are five entries for the as media switch chip and then the jmicron sata controller and look at that it says the ahci kernel module is in use so it looks like the pios is already set up for sata support next up is the real tech ethernet controller if i look at the bottom i don't see any kernel module loaded for it so that means the pi doesn't have a driver for it yet i'll put a pin in that for now and move on to the last entry for the nvme drive it's using the nvme kernel driver so it should be good to go too and to verify all the drives are visible i'll run lsblk which lists all block storage devices on the pi and look there are all the samsung ssds sba to sbe and there's the nvme drive good old nvme 0n1 now let's get back to that ethernet controller i just spent a few months upgrading my home network so i have at least 2.5 gig ethernet everywhere and the taco's a perfect way to test that out file copies over 2.5 gig networks can get up to about 250 megabytes per second in ideal conditions which beats the 120 megabytes per second you get with one gig and unlike 10 gig networking 2.5 gig works fine in my home's older cat5e cable runs which means i don't have to pull new cables to get that faster speed now to get the nic working i downloaded the driver from realtek's website and after installing the raspberry pi kernel headers package i ran its setup script i checked if the nic was working by glancing at the d message logs and there it is the r8125 2.5 gigabit ethernet driver says it's loaded and it looks like if i plug in a network cable it shows here the link is up i wanted to make sure it was working at full speed so i also ran ethtool eth1 and that shows me it's running at 2500 megabits full duplex meaning i should be getting the fastest possible speeds through this taco to my network to make sure it's actually working though i turned to the trusty network benchmarking utility iperf and in a funny turn of events it was super fast at 2.35 gigabits from the taco to my mac but it was really slow and jittery in the other direction as it turns out the 10 gigabit transceiver i had was flaky at 2.5 gigabit speeds this is one of the many fun problems you run into with 10 gigabit networking and that's why i think that 2.5 gigs is the next logical step for most people redshirt jeff told me he'd rather go to a hundred gigs but i don't have the budget for that at least not yet let me know below if you think i should accept a fat juicy raid shadow legend sponsorship speaking of raid it's time to check on drive performance i wanted to test a number of setups to see what options you might have with a raspberry pi based nas i'm going to test drives individually then in raid 0 raid 5 and finally in raid z1 which uses zfs oh you didn't realize the pi can run zfs well it can zfs2 is easy to install on 64-bit pios you install the pi kernel headers and run this apt command it took a little while but i didn't have any problems now i'm not going to get into the weeds on how i set up my raid arrays how i set up my z-pool or how i ran my benchmarks there are very detailed links in the description but i will say that zfs offers a ton of interesting features and i might do an entire video on it at some point once i have a better understanding of all its moving parts it's kind of like systemd it does about 100 things from raid to being a file system to managing mounts and snapshotting and replicating data so i need to spend a lot more time using it speaking of spending a lot of time i spent the next couple days benchmarking and the results were a little surprising the nvme drive tested just as i expected the sequential operations basically max out the pi's bus so you get a little over megabytes per second and large block size random reads and writes are also pretty fast but when you get down to tiny file sizes the nvme drive pulls away a little bit neither drive is slow for a pi both are 10 times faster than the micro sd card but both drives would be even faster in a modern desktop especially the nvme drive which is really hampered by the pi's single gen 2 lane raid performance was also interesting i knew what to expect from building previous pi storage devices and all three raid types i tried were great for random reads raid 5 because it's using the pi cpu to do parity calculations slows down considerably on random writes and even though zfs raid z1 is conceptually the same its caching mechanisms seemed to boost the speed there pretty well small random reads is where zfs seems to really shine but everything falls apart on 4k random writes none of the raid setups that require the pi's cpu to crunch data are all that fast but these synthetic tests highlight the curse of the reviewer no benchmark is perfect there's no substitute for real world tests so i also tested samba and nfs file string i'm only going to share the samba results here for simplicity's sake but they're pretty representative all three raid setups perform well at least for read speeds i got more than a gigabit all around but i found out that cpu interrupts seemed to be causing some bottlenecks especially with nfs while i was copying eight gigabytes back and forth i monitored the pi with a top irq top and other utilities and found that one major bottleneck seems to be the fact that all the data has to be passed through the pi cpu before it can be written to the disk or sent out over the network and the pi cpu is no powerhouse i should note that all my testing is done with the default clock speed which is 1.5 gigahertz overclocking helps a little but what surprised me most was i thought zfs would perform much worse than normal software raid but it's actually not that bad and i found something else too if i used smaller files that could be cached through the pi's ram zfs especially was many times faster so i could actually measure file copies in the hundreds of megabytes per second under certain conditions and using a pi with more ram would definitely help but if you want a serious storage server a raspberry pi ain't that but it is adequate and the taco was rock solid the whole time the last thing i wanted to cover is energy efficiency at idle the board used 11 watts and at full tilt 18 watts with full size 3.5 inch hard drives it'd probably be a bit more but what surprised me most was that it's about on par with the big nas i currently use this asus store in my rack the asus store peaks a bit higher on usage but on average it's close to the same energy use and it gives me about 20 faster file copies anyways i've had a blast building this thing months ago when i built my own hacked together nas i said what i really wanted was one board that had everything on it and radza delivered as far as raspberry pi nasa's go this is the new gold standard you're not going to find a more feature-filled yet compact solution and i love how everything is integrated no usb to sata conversion no extra power adapters and getting faster ethernet is a nice cherry on top radza told me they should be shipping the board by itself for less than 100 bucks by the end of this year in q1 2022 they plan on releasing a 200 kit with the case the board and the compute module until next time i'm jeff gearling let's take a look or close let's okay that's so many s's in this test test i turned to the trusty networking benchmarking all three raid setups perform admirably admir i'm not going to use that word but if you want to twist words escape me

Channel: Jeff Geerling

Views: 174,359

Keywords: raspberry pi, radxa, taco, cm4, compute module 4, penta, sata, sas, nas, homelab, network, storage, attached, sbc, linux, mdadm, zfs, raidz1, raid, raid 0, raid 5, performance, benchmark, 2.5g, gigabits, ethernet, multi-gig, 10 gigabit, 10g, 1g, comparison, review, lambda, gpu, sponsored, networking, samba, nfs, pci express, pcie, switch, asmedia, rack

Id: G_px298IF2k

Length: 12min 57sec (777 seconds)

Published: Wed Nov 17 2021