Why would you build a Raspberry Pi Cluster?

after i posted my raspberry pi blade server video last week lots of people asked what you do with a pi cluster many asked out of curiosity while others seem to shudder at the very idea of a pie cluster because obviously a cheap pc would perform better right before we go any further i'd say probably ninety percent of you watching shouldn't build a pie cluster but some of you should why well the first thing i have to clear up is what a pie cluster isn't some people think when you put together two computers in a cluster let's say both of them having 4 cpu cores and 8 gigs of ram you end up with the ability to use 8 cpu cores and 16 gigs of ram well that's not really the case you still have two separate four core cpus and two separately addressable eight gig portions of ram storage can sometimes be aggregated in a cluster to a degree but even there you suffer a performance penalty and the complexity is much higher over just having one server with a lot more hard drives so that's not what a cluster is instead a cluster is a group of similar computers or even in some cases wildly different computers that can be coordinated through some sort of cluster management to perform certain tasks the key here is that tasks must be split up to work on members of the cluster some software will work well in parallel but there's other software like games that can only address one gpu and one cpu at a time throwing flight simulator at a giant cluster of computers isn't going to make it run any faster software like that simply won't run on any pi cluster no matter how big luckily there is a lot of software that does run well in smaller chunks in parallel for example right now in my little home cluster which i'm still building out i'm running prometheus and grafana on this first raspberry pi and monitoring my internet connection indoor air quality and household power consumption this pie is also running pie hole for custom dns and to prevent ad tracking on my home network this next pie is running docker and serving up the website pidramble.com and the one after that is managing backups for my entire digital life backing up all my data off-site on amazon glacier i also have another set of pies that typically runs kubernetes but i'm rebuilding that cluster right now but there's tons of other software that runs great on pi's pretty much any application that can be compiled for arm processors will run on the pi and that includes most things you'd run on servers these days thanks to apple's adopting arm with the new m1 max and amazon using graviton instances in their cloud i'm considering hosting next cloud and bitwarden soon to help reduce my dependence on cloud services and for better password management a lot of people run things like home assistant on pi's and there are thousands of different pi based automation solutions for home and industry but before we get to specifically why some people build pi clusters let's first talk about clusters in general why would anyone want to build a cluster of any type of computer i already mentioned that you don't just get to lump together all the resources a cluster with 10 amd cpus and 10 rtx 3080s can't magically play crysis at 8k at 500 fps well there are actually a number of reasons but the two i'm usually concerned with are uptime and scalability for software other than games you can usually design it so it scales up and down by splitting up tasks into one or more application instances take a web server for instance if you have one web server you can scale it up until you can't fit more ram in the computer or a faster cpu but if you can run multiple copies you could have one ten or a hundred workers running that handle requests and each worker could take as much or as little resources as it needs so you could in fact get the performance of 10 amd cpus split up across 10 computers but in aggregate not everything scales that easily but even so another common reason for clustering is uptime or reliability computers die there are two types of people in the world people who have had a computer die on them and people who will have a computer die on them and not just complete failure computers sometimes do weird things like the disk access gets slow or it starts erroring out a couple times a day or the network goes from a gigabit to 100 megabits for seemingly no reason if you have just one computer you're putting all your eggs in one basket in the clustering world we call these servers snowflakes they're precious to you unique and irreplaceable you might even name them but the problem is all computers need to be replaced someday and life is a lot less stressful if you can lose 1 2 or even 10 servers while your applications still run happy as can be because you're running them on a cluster now i mentioned that i'm running one instance of each of these applications on this cluster right now i'm planning on splitting up a few of them though and probably using kubernetes on the entire stack so i can have even better redundancy but having multiple pies and having good backups and automation to manage them means when a micro sd card fails or a pie blows up i toss it out and can have a spare running in a few minutes okay so those aren't all the reasons for clustering but two of the main reasons most people would consider a cluster over one computer but that doesn't answer the question why someone would run raspberry pi's in their cluster a lot of people questioned whether a 64 core arm cluster built with raspberry pi's could compete with a single 64 core amd cpu and well that's not a simple question first i have to ask what are you comparing if we're talking about price are we talking about 64 core amd cpus that alone cost six thousand dollars because that's certainly more expensive than buying 16 raspberry pi's with all the associated hardware for around 3 000 all in if we're talking about power efficiency that's even more tricky are we talking about idle power consumption assuming the worst case with poe plus power to each pi 16 pies would total about 100 watts of power consumption all in according to serve the homes testing the amd epic 7742 uses a minimum of 120 watts and that's just the cpu if you're talking about something like crypto mining 3d rendering or some other test that's going to try to use as much cpu and gpu power as possible constantly that's an entirely different game the pi's performance per watt is okay but it's no match for a 64 core amd epic running full blast total energy consumption would be higher 400 plus watts compared to 200 watts for the entire pi cluster full tilt but you'll get a lot more work out of that epic chip on a per watt basis meaning you could compute more things faster but there are a lot of applications in the world that don't need full throttle 24 7 and for those applications unless you need frequent bursty performance it could be more cost effective to run on lower power cpus like the ones in the pi but a lot of people get hung up on performance it's not the be-all and end-all of computing i've built at least five versions of my pi cluster i've learned a lot i learned about linux networking i've learned about power over ethernet i've learned about the physical layer of the network i've learned how to compile software i've learned how to use ansible for bare metal configuration and network management these are things that i may have learned to some degree from other activities or by building virtual machines on one bigger computer but i wouldn't know them intimately and i wouldn't have had as much fun since building physical computers is so hands-on so for many people myself included i do it mostly for the educational value even still some people say it's more economical to build a cluster of old laptops or pcs you may have laying around well i don't have any laying around and even if i did unless you have pretty new pcs the performance per watt from a pi 4 is actually pretty competitive with a 5 to 10 year old pc and they take up a lot less space and besides the pies typically run silent or nearly so and don't act like a space heater all day like a pile of older intel laptops but there's one other class of users that might surprise you enterprise some people need arm servers to integrate into their continuous integration ci or testing system so they can build and test software on arm processors and it's a lot cheaper to do it on a pi than on a mac mini or an expensive ampere computer if you don't need the raw performance and some enterprises need an on-premise arm cluster to run things like they would on aws graviton or to test things out for industrial automation where there are tons of pi's and other arm processors in use finally some companies integrate pi's into larger clusters as small low power arm nodes to run software that doesn't need bleeding edge performance or needs to be isolated from other servers another sentiment i see a lot is that it's too bad the pi doesn't have ecc ram well be ready to be shocked because the pi technically does have ecc ram check the product brief the micron lp ddr4 ram the pi uses technically has on die ecc now when people say ecc they mean a lot of different things and i'd say half the people who complain about a lack of it couldn't explain specifically how it would help their application run better but it is good in a server setting for a lot of different types of software and the pi has it or does it well not in the sense that expensive high-end servers do the on-die ecc can prevent memory access errors in the ram itself but it doesn't seem to be integrated with a pi system on a chip so the error correction is minimal compared to what you'd get if you spent tons of money on a beefy server with ecc integrated through the whole system so anyways those are my thoughts on what you could do with a cluster of raspberry pi's what are some other things you've seen people do with them and have you built your own cluster of computers before raspberry pi or anything else i'd love to see your examples in the comments until next time i'm jeff gerling before we go any further i should say blah blah blah that's definitely something i should say throwing fights fight fight simulator some software will work well in pearl when you talk about fun and you're not having fun that's just insane hands-on or by building virtual machines one one bigger computer but there's one other class of these users the raspberries taste like raspberries

Channel: Jeff Geerling

Views: 232,434

Rating: 4.9368935 out of 5

Keywords: raspberry pi, cluster, sbc, computer, server, hardware, amd, epyc, threadripper, arm, cpu, core, ram, hard drive, ceph, filesystem, kubernetes, k8s, rack, homelab, uses, why, setup, rtx 3080, gaming, games, 3d rendering, render farm, fast, speed, aggregate, manage, ansible, automation, linux, networking, physical, mount, blade, backplane, internet, web server, turing pi, cm4, compute module, uptime, scalability, deployment

Id: 8zXG4ySy1m8

Length: 10min 30sec (630 seconds)

Published: Wed Aug 11 2021