10 Watt HA Proxmox Cluster ft. ZimaBoard

there's no shortage of ways to run servers in your home lab from Raspberry Pi's to 64 core monsters from Windows server to on raid from a single system to a cluster using kubernetes or as we will see in this video a proxbox cluster that's right today we are going to take a look at the newest edition to my home lab a three node highly available proxmox cluster that sips just 10 watts and that's including the switch let's take a look so what is a proxmox cluster and why did I set one up well I've been using proxbox for years now as my main server hypervisor operating system and I have no plans on Switching and while I'm extremely happy with my proxbox server there is an issue if it goes down it takes everything with it now of course I have everything I need backed up but that requires me to manually spin them back up on my backup server which can be quite a bit of downtime with a broxbox cluster you can group a number of servers together into a pool or or cluster that lets you do some cool stuff one of those things is letting you run VMS and containers in high availability mode meaning that if a service is running on server a in the cluster and server a suddenly goes down the cluster will automatically spin that service up on another server in the cluster which minimizes downtime and eliminates any manual intervention now I don't really have anything running in my home lab that would be considered Mission critical if my lab goes down for a few hours or even a few days the world's still going to go around but that doesn't mean I don't want to have some of my services running in high availability mode plus it just sounds like a fun thing to do and that's pretty much why I'm doing it essentially because I can and I'm a nerd all right let's talk about the hardware that I chose I wanted something compact low power and x86 based this eliminates Raspberry Pi's or other arm-based boards as well as just buying some Dell optiplexes or building my own servers there's no shortage of many PCS that would suffice for this project but I went with three zema boards if you don't know what azima board is then feel free to check out my video where I review one but the quick answer is that it's a tiny low power x86 based board with solid i o includes two actual SATA ports and a full exposed pcie 2.054x slot these are their top models which come with a quad core Intel and 3450 and 8 gigabytes of RAM so yeah it's not going to blow you away with its performance but for a compact test cluster these are great at the time I'm making this video they have a pretty good sale going on so I'll link everything down in the description below if you want to snag one for yourself also each one of them has a one terabyte SSD connected to them for ceph storage more on that later now you don't need a dedicated switch but I think it makes things easier and cleaner so I have them hooked up to their own little five port tp-link gigabit switch all of this sits on a custom stand that was handcrafted by Artesian Italian monks in the Swiss Alps overall I think I accomplished what I wanted seeing is how it's compact enough to sit on my desk it sips about 10 watts total at idle it's x86 based and all of this including the switch and storage comes in at just under 800 brand new could you do a cheaper absolutely you could buy a couple of older 50 PCS off Craigslist and call it a day but I was down to prioritize form factor and power draw over cost you may be different and that is okay cool so now what well set up a cluster you're going to need to install proxmox on each of these systems like you would any other device I'll be walking through this assuming that you know the very basics of proxbox if you don't feel free to check out some of my other videos on proxmox or if you want an actual good video then go check out techno Tim one thing I did change on my installation is disabling swap since I installed proxmox on the internal emmc storage which also required me to edit the install script I'll leave a little write up on that in the description once I got all three of them ready to go I logged into one of them and created the cluster which is pretty much just a single button click when this is done you can go into the cluster section and click join information which will display some information that you will use on your other devices to join your cluster then on your other devices click join cluster and paste that information from the previous screen everything is set up properly then you'll now see all of your devices listed in your side panel meaning you've successfully set up a proxmox cluster before actually spinning up any Services I wanted to get my storage setups and storage in a cluster is a little bit different when you're running services on a single server then it's easy to just spin stuff up on your local storage and you're good to go however that doesn't really work in a highly available cluster why is that let's think about it let's assume we create a virtual machine on server a in the cluster and use the local storage sure it'll work work but what happens when that server goes down that local storage is no longer available and that VM is gone now you can run a cluster without ha mode and manually migrate services from one server to another but it still doesn't protect us during an unplanned outage this is where shared storage comes into play and more specifically in my setup SEF if you don't know what Seth is that's okay because before this I really didn't know either and honestly I still only know the basics stuff is a block level storage solution that is designed to be used in highly available systems comes integrated into proxmox by default and only requires you to click a single button to get it installed on each server once it's running you can go into your SEF settings and add osds or object storage damage these are essentially disks that will be used to store our data and you'll need at least three of them for an aha system if you remember I mentioned each one of these zema boards has a one terabyte SSD Plugged In which gives us a nice little ha SF storage solution after adding each of these disks as an OSD we are able to create our ceph pool two parameters you'll want to make note of are the size and the Min size size tells us how many osds we want to copy our data to or essentially how many copies of our data we want to keep Min size specifies the minimum amount it will be okay with before it starts screaming us for our setup we will be using the minimum needed for an ha storage system which is a size of three with a Min size of two after this is created you'll now see that each one of our nodes in the cluster has access to our CEF pool meaning that we now have some shared highly available storage for our cluster pretty neat you could also just use external storage on a mass and connect via SMB or NFS this works but isn't really aha since if that Nas goes down you pretty much lose everything at this point we can now go ahead and spin up whatever we want in our cluster I went ahead and installed an Ubuntu desktop VM time to play around with certainly works but it's not the snappiest experience ever it has nothing to do with our self storage though and more to do with our extremely low power processors these things have a base clock speed of just 1.1 gigahertz and very limited graphical power I did only allocate four threads to it but still not the greatest experience however I still can use this as an opportunity to show off a really cool feature of the cluster that I mentioned before and that's the ability to migrate a live VM to another node in real time simply right click the VM select migrate and pick which node you want and just like that it'll start moving once it's done it'll now be listed under the new node and using its new resources this makes Rolling updates to our cluster so much easier as you can migrate all of your services perform updates or maintenance you need and then take the node down then once it's back up you can migrate everything back and you're done no downtime required you can stop here if you want but a lot like I said we wanted a highly available cluster which means we will have to set our services and resources to be highly available this is where I remembered techno Tim's ha setup he has his kubernetes cluster running in an aha proxmox cluster and I had the totally original idea to copy him so I spent what's up like 20 years setting up a virtualized k3s cluster within the proximox cluster along with a Debian lxc container for some Docker Shenanigans I was now ready for all that highly available goodness setting up your resources for ha is pretty easy just go into the ha section under a Data Center and specify which resources you want to become highly available I like that you can pick and choose which ones you want and it isn't really an All or Nothing kind of thing because there's always some services that I really don't need to be up all the time once you've added them then that's kind of it there are some more settings you can play with but for a simple setup you can just leave everything to default to test this out I shut down one of my nodes to see if it would actually move my kubernetes VM over to a new node after about a minute of the node being down the VM suddenly popped up in a new node and was up and running sure it may not have been as quick as ha within a kubernetes cluster but we're dealing with a full VM here not just a single pot I want to see if this VM would go back to the old node once I booted it back up but unfortunately it did not I'm not sure if there's a setting to change this but it's not really a huge deal I just manually migrated it back on my own all right well we've seen that the ha functionality within the proxmox cluster works and that's awesome but what are my thoughts on virtualizing a kubernetes cluster on this Hardware specifically it's okay I had a feeling we would be limited on resources for two reasons one is because these are super low power boards with only eight gigabytes of RAM each and two running in ha cluster means that you can't just allocate 100 of your resources since if one node goes down there's gonna be nothing left on the others to pick up the slack and you can see just hosting our k3s cluster with Rancher already has this using more resources than I'm comfortable with don't get me wrong it's still usable but I wouldn't throw anything intense about it honestly for this setup you could probably get by with just running Docker in an lxc container that will get the benefits from being highly available in your proxmox cluster sure you will see some actual downtime if a node goes down as it takes a minute or two to migrate which isn't really an issue with our k3s cluster but if you're like me then you're okay with that so what would you run on something like this well like I said nothing intensive like a big Plex server that requires a lot of transcoding or a gaming server or you know machine learning or anything like that but there are plenty of useful less resource hungry services that you can run personally I'd throw some networking stuff on here like pie hole VPN server load balancer and a reverse proxy then you could host some other stuff like a WordPress site your own URL shortener an R stack etc for your home lab you could run your logging systems and uptime Kuma to keep tabs on your entire lab all of these things definitely benefit from being in an ha environment all right overall thoughts on this setup I'm very happy with it like I mentioned this specific setup is not for everyone I know that but I wanted low power zero noise and a small footprint so that I could keep it on my desk so I was willing to pay a little extra for that we've definitely accomplished this as it pulls just 10 watts at idle in around 20 under load there are no moving Parts not even a fan and as you can see it is very compact for what it is I like this setup so much that I've honestly already started looking at planning a new production proxbox cluster that will go in my server rack that one will have much more power don't worry and make sure you're subscribed so that you can see how that actually goes let me know if you are running a proximox cluster and if you have any tips for me moving forward I love hearing from you guys but that is it if you like this video then drop a like leave a comment and I want to give a huge shout out to my patreons and YouTube members you guys are my highly available deployment within my highly available kubernetes cluster within a highly available proxbox cluster just always there for me you guys are doing okay well I'm done here thank you guys so much for watching and I will see you in the next one [Music]

Channel: Raid Owl

Views: 207,106

Keywords: cluster, homelab, proxmox

Id: JfZuZ6zE7AI

Length: 12min 41sec (761 seconds)

Published: Sat Apr 15 2023