SMALL Proxmox Cluster Tips | Quorum and QDevices, Oh My! (+ Installing a QDevice on a RasPi)

so you're currently running a single proxmox server and you really enjoy it but you want to expand to a small cluster so your virtual machines have more room for activities do you add a second host as a standalone proxmox instance or do you cluster the two of them do you go for three notes and build a full three node cluster do you need a q device what even is a q device and how do you install one if you need one in this video i'm going to cover tips and tricks specific to small proxmox clusters of two or three nodes i'm also going to go into how to install a q device on a raspberry pi and whether or not you should have one at all so here's the hardware we're working with for this video these are my trio of dell wise 50 60 thin clients that i used in the hundred dollar proxmox high availability video link up there for that video today we're going to configure them as a two node and as a three node cluster to show some of the differences in failover when using two versus three nodes is a raspberry pi it's not a new one it's a raspberry pi 2 and it's going to play the role of our q device and we're going to see how that affects our 2 and our 3 node cluster stability since it's kind of hard to buy a raspberry pi i'm going to show how to use this dell wise 3040 thin client which is very cheap as a sort of q device to help stabilize the two node cluster as well so for our first example we have two separate nodes pve-1 pde2 they're not clustered together as you can see we have a vm 200 here which is called test ubuntu 1 and we have another one on the other node that's also vm 200 called test ubuntu 2 so we can have vms with the same id on each system because they're completely separate they're not clustered together we also have to log into each system independently to manage them so both of these systems are up i can go ahead and start this vm it'll start successfully and turn green there we go it's running now if i go here and i disconnected so it's not working anymore there we go now we're unable to connect because the node is down so if i go back over here to the first node and i say stop not going to have a problem because the two nodes are unrelated nothing they do affects each other if we had a cluster we would lose quorum as soon as the second node is shut down the firstnet will be unable to do anything so now what happens if i want to migrate a vm from pve 1 to pve 2. well if we go here convert it to a template manage h a none of those are good options and we don't really have a good option here there's no migrate button because we can't do it so if we want to move this vm from pb1 to ppt we have to stop it then we have to go here to backup and do a backup so we're going to send it to our shared storage on the nas it's only got a six gig disk because i only have 16 gigs to work with locally done that was quick because it's so small so now we go over to pv2 we have a different vm but i had the same number 200 and when i click backups i can see that backup from the other vm 200 on the other proxmox host so you have to be careful that you don't mix up the numbers between the two systems in this case i could try to restore it but because it has the same number it's going to overwrite my 200 on the new system so we should probably cluster these nodes to make managing them easier but as long as we have vms on both nodes we can't cluster them so we have to either delete this one or back it up and restore it under a different number later so now that we only have vms on one node we can use it as the first node in the cluster and then we can add other nodes to it so we get the join information go over to pv2 go into cluster get the root password go so now it's going to appear that this never finishes but we just have to refresh because it's going to get a new self-signed certificate except the new certificate and now we have a cluster pve 1vb2 and we have our one vm that we kept because we deleted the other one so now from now on the cluster is going to guarantee that all vm numbers are unique across the cluster so now that we have a cluster what happens if we lose one node so we only have two nodes so if we go to data center h a we say our quorum is okay so we have number of nodes two each one has one vote so if we shut down the second one over here so now we shut down the second nodes second node is unavailable we still have two nodes now we have no quorum on node pve-1 so if i try to start up this vm here it says cluster not ready no quorum so we can't start the vm on pve-1 even though one is fine because it can't be sure that the cluster state is going to stay coherent if it does a start of em so because we only have two anytime we lose either one we effectively lose both of them so from the perspective of pve 1 it can't communicate with pve 2 but it doesn't know what's wrong maybe pve 1 lost its network connection and pve 2 is running just fine so if pve 1 goes and changes settings pve 2 won't know about that and i'll have to synchronize them later or maybe pve 2 is actually dead and pve 1 should do something about it but it can't tell so without something that can corroborate which node is actually dead or which node is actually having problems the cluster can never really be stable with just two nodes and nothing else so we have three options on where to go from here our first option is to give one of these nodes an extra vote so we can see here each node has one vote we need two votes to maintain quorum so if we give pve 1 a second vote then pve 1 can operate on its own because it would need 2 out of 3 votes in that case and if pve 2 were to shut down pv 1 would be fine but the inverse would not be true if we were to lose pve 1 pv 2 would not be able to operate so this does not give you high availability it's pretty easy to set up though so we just have to go to pve one so i booted back up pve 2 so pv2 and pv1 are both running so now we can make a change to make pv1 has a have an extra vote so we're going to add it scpe quorusync.com and so we just go down to here quorum votes two and we also need to edit the config version to increment it by one so it's currently two we're going to make it three we're going to save this make sure you don't save if you're in the middle of editing the file make sure you only save when you're all the way done all the changes are correct otherwise you could end up with a bad configuration file and course sync would be very angry so now that we've done that core sync should reload itself we go back here we see suddenly we have two votes now pve 2 is shut down so i should still be able to start vm 200 and looks like it was successful there goes starting so quorum is still okay all is good so our second approach is to add a third node and if you're adding an x86 based node it's easiest just to add a full proxmox install instead of adding a q device so i'm going to do that here so i got this dell wise 3040 thin client that i set up to run proxmox it only has two gigs of ram and 16 gigs of emmc this will be just fine as a third node we don't have to run any vms on this or if you use containers you could run some really small containers on here but we're going to configure this as a third node to balance the cluster so here we have pve 3040 little tiny 30 40. it's got a tiny cpu and it's using half its ram just running proxmox that's not a good sign but we're going to add it to our cluster so copy the join information and then refresh again because it'll get a new certificate yep so it got a new certificate because we added it to the cluster we continue we log in and we good so because this node is too small to really run vms we should prevent our vms from actually running on this third node because we're using it effectively as a queue device without having a q device so to prevent ha services from being migrated around onto our third node because our third node is just there to maintain quorum we're going to create a group that contains pve-1 and pve-2 called the h-a group so now that i've created that group in the h-a system when i go to manage h-a i can pick the h-a group and it'll only allow pve-1 or pve-2 to run this vm set that to started and now it should start up the vm automatically so if we were to put this vm on shared storage such as a nas we wouldn't have to do anything special but if you don't have a nas or you really want high availability storage without using a cluster file system your other options to replicate so i go to this vm and i say replication i can add a replication job so target is pve 2 and we say every 15 minutes that's the default and what that'll do is that'll copy the disk from pve 1 to pve 2 so that there's always a copy of the disk on both systems and that way if it ever has to migrate from pve 1 to pve 2 and worst it's lost 15 minutes of data now if you have more than two nodes in your cluster you have to add all of them here so if pve 3040 were capable of running this vm and not a not excluded by the pool rule then we would have to copy it to all of them as well and this also means you're using disk space on all of your nodes for all of your high availability vms so say this was home assistant and it's only maybe 10 or 15 gigs that might not be too bad to have a copy of it on every system but if this is a larger thing say your security nvr or something like that you might not want to keep three copies of it everywhere but if you don't have shared storage or your high availability requirements require you to have highly available storage then you have to duplicate the drives to all of the nodes your other option is to use a cluster file system like sap and stay tuned for the next video on that so we can see here replication was successful that means that we should have a copy of vm 200 on pve 2 which we do but there's no vm 200 on pve yet because it doesn't migrate we could force it to migrate or we could kick pve one offline and see if it can migrate itself automatically so pv1 lost power so now it's offline it still thinks that vm 200 could be running but we're not really sure we click on h a so quorum is still okay so the cluster can still operate so now it's fenced to the vm that means it's given it enough time that it's pretty sure pve one is actually dead it's ready to try to migrate it so now it's decided to start the vm on pve 2 and because we don't have shared storage setup for this vm it had to use the replicated copy which could be up to 15 minutes old but our high availability service is now running again and only took us a few minutes to get it switched over with only a few minutes of downtime not too bad so if you do want to use a q device let's set one up on a raspberry pi so we're going to choose raspberry pi os lite unless you want to use desktop you don't need to i got a 16 gig sd card i'm gonna click a little thing here set the host name we're gonna enable ssh and it requires us to set a password for the pi i'm not going to configure wi-fi because i'm using wired ethernet and let's go now we let this run okay let's get this sd card into the pi and get it booted up so now we're on the pi we've logged in over ssh so i'm going to do a couple things so in order to provision the q device in our proxmox cluster proxmox wants to log in to the pi as root to do some configuration so we need to set the password for the root account and then enable ssh login as root so so the new password for root so now root has a password now we need to let them log in so we need to edit etsy ssh sshd forgot to use sudo so sudo nano etsy sshd config permit root login is the line we're looking for we need to uncomment it by default it says prohibit password which would mean that if we enabled this root would only be allowed to log in with an ssh key instead of with a password we're going to set this to yes that means that root can log in with a password not the most secure thing to do but we can revert it back when we're done by changing this back and commenting that line out so now we need to restart sshd and that should allow proxmox to log in as root we can also start installing the coresync qnet and q device packages so we're going to do sudo apt update and upgrade like we love to and then we wait for apps to update and app to upgrade raspberry pi is not the fastest device ever built so it'll take a while so now we need to install chorus sync qnet d or async new net d and core sync q device yes okay now we're done with the raspberry pi we move back over to our proxmox cluster so we're back at the two node cluster we set up earlier before we added the third node so on proxmox we also need to add support for the q device and we do that by running app install chorus sync new device and we need to do that on both pve1 and pve two so now from either either one we need to provision the q device from the proxmox system so to do that we call pvecm q device setup then we give it the ip address not the dns name of the raspberry pi and then dash f and dash f will force it to overwrite any settings that are already there just in case so we need to trust the fingerprint yes we need to give it the root password of the raspberry pi that we just set up we let it do its thing it's going to reconfigure chorus ink and then all should be good so now we notice that the new node does not show up here as a node in proximox because it's not a proxmox node it's just chorusync we still only see that we have two nodes and the cluster is corey so again here we only have two nodes we still have quorum so what happens if we shut down pve two so with only one node remaining in the cluster if we don't have the q device we should lose quorum so pve two is now shut down and we have one node but we're still chlorate and that's because the q device is there to provide a vote so here on the aj screen we can see the master is now pve one so because pve two is dead the q device is saying i can talk to pve 1 but i can't talk to pve 2. so i'm going to vote for pve 1 to be the master because i believe it is still healthy and that allows pve 1 to be a soul master by itself because the q device is there to prop it up with votes so again like the situation where we had three proxmox nodes but one was only used for quorum we can still migrate between these two nodes we still have to replicate this node using replication we we don't get away from that with the q device the only thing the q device gets us is not have to install proxmox on the third node that's only for voting so the only reason you would really use a q device is if you're using a device that can't run proxmox natively or you want to run it in a container on your nas or something other some other weird thing like that so now we're going to bring in a third full node to the cluster so we have pve 3 which is our third little thin client so we're going to join this to the other two get the join information so again because we joined the cluster we got a new self-signed certificate and we log in again now we have a full three node cluster all three of the nodes are full nodes they're capable of running any of our vms so now we don't need to add a mini node or a q device we can just let all three nodes share the load between all the high availability vms we don't need to set up an ha pool because all of the nodes by default are part of the pool we want this node to become aj we can say manage aj don't give it a group and it'll use all of the nodes say add and go ahead and start so what happens if we were to add a q device even though we already have three nodes and we don't need them could the q device help us here so we're going to install q device again on all three nodes now we're going to add the q device okay so now it's done now if you're in pbecm status this will give us a bit more information so now it says we have five votes total in the system because the q device is providing two votes so the quorum is three that means that if we were to lose the q device we would not be able to sustain another failure in the cluster and this is by design proxmox only intends you to use a q device when you have an even number of nodes if you have an odd number of nodes it will intentionally give extra votes to the q device so having a q device in a three node cluster which is already stable on its own can only hurt stability because now you can't afford to lose the q device and another device with an even number of nodes the q device will always get one vote so having the q device can only help support the cluster stability because with the q device you can afford to lose any device in the system whether it's the q device or a regular node and the cluster will still operate so what have we learned today well if you have a two node proxmox cluster you probably want to do something to keep the cluster stable if you lose one node you could add a q device on a raspberry pi or you can add a third small node and don't allow it to join the h a pool either of these solutions keep the cluster chlorate if you lose any single node if you have a three node cluster where all three nodes are running vms and containers you should not add a key device adding a q device to a three node cluster makes the cluster unable to sustain a failure greater than the q device itself so you can lose any one regular node and still be fine but if you lose the q device you now can't lose any of the other nodes also so effectively you're worse off than just having the three nodes by themselves where you can lose any one node and be fine we also touched a bit on the replication requirements for small clusters one option is to use shared storage so all of the nodes have access to all of the vm disks but this creates a single point of failure at the network storage the other option is to use local zfs storage and replicate the vm disks across all nodes in the cluster if you use local lvm storage that's just not an option for high availability so replicating with zfs might work well for high availability and it prevents you from having a single point of failure on the network storage but it does take up quite a bit of space especially if you have large vm disks that you don't really want to be duplicated three times stay tuned for the next part of this series right hyper converge this cluster using these 128 gig flash drives we're going to give every node its own storage share the storage across the whole cluster in a way that keeps high availability resources replicated but doesn't triplicate all of our data thanks for watching bye

Channel: apalrd's adventures

Views: 21,002

Keywords:

Id: TXFYTQKYlno

Length: 22min 52sec (1372 seconds)

Published: Thu Apr 14 2022