Proxmox 8 cluster setup with ceph and HA

in this video I'm going to show you how to create a proxmox 8 cluster using three nodes in this case I'm using three HP Mini PCS and if you haven't seen my previous video I'll link it up here I show you how to install and set up proxmox so without further Ado let's jump right into it so I have prepared three notes here this is just called pve1 pve2 and pve3 first thing we do is we log in so this is a root account that you created during the installation process it's normal that you don't have a subscription but we're using the Community Edition so the repos are already set up on these notes we need to create a cluster and I'm going to use pve01 as the master of the cluster we're gonna click on Data Center and then we're gonna click on cluster and then we're going to create a cluster so create cluster here the name you can put anything you want I'm going to put PVE cluster and the network that we're going to use is the network of the note perfect so we're just going to click on create okay we see that all the tasks were okay so now we can close now we need to extract the join information so we click on join information and it's going to give us the basically the API key that we need to use in the other nodes so that we can join so we're going to click on copy information now we're going to go to our second node now we're going to go back to Data Center we're going to go to Cluster and then we're going to click on join cluster here's where we paste our join information from our previous node and we need to input the root password of node 1. perfect now we click on join PVE cluster while this one is joining let's do the third one now we click on data center we click on cluster then join cluster again and we copy our join information we put the password of Node 1 remember we click on join okay so it says join request okay now we can close you might see this error here but it's normal we'll lose connection to the nodes for a few seconds but all you need to do is refresh refresh this one too the third one okay we can click on advance and proceed don't worry about that warning put in the password again and now as you can see we have three nodes in our data center it doesn't matter which node you connect to it can be Node 1 node 2 or node 3 web UI you're going to be able to manage the entire cluster from any of those web uis so as we can see we have the three nodes here three nodes and we have our three notes here too so at this point we already have a proximox8 cluster but we want this cluster to be h a h a stands for high availability and to do that we need to configure set is basically a shared storage that the three nodes are going to use to run VMS this will enable us to migrate VMS from one node to another without any downtime or in the case that one of your notes fails the VM can be automatically restarted in a new node so let's configure Sev the first thing that we need to do is install save on the three nodes and to do that we're going to go to node one we're going to go down to theft and as soon as you click on Save you're gonna get a message telling you that save is not installed would you like to install it now we're going to click on install save now if you have a subscription you can use the Enterprise repo to install CEF but in this case we're using communities so we're going to change the repo to a no subscription now we're going to click on start Quincy installation this is going to go and fetch all the packages that we need to install here we just need to click on why to say yes you want to continue yes we want to continue and it's going to install all the packages okay as you can see install safe Quincy has been successful now we can click on next now the configuration in an Enterprise environment you would want to have a second Network only for theft save needs low latency and very high throughput to keep all the disks synchronized in this case I'm going to use the same network as the host just because I don't have a separate Network for that but for our demo and for our environment it should be good enough so now here we can click on the network that we want to use which is the local network of the node and then for number of replicas we're going to go with the default which is three number of replicas and have at least two okay now we can click on next and as you can see the installation is successful of course there are a couple of steps that we need to finish so that we can have save up and running I'm going to go through the installation on the other nodes I'll be right back okay so now Seth is installed on all three nodes the next thing that we need to do is we need to configure the OSD what OSD means is we need to select the drives that we want to use for our self configuration in this Mini PC I have a 500 gig SSD that I'm going to dedicate for Sev this is replicated on each of the nodes so I have 500 gig SSD on each one of them so let's now configure the OSD we click on pv01 we go down here and then we click on OSD and here we're going to create OSD okay so as you can see it automatically detected my drive which is the only drive that is available if you have any partitions or anything on this drive just make sure to wipe them and delete all the data and all the partitions that way proxmox will be able to recognize it so now we can click on on the drive and here basically there's nothing else that we need to do we're going to leave all of the all of the settings by default the defaults are pretty good so now we can click on create so now that's creating OSD and OSD on SDA okay so we click reload we can see that one of our drive is there now we're going to go to pve02 we're going to do the same create OSD we're going to select our 480 gig SSD and we're going to leave everything by default and create give it a second for it to load and to configure the drive and if we reload we can see that now we have our second drive here now we're going to go through a third one pve03 create OSD same Drive and we can click on create click on reload and as you can see we now have three OSD drives one in each of our nodes so now the next step that we need to do is we need to create a pool basically a pool is going to be the shared storage that the three nodes are going to use to store the VM disks let's go to pve01 go down to save and then we click on pull and then we click on Create and the name of the pool can be anything but I'm just going to call it PVE storage and the same default Supply size of three minimum replica of two and the rest can be left by default so now we're going to create our pool okay so now we have a PVE storage pool this is the place where we're going to create all the VMS that we want to have high availability this you only need to do it on one of the nodes and as you can see if we click if we open the other nodes there's going the PVE storage is there you can see PVE storage on node 3 on node 2 and on node 1. now the last component of the cluster that we need to set up for Sev is the monitor the monitor is basically a node that is just checking all the components of SEF and making sure that all of them are working properly that all the data is being synced correctly and that any part of the save environment is functional by default there's only one monitor so if we go here to pve01 and then we go down to Steph and then we click on monitor you can see that pve1 is the only monitor at the moment we want to have more than one monitor because in the case that pve01 is the node that fails then our save configuration or our syncing is not going to be able to check for everything so if Node 1 fails basically what happens is Node 1 is hosting the monitor the data is going to be out of sync and then it could get very messy so that's why we're going to add a monitor to the three nodes to do that we're going to go to node 2. we're going to go to Sev and monitor again and then we're going to click on create we're going to create a monitor on host pve02 yes that's what we want so we click on create and that's good you can see pve02 is there now we're going to go to pve03 and we're going to do the same thing we're going to click on on there and then save and then click on monitor then we're going to click on create and select pv03 and create so now for the fun part we're going to migrate one VM from one node to another and we're also going to simulate the failure of one of the nodes and we're going to see how resilient this cluster is okay so let's start with the first demo we're going to migrate one VM from Node 1 to node 2. in this case it's just a rocky 9.2 Linux VM to see how efficient this migration is we're going to start a continuous ping and see whether the machine goes down or not first we need to know what's the IP of the machine so we're going to log in really quickly here check for the IP so it's 192.168.0.219 so we're going to start a ping for that so now let's start our migration to migrate we need to click on rvm here and then we're going to right click on it and then we're going to click on Migrate and we're going to say the Target that we want to migrate to is pve02 so we're just going to click on Migrate let's monitor our ping here okay so migration is completed and as you can see we didn't lost any pings or or the machine was seamlessly moved from one node to another we can we can see here a little bit that the latency increased a little bit so now if we go back to our proximox cluster here now we can see that our VM is running on pve02 there's no downtime if we check the uptime of the machine it's still off so this is how a cluster works we can move VMS from one to another okay so now for the second demo in the second demo we're going to simulate a node failure we're going to completely turn off or Force shutdown one of the nodes to simulate a failure and we're going to see how long it takes for the proxmox cluster to bring that VM back in another node before we do that we need to configure the cluster or the data center to know which VMS we want to have on ha or high availability so to do that we're going to click on Data Center and we're going to go down and click on ha and here we can add resources resources means like containers or VMS or anything that we want to have in ha so we're going to click on ADD and we're going to click on VM and we're going to click the VM that we want to have high availability and request State it needs to be started so we're going to click on ADD by doing this now our Rocky server is marked as a high available VM now we're going to simulate a node failure right now the VM is running on node 2. we're going to simulate a node failure I'm going to start a pink here of 182 168.0.219 okay so we're pinging our VM I'm also going to start a timer to see how long it takes a cluster to recognize that one of the nodes it's down or failed and how long it takes to bring back the VM let's go into pve2 let's go on the shell and we're going to completely Force shutdown this node so for that we do power off and then we do dash dash Force dash dash force and this will kill node 2. I expect our VM to completely stop pinging on here and let's go okay so the VM stopped let's start the timer and let's see how long it takes for the VM to come back alive okay so we can see now in pve01 we have our VM there is it pinging not yet but it's starting we are at a three minute Mark okay let's see if we can see something oh and as we can see the VM is bringing back up into pve01 without user interaction it took around four minutes for the VM to come back online usually the time of of how fast this happens it depends on the type of failure and I also think it depends on your hardware and the speed of the network of yourself cluster drive my configuration is just a one gig and it's a shared network in this demo it might be longer but in an Enterprise environment this can take up to two minutes and not four minutes but as you can see we have a high available cluster and a fault tolerance cluster 2 where our VMS can spin up back again in another node even after one of the notes is completely lost and you can migrate VMS from OneNote to another node without any downtime so with this now we have a high available fault tolerant proxmox 8 cluster running on three nodes thank you for watching and if you enjoyed this video make sure to give it a thumbs up and subscribe to the channel if you want to see more content like this leave any questions or suggestions in the comment section below and I'll see you on the next one foreign

Channel: Distro Domain

Views: 9,329

Keywords: proxmox 8, proxmox, proxmox ve 8.0, proxmox ve 8, how to install proxmox, how to install proxmox 8, how to setup proxmox, how to setup proxmox 8, proxmox insallation, proxmox installation and configuration, proxmox tutorial, proxmox passthrough, proxmox gpu passthrough, gpu passthrough, proxmox cluster explained, proxmox cluster mini pc, proxmox cluster ha, proxmox cluster with ceph, proxmox 8 cluster ceph, ceph configuration, what is ceph, how to install ceph

Id: Mz-nXlqovLI

Length: 14min 12sec (852 seconds)

Published: Fri Sep 15 2023