Setting Up Proxmox High Availability Cluster & Ceph

hey guys what's going on it's Don here from Nova Spirit Tech and welcome back to the channel and today I'm going to show you guys how to create a ha or high availability cluster on proxmox so let's get [Music] started one of the cool things that you can do with proxmox is actually set up a ha cluster which allows you to migrate your VMS from one node to another and you also have shared storage now this is very similar to vmware's vs sand so if you know about that this is basically the same concept now I am doing this on a VM so I actually have three proxmox VMS inside a VM so the speed of this is not accurate because you should be running this on actual Hardware but for the demo I'm actually going to just show you through a VM now there are some requirements to this you do need obviously three servers but also on those three servers you do need additional storage so the original storage that you're using to install proxmox on will not work you will actually have to add an extra hard drive or different type of storage where your proxmox is not installed on now this is also very Hardware dependent obviously if you have something that's 10 GB Network ssds that will be better for this cause because you are sharing everything through the network interface now with that being said let's jump into the desktop so here like I said I do have it nested inside a proax environment and then I have three more proax environments as well inside of there so speed is definitely a concern because I don't have that much resources to run all three of these but I am able to get at least the demo through to show you guys what we need to do to get these all hooked up up now I am going to leave a link to the high availability website this has a lot of information if you have any questions about migrating freezing or stopping a service or yeah anything like that it's all answered in this document over here through proxmox high availability uh site now to begin um we do have three proxmox and I could use any one to start up a cluster that's the first thing we need to do so I'm going to head over to data center right over here go into cluster and then I am going to create a cluster now from here we just need to name the cluster name so I'm just going to call it prox cluster one and then from there we can just hit create I'm going to leave everything as default and this is quick it just takes a few seconds to create once you have task okay then you're going to see that this joined that cluster now we do need the other ones to join so we're going to hit join information and then we're going to grab this basically copy information head over to prox Max 2 and then in their data center we are also going to go to Cluster but this time we're just going to join it instead so we're going to join the cluster here paste that you see how it pulls up all the IP information everything I need all I need to type in is the root password over here join give this a second and everything's going to look like it failed but it actually didn't see if I go into prox max one I actually have the second node right here it joined already and then over here again you just need to refresh this site so I could close this out refresh this with crlr log back back in and now you see the two clusters I'm going to do the same thing again to the third one and you do need a minimum of three clusters just to get the Seth working for this migration to work so you do need three so I'm going to head back into cluster join cluster and I'm going to paste that in there pop in the password again give this a few seconds and you would see pox 3 join the network as well so now we could see all three of them now in this state it's technically like a manager State you're actually able to manage all different Pro boox nodes if you want to but as far as migrating through here you cannot do that yet until you set up a SE which has to do with storage so now I am going to jump back over to proc 3 and then refresh this and you can see you can log in here and you'll see the same information as proc one and proc 2 so doesn't really matter for me logging into this one but let's just start working off of three now here I have my prox cluster I got prox Max One prox Max 2 you can see I can look at everything I want in here and to start off with seph I have to go to each one of these and install Seth now there is reef and there's Quincy but to be honest I'm not exactly sure every intricate difference between the reef and Quincy but I do know that reef is slightly faster so I'm that's what I'm going to be using now in Enterprise I'm going to use no subscription and then start Reef installation now I'm not going to touch the other ones yet even though I still need to do those the reason being is that the first one that you're install SEF on you will go through the configurations and change the IP address the other ones will just obtain the information you don't have to configure anything so here I'm just going to wait for this one to finish before I jump into the other ones and configure set for those all right there we go once that's installed we could hit next in configurations if you do have a different network I would join that but for now we're just going to keep as default on the network that I'm currently on which is the 10.20 30 Network and then I can hit next and that's it it will tell you what you else you need which is monitors OSD and then the pool so we're going to be going through each one and one of those steps now I'm going to hit finish on that and you can see status is okay for this and then now I have to go to the other ones and go over to SEF install SE change this over to no subscription and you do have to keep it the same start installation hit yes on this let that go I'm going to go to the third one hit SEF on this install SE change this over to non subscription start install hit yes on this and I don't know for some reason why there is this weird little thing that won't go away now but regardless okay so the second one is done I'm going to hit next and you can see there's no configurations because it's already configured and then that's it I hit finish it tells me the same message I have to do now I have a warning message because I have only two out of the three of Seth and you see OSD pull default sizes three let me finish this one hit next configuration's already done that's what I need to do so now I'm going to go back into Pro 3 again it doesn't matter which one you're on oh I am going to go over to prox uh 3 over here all right so from here I'm going to go to OSD create OSD and then you're going to see the disc I added which would be dev. SDC I'm going to hit create I'm going to leave everything as default there's nothing really needs to be changed now because I can manage the other ones from this I'm just going to go to proc 2 scroll down to OSD create same thing and again if you go into Advance there are some things that you can do but I would leave everything as default and then on the third one or prox one I would go scroll down again and do the same thing create OSD and there we have it if I reload I should have proc 3 proc two give it a few more seconds and then proc one right over here so now that we have our OSD we need now need to create monitors so you could see that Pro 3 has the mon already because that's actually where I origined from and all I have to do is go into Pro one create the Monitor and then Pro two create that monitor and now we have all the monitors running so now next thing we need to do is create a pool so we have everything that we could store so we're going to create this pool I'm going to name this prox pool size of three which we have minimum size two and we're just going to leave everything as default and we're going to hit create all right as soon as this is created every one of these will see this prox pool you see this prox pool prox pool so now moving forward any VM will be dropped into this will have ability to migrate over to another server because you're sharing all the data within these three this is what creates the failover when something does go down or if you need to migrate anything that you want now there are a couple of things that you can set up ahead of time which if you head over to Data Center you do need to set up groups for the ha there's a couple of things over here for ha where you could uh set up so basically you want all three of these nodes to be available and you want to name this um procs group or something anything you want and then hit create you can set the priority level to see which one you want to move the service to in case you do have one more server that's stronger than the other and you want everything to move that one first priority order you can there are a couple things you could do with like restricted or nonf fallback or no fallback and in restricted meaning that say you have certain VMS that are attached to certain Hardwares like a video card um you would want to put it in a restricted group this way it won't purposely move that out of the way and breaking the VM because the hardware is missing no fallback would be another one where you don't want it to move to the strongest available server uh when it becomes available so there's a couple of things about this if you want to look into it again this documentation will tell tell you everything and then as far as like um containers go there's other states that you could put this in like uh started stopped uh disabled or ignored so let's create a CT real quick and I'm going to go over to local VM CT templates I'm going to download one real quick so I'm going to go to a template and download to 22 standard all right this should take me only a few seconds there we go now now I can create a CT so I'm just going to run right through this uh I'm going to call this ubu test give this a password next templates I'm going to choose that standard data the dis is 8 gigs that's fine CPU one memory half gig that's fine Network I'll choose DHCP next DNS confirmed and then finish I'm not going to start it after I create it because I want to show you guys all right now that Tas okay I do have it on prox Max one and when you're setting this up you have to go over to the top right because right now it's not on ha state so it won't do any migrations or anything if something was to fail so you do have to go into more manage ha and then from here you could set up the group that you want which is the group we created and then the state that we need you see start stop ignore and disable like these two will not affect this VM if something was to happen it's the two that you want is on top it's either when it's mov it'll re try to restart or I'll keep it in a stop state if it moved over to another V to another node so there's again more details about it in this website with high availability but once you do that you add it to that group and there you have it this will now fall under the rule if um the proxmox was to fail or the node was to fail it'll move over to the next node all right and for the moment you guys all been waiting for is the migration so right here we have a Ubuntu test container and if you take a look at the resources is it's actually on the procs pool so it's shared pool between all of them and for now I could just right click and go over to migrate and then I can migrate to a second procs now before I do that what I do want to set up is a little test now it actually is a little bit better on VMS where they could do a live mation versus container where they will shut it down and put it back up but I don't have enough resources to really run a full VM on here because I'm actually playing around with a VM within a VM like Inception so what I'm going to do is pop into here and grab the IP and this is 208 so I am going to pull up a terminal ping 10.20 do30 208 and then because I'm using Elementary OS I'm able to actually use picture and picture so I'm just going to grab this little bit here and I have a little image down there and then now you could see it's pinging and I am going to do a live migration so I'm going to go right click migrate and I get to choose the Target that I want which will be proxmox 2 and migrate now because I am on a nested VM it is not as fast as I want it to be so if it's on real Hardware it's actually going to be a lot faster now this does say task okay which means everything has been moved over and it's still pinging right now you can still see it's live it's pinging this gets disconnected I kind of stopped pinging I think because it's not moving there you go I lost three pings and then the pings came right back and that's it so it actually works out pretty quick even though if it's in this environment now there are a couple of things that you can do here which if you go to Data Center and you could set your options up for how you want the migration settings to work so right now we have our default settings for migration and if I go into edit this there's actually a few type oops not this this one sorry uh ha settings and if I go edit there are a few different types of uh policies that you could take so default is conditional conditional means it's going to detect whether it's shut down or reboot and then do things appropriately or move it over appropriately so first we have freeze which means do nothing just when the system goes down or fail over or breaks it's just going to freeze nothing's going to happen it's just like you only have one node there so in case you know what you're doing and you're going to shut down the uh server and you don't want any of them to migrate to another server that's when you would use freeze failover is when you know you got um you didn't give it a shutdown command it got unplugged the network got uh unplugged it I don't know it's unavailable it will fail over to the Nyx available um uh node and then on Migrate which migrate is also in the in the conditional migrate means anytime you uh force a shutdown or you write you know you go into the proxmox node and then shut it down it will migrate everything over to another node and then conditional like I said it it combines to both shutdown and restart and detect if you're updating a kernel or something and it sends a restart signal and this will know not to send VMS over to another node or to send VMS to another node depending on what condition you have so most cases you should leave it unconditional just so if you are going to work on say prox Max one or proxmox 2 um it will know to move everything over but in my case what I like to do because I'm running on Mini PCS with limited resources um I would do failover instead and only fail over when there's an issue so say right now I have failover set up if I was to go into Pro 2 and actually not shut down like don't send the shut down command I'm going to go over to my main proxmox and right click here and stop the machine you see VM stop this will actually stop this machine I'm still pinging right now so it's going this will actually stop this machine you're going to see this error out in about a minute or two this will actually move over to another node so again depending on your PC speed and how fast everything is uh this would be a lot faster uh really depends on your uh Hardware which is your network connectivity and also the hardware that you're using as far as storage goes so I can see right now it's a bunch of let me pull this up to be a little bit bigger a bunch of unable to reach the host oh and there we go we just saw that move up to the top to prox Max one and I had no interaction with it it just did it itself you could also see that it's performing some task over here which is ct100 start and there we have it the ping are back it successfully migrated from a failover state to another proxmox now there are some other things that you can do with this which I mentioned earlier which is setting up the group so you know like certain things that require Hardware that the other machine doesn't have it will purposely not move that over if you have it in a group where it's restricted so you got to keep in mind what will move over and what don't but this is how it works and there you have it guys this is a simple ha setup with Seth storage so if you plan to do this on your environment like what I did with my mini PCS and everything you're going to have a great time with this because now you have some sort of fail over if one of the servers go down remember a lot of it does fall back down onto Hardware that you have like make sure it's not on the same network even the surge protector should be on a different one or battery backup should be on different plugs and everything so you're not nested into one area where if one fails all fails but yeah again it all depending on how your setup is anyway if you guys are new to this channel consider subscribing and also hitting that Bell notification icon so you know when the next video is going to be out and I say my nerd cave hack till it hurts

Channel: Novaspirit Tech

Views: 7,126

Keywords: novaspirit, tech, proxmox ha cluster, proxmox ceph, ceph, ha cluster, high availability, proxmox, installing ceph pool, proxmox tutorials, ceph osd, proxmox ve

Id: Eli3uYzgC8A

Length: 16min 54sec (1014 seconds)

Published: Thu Jan 18 2024