Creating a Proxmox cluster with 3 old laptops

today i'm going to be creating what i want to call the laptop cluster i have a few of these older laptops that are about 10 years old now and i'm just going to try putting them in a cluster using proxmox and see how it goes see how the experience works see how it is stability wise see what type of performance i can get out of it and just see how the overall experience is the main reason i'm trying this is because laptops like this for me are essentially free these are about 10 year old laptops running sandy bridge and ivy bridge architectures they luckily have quad cores though so they have a reasonable amount of performance i believe these systems have between 8 and 12 gigabytes of ram which is what i'd say the bare minimum for running like proxmox or another hypervisor is and i've added a gigabit nick one to the one that only has 100 megabit integrated nick and this guy is a built-in gigabit nick so those reasonable network speeds going on right now now one issue is i only have two of these reasonably fast laptops i can use right here and proxmox in most clusters get kind of unhappy with two clusters because you have a problem that if one node dies then you only have one node in the cluster the way that proxmox works with voting is it has to reach a certain number of votes to have anything happen in a cluster and with two devices that means both devices would have to agree and if one node's offline they can't that can't happen because one of the nodes is offline and can't agree so you can either set it so that one node is essentially the master and says what always happens or you can make it so that you don't need less votes to agree but basically both of those were in the high availability you can do it in a home lab setting if you need to but the best solution is actually just get a third system that i have down here this laptop is even older running a core 2 duo system with about 2 gigs of ram but it's more than enough just to be that third vote in proxmox i don't plan on running any vms or containers on it and you can also use something like a raspberry pi for this use case you just need enough horsepower for it to basically be the third vote so then if either the faster systems go offline or just the third system there's always two votes left and the two systems can keep running as a full cluster now one thing you might think is why might you want to use laptops well the first reason is they're pretty easy to get your hands on at least for me i seem to get these fairly free from e-waste bins and other places like that and just find them used from other people relatively easily the next thing is um they're pretty small low power and you can kind of tuck them away easily these are opened up and big right now but i can close the lid on these stack them in a small pile and kind of tuck them in a corner pretty easily and they use fairly minimal space and also because they're designed to run off battery power they use quite little power compared to some other systems so all three of these systems together should be less than 100 watts and probably like 30 watts 40 watts at idle would be my guess as laptops are typically quite good at very low idle power consumption the other advantages they essentially have a built-in keyboard video monitor and mouse so you don't have to add anything if you want to administer a system and they also have a built-in battery which essentially works as a ups so i can yank the power cord and the system will just keep going so i don't need an additional ups but the disadvantages compared to a small form factor desktop or a full on server is much less upgradability a nice laptop might have two hard drive bays but that's pretty rare if with some newer models and smaller ones and you're not going to get more than that like you can with a desktop or server relatively limited memory upgradeability this guy is fairly high-end and supports 32 gigabytes of memory but a lot of systems from this era don't and it can often be quite expensive to find the largest dimms that you need to get that 32 or 16 gigs of ram that the laptop can support up to um like this guy here has here a good amount of low end laptops put 100 megabit network cards instead of gigabit which really limits things like nas uses i'd say streaming any type of shared storage on the network i'd really strongly suggest you get a um usb 3 to one gigabit nick because i find it works better than 100 meg but neither are great solutions as for software i've just put a plain install of proxmox ve 7.1.2 on all of these which is just the newest version of proxmox at this time i chose proxmox as a free hypervisor i'm pretty comfortable with it and it generally works quite well on fairly low power hardware like this as it has fairly low hardware requirements itself i've done some of the initial setup like going through the installer updating these systems and doing kind of my basic setup for all proxmox systems i talked a little bit more about that my proxmox setup guide a few videos ago if you want to see more details so now i've gotten the three laptops pulled up on my computer screen here i have them open in the browser and in a terminal with ssh and they look like normal blank proxmox nodes nothing has been set up i gave them all their own host names and ip addresses so let's go take a look at creating that cluster so i'm going to take a look at my first node here and just create a cluster and let's call it the lappy and it's going to start creating the cluster right now and i've created it so now i can see the um join information here which i want to copy and paste into my other systems and that way they can become part of the cluster proxmox is in general pretty easy when it comes to setting up clusters so now my cluster is being created looking in the web gui the biggest change on the left is now i see my two other laptops here and i can see and manage all of those settings just like if i was on the web page so the easiest thing the cluster lets you do is manage it as if it was one system also i can look in the command line with pve cm status and it shows me my three systems i'm running here the fact that they're all votes are happy here shows all the system shows how many votes they have and everything is working correctly in this cluster so now let's take a look at creating a vm so i've uploaded an iso of open sustain here and now i can see my multiple nodes that i can put it on i can call it like open susse1 um and then i can just create the vm like i normally would be able to on the node so nothing really special creating this vm on here now the one thing is when i'm creating the vm i can still only see storage and other devices on the local system so just because i have the cluster and i can see all the systems doesn't mean i can use the storage or any of the resources from the other systems by default and it essentially just lets me manage the other systems so now i've installed a few vms on my cluster and let's play around with running them on different systems so i have a ubuntu vm that's currently shut down and i have a windows 10 vm that's doing what windows 10 loves to do and updating so i'm going to take the shutdown ubuntu vm and now i can actually migrate it to other systems so i can pick a different node in the system so let's do laptop 2 or this guy my iv bridge quad core and it says it has a relatively large disc so it might take a while and with migrating between the systems it will copy the vm configuration so all the details of how the vm is made and then it will also copy the disk image that's what's going to take a while so if i look at my screen here it's about a gig out of 32 gigs down copying that 32 gigabyte vm image to the other system and this isn't limited to be being done on shutdown systems i can also do this on a running system here so it might take a while it also warns me that and it also migrate my running vm but actually when doing this i got a little bit of an error here which is saying that it can't move it due to missing instructions so one of the things in proxmox that's basically ignored at least by me if you're running a single node is the type of cpu you're running so if you go under memory processors as an edit by default it will do the kvm 64 which is that and i normally recommend just using a host with a single system and host is just going to copy whatever cpu your system has and just show that to the vm but if you're migrating between multiple nodes like i'm doing here and especially since these are different generations of this one being a sandy bridge and this one being an ivy bridge they support different features so i'm going to have to normally set to the lowest common denominator which should be sandy bridge here let's take a look at that so this one says it's set to sandy bridge but it doesn't seem to be working because both of these are at least sandy bridge so i'm going to have to take a look at why it doesn't like sandy bridge and what exactly it's complaining about so it looks like with a bit of googling the ibrs involves some of the specters mitigations that will be force required if it has it if you use the ibrs one if you don't use it it won't force require it but it's still interesting it doesn't want to migrate my one with the sandy bridges i believe they should both be at least sandy bridge so once this finishes i'm going to try turning it down to just a kvm 64 and see if that will migrate and taking a look at my ubuntu shutdown vm that's finished it looks like it's copied all the 32 gigs of the virtual machine disk successfully so it's on the new system and on the left i can see it on the laptop too and let's just try firing it up on this guy right now so it looks like it started and it's throwing that same error that it can't start using that function with sandy bridge so let's so i took a little bit of a deeper look to see if i could figure out what's going on google didn't have an obvious answer other than just use whatever the host says in kvm which does work and taking a look at the cpu features i ran a little online difference and it looks like there's a few features like smx aes and la hf lm that are different that the older system has this new one doesn't have and i'm guessing that's the issue i'm not sure why it isn't supported to my knowledge intel didn't remove those between generations but it might also just be due to the bios configuration or however dell set the system up as it's often that those cpu features that are disabled by either the bios or the system manufacturer or other things but it does look like in this use case setting it to kvm 64 does work fine so i'm just going to do that for right now firing up the console i can see that my ubuntu vm is working correctly on the other system just as i'd expect now that i have a proxmox cluster set up and i have migration working between my laptops let's try to do a little bit more of clustering and try to make these systems work together a little bit more so one way to do that in proxmox is ha or high availability and the idea behind it is if one of these laptops was to die it'll keep running my vms on another system now it's not perfect because it shuts down the vm when one of the systems dies and then the other one fires it up in a little bit but it's still pretty good and doesn't really require any special software you just have to have it start up when the vm restarts but the problem with this and my current configuration is since i'm using local storage on both of these laptops if this system were to go down all of its vm data and vm disks would now be unaccessible to the other systems to boot from so there's a few ways to get around that the easiest way in proxmox is with replication so i'm going to take a look of an example system i have replication set up with here and replication essentially just copies the data from one vm to on one system to another vm on another system by default it's every 15 minutes so the worst case scenario is you have an image that's 15 minutes old if it was to fail and it'll also use zfs send and receive under the hood to make this work so looking at my system i have vm 103 right now which is my windows server 2019 system set up that replication if i take a look under data center application i can see that it's on laptop one but it's being replicated to laptop two every 15 minutes so if i look at my data i'm storing on here i can see that i have my vm 103 disk on the host that it's running off of which makes sense and i also have it on the other host that doesn't have the vm and that way if this guy dies it already has the data essentially on here maybe a little bit old and can just restart it on here so that's pretty simple works pretty well and if i was to fail a system it'll just keep working that way now the other way to do it is to have shared storage and the idea behind this is you have some sort of storage system that all the computers in the cluster like all these laptops can access at the same time and everyone can see it unlike local storage where only that system can see it there's a few ways to do that like having a separate nas or sam that everyone can access but i already have three laptops here so i don't want to set up more than that so what i'd want to do is try to make it so that each of these laptops has an additional drive in it and it uses that drive to share storage and in proxmox ceph is the best way to set that up i am not an expert in seth and really don't know that much but proxmox is pretty plug and play and i probably made one of the worst setups you can create both of these laptops have an extra ssd in it and i just said create a cosd with that in there and taking a look at my cluster right now if i look at ceph on my systems i can see that i have two nodes in my cef both of the osds are working correctly and it says it's really degraded and my guess is this is due to that wanting to have three copies of all my data but i only have two systems that can store it so it only has two copies i believe this is edible somewhere but i can't find it right now so it's going to stay in this mode it does mean if one of these systems were to go down it will keep it running now one thing i was curious about when setting up the ceph configuration was how performance is going to be like because compared to a traditional single internal drive seth has a lot more going on it has two systems that has to talk to it has to go over the network there's just a lot more happening so i installed a vm on ceph and ran the most basic disk benchmark with crystal disk mark on windows 150ish megabytes per second read 44 right and those random writes are really bad and my guess is that's because it has to like confirm the right with the second system and go over the network every time there's a write that's pretty slow and i really don't know how exactly to speed up faster networking would do it you might be able to have it so one local drive confirms it and then it writes it out later to some sort of like caching solution i'm not sure but i believe that a commercial like a correctly set up seth can do it now the question is is this enough performance and if you're just playing around it kind of is so i'm on windows 10 i'm just going to fire up like edge for example it takes a little bit longer than i expect but it's kind of between hard drive and ssd in most uses i'd say this is usable for light use but i probably wouldn't want to push it too hard another thing i did to take a look at ceph performance was using the gnome disk benchmark tool so taking a look at it here i can see those speeds popping up really high likely due to ram cache and then going down as probably the ram cache isn't used as much latency is .2 milliseconds average but i can see some spikes in the couple of milliseconds and the write speeds are pretty crummy just as crystal disk mark showed here it's usable but this is one of the longest windows installs i've done when this guy was installing so i'd say barely usable and this is two ssds on two old laptops if you set it up correctly seth can be very fast but this is not a correct ceph setup i'd probably stay away from using ceph like this on these old systems while it works it's pretty crummy and you're really pushing a limit of what you can do if seth i'd probably stick to replication if you need to have an h a solution now the one big upside with steph and any other solution where all the systems can access the data is that migration works a lot better so let's take a look at migration on here so i'm going to migrate my windows 10 vm over to the laptop system and because it's shared storage it doesn't have to touch the storage at all all it has to do is copy the ram from this system to this system and it'll just migrate the vm so in this case it's super fast to migrate only like a couple of seconds less than a minute whereas it would take a lot longer if it didn't have the storage copied already so now it's almost done copying the vm state 16 milliseconds of downtime and i could take a look at my virtual machine and now it's running on a different host and works just fine the vm doesn't even notice that it got moved between hard systems now let's take a look at testing that high availability here and seeing how well that works so i'm going to set up some high availability in proxmox i set it up for vms 103 which is my one that's being replicated to the other system and i'm also going to set it up for my one on ceph storage so that's my windows 10 bvm and i'm going to set it up with a group i called good ones which is these laptops that can actually run the vms if you don't do it sometimes it put it on my third laptop which can't run the vms and you run into a lot of weird issues so i'm going to add this guy to it and then if this guy is to die which it's going to do right now i'm just going to go kill it it's going to automatically it should move all the vms to this system when it detects it proxmox says this is a up to two minutes thing but we'll see how long it actually takes so it's detected that this laptop's dead and now i can see that these two vms or three of them actually are now running on this system now and it's they had to reboot because they went offline it did force the vms off you can't do it where it keeps the ram state but if i take a look at the console right now it's still a functional vm my luck on camera seems to be pretty bad i did quite a bit of testing for this video and did mini power off cycles and just showed how everything would migrate successfully from one system to the other but every time i seem to try to show it on camera it doesn't work and i think i just found a way to brick two vms my windows 10 bvm that was stored on ceph is now just not happy it'll boot into like the windows 10 logo where has the dots and then it just dies and restarts and then my windows 2019 system wasn't happy when i migrated over to the replicated disk says it can't boot from the volume and won't boot ever again so i just broke two vms with trying to migrate it um there might be a way to look in snapshots and get older versions of the data and i don't know enough about ceph to try to get this working correctly but it just doesn't seem happy with what i've done here taking a little bit of a closer look at ceph on here sef was not happy to lose one of its nodes i'm guessing ceph is kind of like proxmox where it really wants three or more nodes so when it's with two nodes and one of them dies it's just kind of like i don't know what to do and it just fails um that's what it looks like it showed it it's looking a little bit better now because it detects both systems are up but it's i wasn't happy with two systems you want to probably have the third system actually in ceph all running but i'm not going to get into ceph too much more here i've had replication work fairly around proxmox just not now it seems to be so overall is this something i'd want to set up more for the long term or in a new setup my answer is probably not for me personally these laptops turned out to be a lot more of a bayer and proxmox to use than i expected i thought this would be a pretty simple cluster to use but i think i just found a way to keep breaking things in ways i didn't expect i probably want to do just a full wipe on all these systems if i was to use it more because i think i just have something weird in my configuration now that it really just doesn't seem to like i think the other thing i've learned is don't do shared storage like ceph on such a small system it just really isn't happy you gotta make sure these distributed systems are set up with a good configuration i've tried multiple times to have a distributed storage on old systems and it never works out well you kind of just can't do it at a small home level like this with older systems you need to actually throw correct hardware to get distributed storage working correctly but if you just want to play around with clusters this is a pretty good way to do it these laptops are very cheap now you can get the kind of feel for setting up clusters on real metal instead of having a cluster running in a vm and i think it can have a lot of learning in how to set up clusters how to handle failures and stuff like that that you couldn't do if you just had a single node but if you just want a simple home server or something i'd do a single node possibly set up two and proxmox but don't do anything fancy just migrate vms every once in a while if you need to thanks for watching this video on setting up a laptop cluster of these old systems let me know if you're using our cluster on old systems like this and how it's working for you

Channel: ElectronicsWizardry

Views: 30,681

Keywords:

Id: at4J2bVPdwc

Length: 20min 20sec (1220 seconds)

Published: Wed Apr 13 2022