I built another server cluster... - Promxox HA Cluster w/ Ceph

the latest addition to my home lab is a full fat three node highly available cluster running proximox wait didn't you just make a video about this yeah but that was on some underpowered Zima boards and this is on more expensive Hardware therefore it's better I'd say I'm pretty pleased with the setup but nothing's perfect let's talk about it so why why did I need to do this I'm gonna be honest with you guys I don't need this I don't need a lot of stuff but that hasn't really stopped me before the real reasons I wanted this setup are because I wanted High availability for my Docker containers and I wanted to virtualize my k3s cluster like k3s cluster has been running on bare metal on three Intel Nooks while this has worked pretty well it hasn't been updated since installing it because I'm afraid to break something with a virtualized cluster I can just take a snapshot update it and then roll it back when my dumbass breaks something neat so so along with those I also wanted this setup to have a good amount of power be decently power efficient have fast networking AKA faster than one gig and be expandable via pcie well guess what we accomplished all of that let's go over the hardware at the heart of each of these bills is a super micro X10 sdv4c which comes with the Xeon 1521 soldered directly onto the board this is a 4 core eight thread CPU with a base clock of 2.4 gigahertz that can boost up to 2.7 and has a TDP of 45 Watts whoa that's it they got chips out there with 16 cores yeah duh but I wanted a nice balance of performance power and price three PS I paid 250 dollars for each of them which I think is a decent deal if you can find the 1531 or 1541 versions which have six and eight cores respectively then those could be an option if you need more horsepower this board also has a dual 10 gigabit Nic which will come in handy later for Seth for Ram I wanted a decent amount so each of these has 64 gigabytes of 2133 megahertz ddr4 only use two sticks so I could easily add another 64 gigabytes to each of these in the future if I need it but I highly doubt that the cost of the ram was only a hundred and seventy dollars too which was probably the most surprising cost of the entire build to be honest okay storage my plan here was for each system to boot from a mirrored raid one setup so each of them has a super cheap 256 gigabyte silicon power SSD and you can get those for like 15 each which is awesome so what I would say if they weren't absolute cheeks here's my first mistake of the build well yes these things work fine as a boot drive for proxbox you can't do much else with them I thought to myself okay 256 gigabytes that's enough to load up some VMS and proxmox on those two as well as be a boot drive and it is but these drives are so bad that running anything else on here spikes the i o delay of proxbox all the way up I think it's due to the SLC cash on there because once that's filled up and you're hitting the actual flash storage game over if I had to go back and do this all over again I'd probably just spend a few more bucks and get some Samsung evos or something like that the second piece of the storage puzzle is what I'm actually going to run everything from I mean my k3s and Docker VMS these boards have an nvme slots so clearly I went with nvme storage right wrong kinda my plan was to use ceph as a highly available storage in here which would require both speed and endurance for that reason I went with some two terabyte Micron 7400 Pro Enterprise ssds they're udot3 which uses pcie Lanes but just a different form factor luckily they have nvme to u.3 adapters so I snag three of those and rig them up these drives are super fast offering around 6.6 gigabytes per second reads and 3.5 gigabytes per second rights the appeal to me though was the 3500 petabytes of right endurance and there is absolutely no way I'm ever going to reach that so we are good I snagged these for a hundred and forty dollars each which I think was an awesome deal and I got pretty lucky all right to round it all off we need a case and power supply well we killed two birds with one stone there when we snag some in-win iwrf1 100 1u chassis that also come with a 315 watt power supply perfect kind of the only issue I ran into here was not really an issue with the case but the 8-pin power connector from the PSU is a single eight pin and not two separate four pins so I had issues plugging it in to the four pin port on the motherboard since there was a capacitor right next to it a few eight pin two dual 4-pin adapters fix this though I got these for about a hundred and sixty dollars each which I think isn't too bad oh and I guess another thing to mention is that I bought some 90 degree PCI E adapters that I was going to use to fit a pcie card in here but um it it doesn't line up with a slot on the case so I just got some pcie extension cables to serve this purpose I guess you can call that mistake number two oops all in for everything not counting the adapters I bought by mistake this build ran me around nineteen hundred dollars is that a lot of money yes is it too much money I mean if if you don't have nineteen hundred dollars yeah for my needs I think it's a decent price for what I'm getting of course I could have done this cheaper I know that but I also could have spent way more and well um I did spend more and the process of replacing the heatsink and fan on one of the boards I bricked it I'm guessing I nicked a tree swim prying off the back plate because after installing the new heatsink it wouldn't even get past the ipmi initialization yep stake number three try clearing the CMOS flashing The BMC and bios changing the RAM and even cast a witch's spell but that that had other effects so yeah any suggestions are welcome down in the comments and if you don't have a suggestion you have to comment I'm just here taking up space you know the rules now how does this setup perform honestly pretty great granted I haven't put it through its Paces yet it's done everything I asked of it I set it up as a proxmox cluster which is an easy process in itself from there I set up stuff which will provide me with high availability storage for the entire cluster I'm not even going to pretend to be some ceph expert or even competent for that matter but I know the basics and how to press a button to get it set up which is enough for me since this is going to be my production cluster moving forward I'm sure I'll pick up some knowledge along the way but for now I'm a Noob what I do do know is that it provides me with highly available storage so my services can fail over to any of the nodes and that it works best on faster networking this is why I was adamant about having 10 gig networking on these nodes since each machine has two 10 gig ports we've dedicated an entire one just for the ceph network and even went one step further and gave it its own switch I picked up a UniFi Flex XG which is a four port 10 gigabit switch and the only clients I have plugged into there are the nodes with their ceph Network this isn't required but every Forum post I saw basically crucified anyone who didn't have a dedicated theft Network so I didn't want to end up getting Cyber Bullied by some neckbeard at 300 I think it's a pretty decent option so go ahead and throw that into the total cost with stuff up and running everything was working as expected only two terabytes of storage was showing up in each host and every VM could be easily failed over to each node to have the cluster automatically recover the VM on a node failure you'll simply mark it as ha in proxmox I went a step further and created ha groups which lets me specify which node I'd like to be the primary for each VM this allows me to ensure by default there's a single k3svm running on each node then if that fails it will pick one of the other two to spin it back up the cool part is that when the original node is back up it will throw it back on there since that one is primary based on the ha group cool stuff oh and when you do this the higher number in the group is the more primary I initially assumed the opposite where 0 would be used before one and so on but no it's it's the opposite two other things I need to mention about the whole setup one is that the 10 gigabit Nix on here only work in one or ten gigabit mode I initially planned on hooking up the other 10 gigabit Nic to my 2.5 egg switch to get 2.5 gig networking to the proxmox hosts but that's not supported for my use case it's not really a problem but it's definitely something to note the other thing is that I could not for the life of me get GPU passed through working on these machines before you link me a guide you found on the first page of your Google search just know that I tried it I spent days tinkering around with different VM settings host settings bio settings different cars different operating systems nothing would work I got the famous code 43 driver error which is absolutely annoying as far I don't really need a GPU I was just throwing stuff in there to mess around as of right now the pcie slots are vacant and ready to have a nice pcie card inserted right inside it let me know what you put in there and um don't say you're subscribe okay overall thoughts here I'm happy with this new setup is it perfect now could be a bit more powerful I could have spent less could be easier to pass a GPU through sure but none of that stuff is even close to a deal breaker for me I bought this setup to be a power efficient highly available cluster for me to run my production Docker and k3s cluster on at 100 Watts total for all three machines I am extremely pleased no it's not going to win any benchmarking drag races but I don't need that I still have my epic system if I want to run anything resource intensive if I had to do all of this over again there are a few things I changed those silicon power drives just suck if you do this or anything with ssds just spend like 10 more dollars and get a Samsung or a crucial MX draft also I did see some 1531 machines for just a bit more money on eBay that included a chassis and RAM so maybe I would have gone with those I I don't know maybe not other than that though this this is an awesome setup I know it isn't a super deep dive into everything but that will come with using it for a bit once this setup has a few decades under its belt then maybe I'll be comfortable enough with Seth to come back and teach you guys some stuff but for now that's all I got if you like this video then be sure to drop a like if you like seeing a dude spend too much money on stuff that he doesn't need then subscribe because you'll you'll see a lot of that I want to give a huge shout out to my YouTube members and my patreons you guys are my ceph cluster AKA my certified Easter party hat if you're still watching I appreciate it thank you so much and I'll see you in the next one [Music] good

Channel: Raid Owl

Views: 27,346

Keywords: linux, proxmox, homelab

Id: lELrvffVP04

Length: 12min 18sec (738 seconds)

Published: Tue Jun 27 2023