Fully Routed Networks in Proxmox! Point-to-Point and Weird Cluster Configs Made Easy with OSPF

are you playing with small proxmox clusters and want to get into high speed networking without paying big bucks for a high-speed networking switch maybe you've got two or three nodes and you just want to put some 10 gig cards in each node and connect them with direct attached cables or maybe you've got some crazy setup with five nodes and quad gigabit Nicks in each card or maybe you're the guy that's got two Intel looks wants to connect them directly to each other with Thunderbolt well in this video we're going to cover fully routed cluster networks which can handle any topology you throw at them it will find the most efficient path to send packets all throughout your cluster so let's get started on this adventure so we're going to start with a setup where we have three nodes and they're all connected to a plain gigabit network switch we are going to call this network the public network and all of our nodes are going to be connected to it this is where things like the web UI runs corusync is going to run on this the proxbox cluster service if you have any interaction with the internet it'll be over this link now we're going to start adding some high speed links that directly connect nodes to each other some of these in blue become a bit bigger so we know what's going on so we could just put two in we could put all three in we have all three then we have enough redundancy for PVE ring 1 to send a packet to ring two ring three if any one of these high speed links is down and if somehow all of our high speed links are down or a particular node doesn't have any high speed links it can fall back on the public network to send its private traffic so now for our fully routed Network we're going to give it a separate subnet than our public network this is going to become our private subnet or our cluster Network for this example I've chosen the subnet fd69 beefcafe 64. you are free to choose anything you want as long as it doesn't overlap with anything else in your setup this is a completely unique subnet even though it won't show up on any link so given that we now have a private subnet we can assign addresses to each of our nodes on the private Network and let them communicate via these point-to-point links so to manage these routing tables across the private Network we're going to use a protocol called ospf open shortest path first and that's implemented with a package called frr free-range routing so I've got a Proxima cluster I've already set up it's got three nodes they're already clustered together let's hop on and see how it looks so in our clip here we got our three announced PVE ring one two and three each of them have three network adapters one of them is our gigabit and that's our vmbr0 it's already existing that's ens 18 and then we have two high-speed links ens 19 and 20 and they're all the same no they don't have to be the same on your setup in my case they were but you just have to know what interface is which so whatever interface you're leaving as is leave connected to your vmbr0 and whatever interfaces you want to use for the routed Network we're going to go in here and say auto start but we're not going to give them an address at all we're just going to auto start them so check auto start just make sure it's not changing anything else here so it's just adding those two is on that manual that's good and then do that to all your notes so make sure your networking drivers are working the interfaces come up and they're enabled so now our nard here we're going to look at the IP addresses we have just to make sure that the interfaces came up so IPA so in this case we have four interfaces the first one is called Lo that's our loopback and that has the usual colon colon one we have ens 18 which has nothing on it because it's part of vmbr0 ens 19 which got a link local IPv6 address it's great ens 20 also got link local IPv6 also great and then vmb or zero that's where our real address goes and that's the bridge we use for VMS it's the default when you set up proxbox so now that those links are up we're going to install frr so this is pretty easy we just apt install frr and do that for all the nodes of course so once this is done we needed to have frr that we want to run the ospf V3 Damon so through that we have the file Etsy at CFR art demons and here go to ospf6b so technically the protocol is called ospf V3 for IPv6 frr calls it ospf6 for IPv6 so ospf60 yes and I save that file do that for every node of course normally we would restart frr so it starts the Daemon but in this case we don't have any IPv6 config at all so we need to edit the frr config file I see frr.com and So currently it just says logged assist log that's it so we need a little bit more than that so first thing I'm going to do is I'm going to enable IPv6 forwarding [Applause] simple as that next up we're going to configure each of the interfaces in the system for how we want them to behave with ospf so first up the loopback interface thank you so exclamation point and then new line interface hello tab in so I mentioned we're going to assign each node and IP address on the private cluster Network we're actually going to assign it to the loopback adapter as weird as that sounds Linux will accept packets for any of the system's address if they arrive on any interface if we were to set an IP address on all of the point-to-point links and one of the point-to-point links were to go down that address would no longer belong to the system because the address the address only exists when the interface is up so by adding an address to the loopback interface that address always exists in the system and Linux will accept packets for it no matter what interface they come in on so whether they come in our Ethernet or whether they come in on our point-to-point links they'll always end up to that destination and we can use that subnet of the cluster Network to identify where our source address should be so over here they can take vial IPv6 address we're going to put the address of this particular node on the private cluster Network [Applause] and that's going to be a 128. so this is a single address so the sub to mask will be a single address we have 32 or 128. then we're going to set it as part of the default ospf area IPv6 plus pf6 area foreign and we don't want ospf to be advertising itself on this address that would be kind of silly so we're going to say passive good so next up we're going to configure our vmbr0 this is going to be our backup link if all of our point to point links go down we can still route across that Network even if it's slower if some of your nodes don't even have point of boiling because they can rely on this network as their only way of communicating with the cluster Network so go again exclamation point interface vmbr0 and this time we don't need to give it an address because there's already an address on the interface we just have to tell it how to configure ospf [Applause] so again it's part of the default area [Applause] it is a broadcast type Network this means there could be potentially many devices speaking ospf V6 on this network we would like to talk to all of them basically means that ospf is going to establish a designated router so one of the nodes on the network will decide that it's going to be the one that computes paths and last up the cost so cost is a number that ospf uses to calculate what the optimal route is so if you have a tree of routes and each route has a certain cost to take a to take a packet across that route and you need to go across say three different links you add up the cost of those three links to get the total cost of the route so when ospf is Computing the ideal route from one point to another it's going to take these costs into consideration the standard way of doing this is to take some high bandwidth say 100 gigabit and divide it by how fast your links actually are so a one gigabit link would have a cost of 100 100 Gig divided by one technique would have a cost of 10 25 giggling would have a cost of four for example but you can manipulate these to drive traffic the way you want it to so my recommendation is to start with the 110 if you if you know the speeds of your network you can put that in if you're using all gigabit you should make sure your point-to-point links have a lower cost than the broadcast network otherwise it'll just send all the traffic over the broadcast network and now each of the point-to-point links now the point-to-point Links don't need to have IP addresses explicitly assigned because we can just rely on the automatically generated IPv6 link local addresses foreign [Applause] 19 this is the point-to-point network point to point networks don't designate routers they just talk to each other as a pair so that and the cost [Applause] and finally some general configurations for the router so we need to give our router an ID so OS df6 and this has to be unique across your system in my case I'm just going to use the address of the node 551 is this particular node 552 583 Etc then we need to tell ospf what routes on the system it should push out so not only is ospf going to spread routes across the network it's also going to find destinations on the network and advertise them to everyone so in this case we want to say we're going to advertise everything that's connected and since the loopback adapter is always connected it'll always show up as one done with that file now we restart frr [Applause] so you can see if we do IPA again so our address on the cluster network is now added to the loopback adapter that is great now we do this to the other notes so for this node the address seems to be different of course this one's going to be 552 and the router ID needs to be different and if you have different interface names on each system make sure you update that too of course update the daemons file [Applause] so if we go here to the node we do IP dash six how it should show us the routing table oh look at that so fd69 beef Cafe one is our own so that's devlo and then 552 via the point-to-point link the Dev ens 19. and 553 also going via ens 19 so I'm not sure why it's not taking ens 20 there just take a little bit of converge so you can see 551 is via our own interface 552 is via the link local address on the point-to-point interface Dev ens 19. and 553 is violent local address on the point to point interface Dev ens 20 and those both came from ospf so now if I ping one of the other nodes [Applause] and look at that so now that we can ping across the link let's see what happens with iperf so I started iperf server on PVE ring three I'm going to use a tool called analog so we just give it all the addresses we want to do so we're going to do vmdr0 yes 19 the ns20 and so I can just use the keyboard to switch between each of these show you how much bandwidth is going on it so from pve2 we're going to connect a pve3 so 553 and you can see we're doing a heck of a lot of gigabits if I just run that for a long time come back over here so vmb are zero not much traffic ens 19 not much traffic that's connected to pve1 and then ens 20 oh boy look at all that traffic so 14 gigabits going out eight megabits coming back that's tcpx probably so what happens if I turn off this connection so there's these proximos are actually running in this proximox node so they're here and we we unplugged the cable oh no look at that cable unplugged but switched right over to the other link so that means all this traffic is now running through PVE ring one even though it's not part of the conversation [Applause] so if you're on end load over here sure enough vmbr 0 does not do anything ens 19 has a bunch of data coming in ens 20 has a bunch of data going out so because pb2 and 3 lost their connection to each other they're routing via pve1 so let's say hypothetically pve3 isn't connected to the other two by high speed Links at all so we'll turn off both of our high speed links what happens if we do that same test looks like we're going across vmbr 0 this time so it's going to fall back on the public network if it can't communicate on the private Network because it's allowed to use ospf on the public network so what if you want to be able to even crazier setup so here I have set up five nodes in a ring so they're all connected to the public network as they were before and now each of them is connected to the nodes on either side of it in a full circle so there's a ring Network here and it's fully routed so any node can talk to any other node so I pull it up here so I pick for example pv2 say I want to trace a route to pve5 so can I ping pb5 sure can how about trace route [Applause] two hops there I'm going via one so my im2 and I'm talking to five so two to five traffic is relayed via one makes sense so it happens if I take this link down okay a little different about this time going three to four to five so this link is down so we're going to three to four to five yeah makes sense makes sense now we'll run eye proof here I'm running it on PV one and I'm going to connect from PVE internet or pv3 yeah this looks like we got pretty good throughput there this is virtual so no real 10 gig and then what's going on with the load on pv2 this looks like pve2 is relaying our traffic makes sense so hopefully this video helps you understand how to build fully routed cluster networks like this the really useful in cases where you have small clusters you want to do high band with links between specific point to points but you have more than just two nodes you can't just create a single point of one link but less than enough to buy a high bandwidth switch or maybe you want to try something wild or you have a bunch of networking Hardware laying around whatever it is hopefully this can help you so as for what traffic you can put on here like saf or migration or replication I got a video coming up on that so stay tuned I got a Discord link down below if you want to chat or ask me any questions about this of course I love IPv6 so this is all IPv6 it transists ipv4 yeah not quite so cleanly but if you use IPv6 they'll be all good to go and as always I'll see you on the next adventure

Channel: apalrd's adventures

Views: 22,536

Keywords:

Id: dAjw_4EpQdk

Length: 16min 40sec (1000 seconds)

Published: Thu Mar 16 2023