Proxmox NETWORKING: VLANs, Bridges, and Bonds!

yo dog I heard you like networking this bad boy right here is my HP micro server gen 8 that was kindly sent as a donation to the channel and it is going to be our test subject for this video we are going to learn the ins and outs of proxmox vlans bridges and bonds turning around our little star of the show here we got three ethernet cables in green all of these are going to a managed gigabit switch to go to the built-in ethernet adapters these are one gigabit one goes to the lights out management hpilo we're going to ignore that one for this video so with these two interfaces we can test out different styles of setting up the nics bridging them bonding them failover link aggregation lacp all that kind of good stuff and I heard you like multi-gig too so I got a 2.5 gig USB 3 ethernet adapter connecting up to the same switch between the Dual one gig and the 2.5 gig we can test bonding asymmetric connections simulate what would happen if you had a 10 gig with a one gig failover things like that all of these are connected up to a switch on the switch I have another server connected with the full 10 gigs so we can test iperf across all of these different scenarios so let's hop on over to the proxmox web UI and get this bad boy configured when you first set up your proxmox system through the installer you'll pick one interface and it will create a bridge in that interface in my case that interface was eno1 and the bridge is vmbr0 all the other interfaces are disabled by default any installer so if we want to do anything fun with networking we have to come to this place so data center name your node system Network I scroll over a bit you'll notice that there's only an IP address assigned to the bridge not to the individual ports and that's very important once you take a port and you start assigning things to it you're going to end up putting the IP address on the higher level device so if I take two ports and I Bond them the IP address and the MAC address are now tied to the bond not to the individual Port the individual ports don't have an address and they don't use their Hardware Mac address they use the MAC address of the bond they share it when we had a bridge on top of it we again do the same thing the bridge now has a MAC address and IP address for the local system so do the virtual machines that are attached to the bridge and the hardware Mac address and there is no Hardware IP address on the individual interfaces aren't used anymore just the bridges so if you want to change the IP address to your system you need to change it here in the bridge of course this depends on how you want to do your networking and that's what we're here to talk about today so assuming you're not in a cluster and you're not using ceph there's a few different types of traffic you're going to see on the proximox network there's traffic to and from proximox itself so that would be us visiting the web UI via the IP address of the proxbox server itself we have the proximox server going out to the internet for updates and connecting to remote storage via Networks then we also have traffic which we're forwarding to Virtual machines so if you're using a firewall virtualized you might be forwarding one interface directly to a virtual machine so you don't want anything else to have access to that Wan interface or you might be putting everything on your Lan subnet and bridging it all to the same interface and that's fine too maybe you have more than one interface maybe you have a 10 gig and a one gig and you want to bridge some VMS to one gig and some VMS to the 10 gig that's that's fine and make more bridges so basically you need to think about who is connecting to proxbox with how much bandwidth how am I connecting to my storage is with how much bandwidth and who are my virtual machines connecting to again without much bandwidth then we can start to allocate what our network resources should look like so the first and the easiest concept is the bridge you can really think of a Linux Bridge as a network switch so here on the bridge we have Bridge ports over here the labeled slave ports these are Hardware ports that are connected to the bridge so this would be like taking this network interface and plugging it into the switch this network interface gets plugged into this switch the other things we can plug into the switch are the node itself so here we've assigned the bridge and IP address the Gateway and all that good stuff and that means that we're plugging our proximox server into that same switch the third thing we can plug into that switch are virtual machines so if I make a new virtual machine when we get to the network tab we can add a new network interface and by default it'll just add one but we can add more later so I'll show you how to set this up so first we can choose the bridge we want to connect it to we usually want to use a pair of virtualized interface if your software supports it if it doesn't you can use the E1000 that's pretty good option and the VLAN tag so in this example I could set it to be then 50. it'll automatically create a MAC address for the new VM here we go so what this will do is this will plug in the virtual machine's Net Zero interface to the bridge vmbr0 and assign a VLAN tag of 50 to it so this would be like using a manage switch and setting the pvid on the port to 50. we don't have to add a VLAN tag if we don't want to you can always add more network devices here so maybe we want to give it no VLAN as well so now we have Net Zero and net one one has a tag and one doesn't we can add as many of these as we want it'll pass them all through the virtual machine and one thing to be aware of is that this VLAN aware needs to be checked if you want to do that it's a little Nuance that I just got to watch out for when we set it to VLAN aware that basically means that it's acting as a managed switch which is aware of vlans and lets us set the pvid on each port and usually we want that to be the case you can see we have pending changes here and it'll tell you what the network configuration will look like if you're familiar with Linux networking this will show up down here so what if I want to directly connect two virtual machines together well you have two options for that the first option is to use an unused VM ID if you're using a VM ID that's not used elsewhere in your network you could just set this to something like 1000. and then on the other VM you could also give it that same tag and now the two of them can talk to each other that does mean that that particular VM tank will be exposed on your interfaces here of the bridge so if you're using that VM ID elsewhere in your network you could have conflicts the other way to do it is to create a new Linux Bridge just to dedicate to binding the two VMS together or three VMS however many VMS you want now you can create a new Red Bridge in this case we'll create it vmbr1 and we won't give it an IP address because the proximox system itself won't be able to communicate here and we won't give it any Bridge ports either because we're only using it for V VMS so we'll say go ahead with that we'll apply that configuration now I can go to my test desktop and I can switch it over to vmbr1 and so now any VMS I add to this vmbr1 will be able to talk to each other but not to anything else so you can use this to create virtual networks if you're doing home lab networking that kind of thing and it's a very powerful feature so now what about bonds bonds are useful when you either want to do failover or you want to do link aggregation and the easiest setup for that is failover so we're going to set up failover first so I have eno1 and eno2 they're both gigabit Ethernet and enx with a long number and this is USB so it's going to have a long number so what I'm going to do in this case is I want to use my 2.5 gig Network and I want to fail over to my one gig Network so let's create a new Bond first we're going to go to this bridge and we're going to get rid of the Bridgeport and now we have unsaved changes but we need to leave them unsaved otherwise we're going to lose access now we're going to create a Linux Bond we're not going to give it an address because the bridge is going to give it an address and our two slaves are going to be you know one and now we're going to type that long number out again x0242788 root 2 FB and our mode is going to be active backup and the primary is going to be a 2.5 gig adapter which is this one let's go so now we have the bond created and we can assign the bond now Bond zero to the bridge and that means that we'll take the two ports and aggregate them into a bond the bond is using active backup mode with a 2.5 gig as primary the bond is then assigned to the bridge so all the traffic on the bridge shares the bond and we can go apply that yes foreign still works so if we want to see the status of the bond we can do Cat Rock net funding on zero so it tells me that link failed once because I was testing it earlier let me scroll up a bit says that's the currently active slave which is good because that's our 2.5 gig Nick so this doesn't require any special features on the network switch to implement failover bonding all it does is it looks at the state of the link does it detect the cable or does it not and as soon as it doesn't detect a cable it'll instantly advertise its IP address and its MAC address on the other physical interface the backup so no need for manage switches that support lacp or link aggregation this will always work so with the two and a half gig link up I'm error and I perform three this normally isn't installed but it's used for bandwidth testing iridium so you can see we're getting very close to 2.5 gigabits per second the bitrate will always be a little bit lower than the physical rate on The Wire because 2.5 gigabits refers to the actual number of bits on The Wire and you'll have a lower number because there's some bits used for synchronization there's some bits used for the layer 2 Mac frames and there's some more bits used for the layer 3 IP frames so before they look at 2.5 gigabits as soon as I take down the fast interface so we fail back to the slow interface so now we can see the interface is down for an iPhone 3 again we should get about a gigabit and 158 megabits 141 megabits that's pretty good that's about what we'd expect now what if we wanted to bond eno1 and eno2 and these are both identical and these could be one gig interfaces or they could both be 10 gig interfaces or maybe something else I found that reconfiguring the bond with the bond running didn't seem to work um I'm not sure why proxbox wasn't reconfiguring it but basically to switch this Bond mode I would switch the bridge to have no interface bound I would delete the bond and then I would assign the bridge back to a single interface I'll say Eno one apply that and then once it's applied I would create the new Bond again so we're going to unassign Eno one we create a new Bond slave so it'll be another one two it is lacp Hash policy three and four comment now that's created we can add the bond here and apply it you want to see how lacp is working we can do that same trick to cap again so cat proc net bonding Bond zero and we get a whole bunch of information this time so starting from the top it says lacp active is on and so what that means is so with lacp at least one side of the link must be active that means it is initiating the licp bridge to the switch in my case by default proxmox is the active side so the switch has to be configured to allow passive lacp if the switch isn't allowing lacp Linux can't configure it so then it's got some more information so there are two ports that make up the bond and then there's a slave interface Eno one which is 1000 megabits a gigabit and it says the the two details so the actor and the partner so the actor is our side and the partner is the remote side and the remote side is on Port 24. and has this Mac address so it's on 424 my switch which makes sense and the other slave interface is you know two and its partner is Port 23 which makes sense because I have the two switch ports plugged in next to each other so if your switch supports passive lacp you shouldn't have to configure anything on the switch other than enabling that on the ports you're using now I can't give you specifics on how to configure your switch but I'll show you what mine looks like so this is my CRS 328 it's running switch OS I go over to lag by default I have them all set to passive so someone were to plug in an lacp device to random ports around my house it would work not that you would necessarily want to do that but it would work and then again down here reports 23 and 24 found a partner the partner is cecd e0 dot which is our proximox system and they are on trunk one so that means they're first lacp group on the switch so even though we now have two gigs aggregated with lacp we still can't get more than one gigabit per second per session because it's taking the hash of the Header information to determine which interface to send a packet to and if the packets are all going from the same source to the same destination they'll all have the same hash so if I run iperf again I'm still just going to get about 950 megabits per second 947 938 about what I'd expect so I install LifeProof on a virtual machine so this is running on the proximox system so it's sharing the networking from proxbox and if I run this with the same command it's again getting 950 megabits per second so now I'm able to get the full gigabit from the VM and separately from proxmox I'm doing this by running iproof in reverse and that means that because I have one iperf server all of the packets are destined for the same Mac address the same IP address and they're not getting spread out across the interface but I run in reverse the packets are destined for the two different clients so if I run this one in reverse and I run this one in reverse you can see they're both going to get the full bit rate at the same time that one's getting about 950 this one's getting about 930. so they're spreading across the two links in One Direction so because we have three interfaces we could bond a bond so what if we have say a dual 10 gig card we usually want to bond and we also have two one gig interfaces we like to fail back on then we can Bond the bond in my case I only have three interfaces so I'm going to have my lacp bond as Bond zero I'm going to create a new bond that has my 10 gig I'm going to bond that with the lacp and then we'll fail over from the single 10 gig back to the Dual 1 gig so we're going to again remove the Bridgeport so that the thing becomes free create a new Linux Bond our slaves are going to be Bond zero and giant name and my USB adapter the nx002427 8832fb and we are going to use active backup and that is our primary so now we have 2.5 gig that's our add-in network card and we're going to fail over to the two one gig network cards built into the motherboard and then that new bond is going to become our main bond for the bridge let's apply that so let's see what we see in bonds so we see two bonds again we can cap both of them so Bond zero is our dual one gigs and it is up they have found their Partners Sports 23 and 24 on the switch they are good I wonder if we can't find one so we're reporting that the 2.5 gig is down maybe I left it down from earlier yeah I did let's turn that back on I can just turn off switch ports on the switch management UI here we go so now it's up so now our active slave is enx if I run I prevail I should get two and a half gigs and I do and then if part way through this I disable my long port should fall back I just did the end there right again you'll see we've dropped on the 950 because we fail back from the 10 gig interface of the two and a half gig interface to the Dual one gigs now there are other things you can do with your onboard Nicks other than buying them like this for example if you're using a proxmox cluster the proxmox cluster should have its own interface dedicated to clustering and if you're using SEF ceph would really like to have really high speed storage for its private Network that doesn't mean you can't use vlans to segregate ceph if you want to segregate them for Network management reasons but you shouldn't be using vlans to segregate the cluster traffic you should give them dedicated interfaces that said chorusync does not need much bandwidth so having a one gig interface dedicated to chorusync is perfectly fine even 100 megabits would probably be fine too you don't need much you just need something dedicated so that other traffic isn't affecting the latency of those packets for chorusync so hopefully I've shown you how to make the most of networking in proxbox one thing I didn't mention is Linux VLAN and that's because it's very rarely used it's much easier to set the bridge as VLAN enabled and tag individual VMS on specific vlans what this doesn't let you do is set the proximox system itself onto a different VLAN so if you need to VLAN tag the proxmox system itself that's where you could use Linux VLAN so in this case what we could do if we wanted to add a interface for proximox management on top of our Bridge we could say VM vr0 vr0 dot the VLAN number so then it'll automatically fill in the VLAN raw device which is vmbr0 and it'll automatically fill in the VLAN tag which is 100. management then we would set up our IP address here for the proxmox local system and remove that configuration from the bridge then our proximox local system would be accessing via VLAN 100. normally you don't need to do this if you're using a VLAN aware Bridge or your VMS you would only need to do this if you have a specific reason to have a dedicated Linux interface such as assigning the local proximox system IP address to the VLAN so another possible use case for vlans is that you might want to pass through vlans to the VM and by default that just works so I have an example here that's a router it's a Microtech CHR running router OS and so I have to set up the network here to be on vmbr0 with no VLAN tag with the vertio interface that's important because some of these other ones don't support VLAN tagging in router OS I've added a VLAN VLAN tag 5 and I've tagged my test desktop onto VLAN 5 as well and come into the test desktop see that it's getting an IP address in the correct scope and I can ping the router os's VLAN interface so if you only want to let the VM access VLAN 1 or your default VLAN you can put the VLAN tag as one so if I go in here and I set the VLAN tag as 1 . now it's not going to let it access VLAN 5. so now the now the desktop virtual machine on VLAN 5 can't get an IP so what if I want to give this VM access to some but not all VLAN tags well I can do that with the trunks command so here's the documentations is trunks vln ID separated with semicolons so to do that we go here into the shell and we need to edit the configuration file for the VM and if you're on the system the VM is running on in a cluster or if there's no cluster it will be an Etsy PVE local QE move server and then the vmid.com if you're not a cluster it might be a slightly different path but you'll figure that out so we have net here in this case you can see this is basically the same configuration we have in the giveaway we have Bridge tag and we want to add try so if I add the trunks command I can add vlans that the VM is allowed to tag so I'm going to say five and six separated with semicolons we'll save that and if we go back here it should show up here so you would only use this trunks if you only want to give the VM access to specific VLAN IDs and make the VM do the VLAN tag on its own if you want to use a router you can leave out the tag and leave out the Trunks and the VM will have access to all of the tagged traffic on the bridge vmbr0 or whatever Bridge you've assign it to I'm sure you've also noticed OBS here this stands for openv switch so OBS is a little bit different and that it's for managing large-scale cluster networks using vxlan which is a much more complicated but much more feature-rich protocol than standard vlans so we're not going to do with OBS today but hopefully this video gave you a pretty good understanding of the Linux Bridge the Linux Bond and the Linux VLAN and how to configure those in proxmox and when you might need them it's also important to remember that some of these bonding modes can still be used even if you don't have a managed switch for example the active backup or the balance Ron Robin or xor all of these will work with a dumb switch active backup will work for anything and then round robin and xor you'll end up where the traffic goes out on two ports but always comes back on one if the switch isn't managed lacp is what you should be using if you have a manage switch that supports it because you'll get balanced traffic from both sides automatically if you have an older switch that doesn't support licp you could try to use balance RR balance xor and set a corresponding manual link aggregation profile on the switch side but that's beyond the scope of this video for most things with modern equipment you should be using active backup or licp so thanks for coming along everyone on this proxbox vlans Bridges and bonds video hope you guys all had a great time if you have any questions or comments you can leave them down below if you'd like to continue the conversation or suggest topics for future videos you can find me on Discord there's a link in the description for that always love talking about proxmox and anything home lab or network related and as always see you on the next adventure boy howdy do we have a topic to cover today and it's networking Yeehaw time for networking and proxbox yo dog I heard you like networking it's one of those is the least cringe

Channel: apalrd's adventures

Views: 49,330

Keywords:

Id: zx5LFqyMPMU

Length: 25min 9sec (1509 seconds)

Published: Thu Nov 17 2022