How To Bond NICs In Proxmox That Support VLANs

when you first install proxmox the setup process will pick out a single interface card from your computer and then assign that to be the management interface but the network redundancy for servers typically what happens is two physical interfaces are bound together and then vlan tagging is used to give the hypervisor as well as the virtual machines access to multiple vlans so how do you configure proxmox with network redundancy and support for vlans well that's something you're interested in finding out then stick around and watch this video because that's what we'll be going over [Music] now i'm going to assume that you've already installed proxmox and that you've got remote access to it through a web browser like i have here in which case the first thing i'm going to do is to take a look at the existing network configuration or at least what the actual installation process does so over here in the left-hand pin i've only got one server to pick from but i'm going to pick that one which i've called pde-1 and then down here where it says network just below system i'll click on that then this gives me a graphical overview of the actual network setup now this particular computer has two physical interfaces and during the installation process proxmox it asks which management interface do you want to use now the defaults was enp0s25 what i could have done was i could have asked it to use this one instead enps70 but i just left it at that it also asked me what ip addressing do i want to use and that's what i've set up here so what proxmox has actually done then is to create a linux bridge called vmbr0 it's assigned this interface to the bridge and it's assigned the ip addressing to the bridge now during that installation process it didn't ask me do you want to bind these two physical interfaces together and yet that's a sort of a typical design for hypervisors you would have two physical interfaces usually high capacity interfaces you bind them together and then what you have are virtual interfaces within there so you have a virtual management interface a virtual interface to get access to storage and so on if you want a dedicated management interface that's usually an ilo card for example something that gives you dedicated management access directly into the console into the bios of the actual physical computer itself you can actually have a dedicated management interface like this but it's a it's a bit unusual it's kind of overkill i'd say unless there's some special thing you need to do with that management interface but usually in most hypervisors you would take two network interfaces bind them together put those into the actual virtual switch as this is a representation of and then you would actually start carving this up to give the hypervisor access to various vlans using virtual interfaces basically now at the moment this bridge is it's called the linux bridge it's not vlan where either so there's quite a bit of work i need to do here now it is possible to do all that through the graphical user interface it's just going to take a lot more time to do it especially because the ip addressing is assigned to the actual bridge here we want to be able to create a new vlan interface specifically in our vlan for management purposes and reassign the ip addressing to that and the easiest way to do that is through the actual shell if you will there's an option here to connect into the operating system by just clicking on shell or you can actually just get a remote ssh session so what i'm going to do next is to literally reconfigure proxmox to create a bonded interface to use that bonded interface on the linux bridge and to reassign the ip addressing to a new virtual management interface we'll create now because proxmox here is going to allow me to do copying and pasting through this web browser if i connect in through the shell i'm going to use that method to get access to the command line so i'll click on shell and then that takes us straight at the command line and logged in as root as you can see we're using debian linux so i'm going to change over to slash xc slash network which is where our configuration file is and i'm going to take a backup copy of that file in case i need something to revert back to and then you want to edit the file so what we've got here is a loopback interface a physical interface a configuration for the bridge and then our other physical interface so i'm going to make a space here because what i want to do is i want to actually move this line up to there so i'm just going to copy that line paste it into here let's go back down there and i'll just delete that line now our goal here is to take these two physical interfaces and bind them together what we do with them depends on our network infrastructure you know what we actually want to achieve here but i'm going to paste in some lines here to create a new logical interface called a bond interface and i've given this one a number of zeros so this is bond zero you can give it a different number if you like it's entirely up to you but traditionally in linux things tend to start as bridge zero one zero and so on so we've defined our bond interface and we said we want it to automatically start and then i've got a line here to tell it which physical interfaces to use so those are the two interfaces we've got defined up there the name of those interfaces are going to vary depending on the actual hardware you've got but that's what mine two are and one thing i'll point out is that you will want to use an even number of interfaces if you're going to be doing load balancing for example on physical interfaces the algorithms don't play well with any air with an odd number so if you've got four interfaces and one breaks then that's fine it'll still work with three interfaces but you do want to get back up to those four interfaces you don't want to leave it permanently stuck with three because the algorithms um aren't as efficient then the next line we've got is to do with the failure detection rate proxmox is example is using 100 milliseconds like i've got here so for the vast majority of networks that's fine if you make it lower you're going to make the detection more aggressive uh so it's going to depend on your computers your network infrastructure as to how it copes with that so it's entirely up to you if you want to lure it if anything you find it's unstable then you might want to increase it it's some of these things that you might want to fine tune it depends but to be honest 100 milliseconds is really quick um for the typical network anyway so in my case i'm just going to leave it as is now there is some more configuration that needs to be done on this bonded interface but i'm not going to do that yet simply because it really depends on your requirements it depends on your network capabilities now the rest of the configuration is going to be the same no matter what we do with that interface so i'm going to cover the actual bridge configuration the rest of the work that needs to be done that on that and then we'll come back to finishing off the bond interface in two separate sections so the next thing i'm going to do is create a vlan interface specifically for proxmox to use at the moment down here we've got our bridge and as you can see it's an interface has been defined on it essentially that's got an ip address and what it means is there's actually no tagging going on there's no reference in here to say this belongs to the 100 for example or vlan 200 it's set up as if it was a dedicated management interface which is what we don't want we want to take advantage of two interfaces and we want to be able to create lots of logical or sub interfaces each in different vlans for proxmox to tap into different vlans so in this case i've created a an interface called vmbr0.100 so the vmbr0 refers back to the bridge that we've got here the dot 100 and it says that this is a sub interface and it's um in vlan 100 now there is different conventions that you can do to configure that i'm just familiar with that strategy because i'm used to configuring routers and layer 3 stitches for example so to me it comes out as being a sub interface in proxmox it's a vlan interface and it's one of the ways you can define them but that's all we need to define our logical interface so that proxmox will have access to 100 and what it means is that going forward is that proxmox if it needs to access 192.168.100 as the network this interface is in that network and it knows now to actually tag the traffic that gets sent to the switch with a vlan id of 100. likewise when traffic comes in from the switch with a tag of 100 proxmox has an interface now in that vlan and it'll be able to look out for that vlan tag so because we've moved or more to find an ip address up here that matches this one we need to take those two lines away so i'm just placing the cursor down on this line so i can hold down the shift key press the up arrow twice hold down the ctrl key and press clear to delete the two lines and because we're no longer defining an ip address on this i'm going to change this over to be manual it's the same as the rest the the physical interfaces are set to manual the bond interface is set to manual it's only where we're defining an ip address that's going to where we define an interface that needs an ip address that's either defined statically or needs dhcp that it's going to be different now another thing i'm going to have to do is to change the bridge port here this tells the bridge which interfaces to actually use so before it was referencing this one single physical interface we actually now wanted to use a logical interface so the idea is that traffic gets sent to the bridge the bridge sends it to the bond and then the bundle decide which physical interface to send it to and vice versa when traffic comes in a physical interface it gets into the bond when the bond has the traffic it'll send it off to the bridge and then if it's destined for this virtual interface the bridge will pass it off to that if it's for a virtual machine the bridge will pass it off the virtual machine the last thing i need to do is add in two more lines here now these are to make this bridge vlan aware it has to be vlan aware to be able to do the vlan tagging so pretty straightforward definition i mean there's even a tick box that you can do within the graphical user interface for that another thing we want to set is the range of vlans that we're willing to accept on this actual bridge now if we go back to the example that they've got they're going from two to four thousand and ninety four now this makes sense because 4095 is a special vlan especially amongst hypervisors so they've removed that from the list of vlans that the actual bridge is willing to accept the reason we have limitations it's part of security really the idea is least uh least privilege if somebody doesn't need access to a specific vlan you don't give them access to a vlan now in our case what we're doing is we're kind of making the administration simpler really here the plan is that we make restrictions of what vlans can be used on the actual switch the physical network switch and then we just leave proxmox for example with access to all the potential vlans we could be using that way we don't have to configure both proxmox and the switch every time we need to provide access to a specifically new vlan now we so for that reason as i say we've got this complete range of old potential vlans we've removed five f4095 because it's a special vlan and we've removed one because it's it's a it's a best practice not to use vlan 1. vlan 1 is typically the only vlan that a switch knows out of the box i mean you'll tend to find in in offices for example bridges might be just left open on a desk for example or in a cabinet that's easily accessible might be in a reception room that's where there's not always somebody there in which case you don't always you don't want to have this potential risk of somebody being able to simply plug a computer into a port and being able to get access to other computers on your network so the best practice is do not use vlan 1. similarly cisco switches for example they actually use vlan 1 to exchange information between each other so it's especially important on cisco switches to not use vlan 1 because you don't want the risk of computers being able to get access to that information that the actual switches are passing between each other so in this case makes a lot of sense why they're going from 2 to 4 four now in my case i'm actually going to drop two vlans from the end now you can mix and match match this as you see fit um you can use other vlans if you like this you you might wanna just keep it at four thousand and ninety-four it's entirely up to you but because i've been using cisco switches that long i'm now used to having what's called an unused port and an unused vlan the idea is that if you've got switches out there and some of the ports aren't used you put them into an unused vlan in other words you don't leave them in vlan 1 you don't leave them in an actual production vlan you don't want the risk of somebody coming along and getting access to a vlan they shouldn't do so by putting them into an unused vlan they can't get anywhere they can't get access to anything so that's one of the reasons i need to drop one vlan the second vlan i need to reserve is for what's called a nada vlan now when you create trunk ports and we're going to be using 802.1q tagging to do this now that has a special vlan called the native vlan and in in the 8021q configuration you have what's called an untagged vlan or at least a way to deal with it the idea is that if you've got a computer and it's plugged into a trunk port if it sends the traffic without a tag the switch needs to know what to do with it and for that reason you have a native vlan so the idea is that when the traffic hits the actual switch if it doesn't have a tag the switch assigns it to the nade of the vlan but you don't want that native vlan to be in use anywhere on your network you don't want somebody to be able to send on tag traffic and get access to places they shouldn't be so it's always recommended for that reason to have another vlan set aside which is unused on your network so that's why instead of ending at 4094 for example i'm going to in this example at 4092 because i'll reserve two vlans one for unused ports and one for the native vlan id so the next thing we need to do is really just finish this configuration off by telling the bond how to behave do we want to have network redundancy do we want to have a higher throughput or do we want both well it really depends on what your network's capable of and what you're looking for so the next two sections are going to cover those options now one of the common reasons for wanting to combine multiple interfaces together is to give our computer access to more network capacity so at the minute proxmox is only using one interface which is capable of one gig that means the overall network traffic for our virtual machines and proxmox cannot exceed one gig but what i can do is set up this bonded interface combine two one gig interfaces together and then i'll be able to get access to two gig instead to do that i'm gonna copy and paste in what proxmox themselves recommend for load balancing traffic now that's these two lines here which i'll explain in a minute but basically what we've now got is two interfaces and they've being configured together as a bonded interface a single logical interface now whereas before if we had say four virtual machines but only one physical interface all that traffic would have been on one physical interface now because we're using as load balancing it means the traffic's going to actually get shared so we might see for example two of the virtual machines have their traffic on this interface and then two of the virtual machines have their traffic on that interface overall the network capacity has increased and overall our virtual machines will see a better throughput what i will point out though is that one individual virtual machine connecting to another single server will not see a benefit beyond one gig in the sense that if i've got a virtual machine trying to connect to a web server which is reached through the physical network i can't get more than one gig of download for that session simply because the load balancing doesn't work like that it's balancing all of the traffic it's looking at all the different connections between all these computers that are talking to each other and it's balancing them between the interfaces what it isn't going to do is it's going to look at the session between the virtual machine and the web server and then split it up between these two interfaces so either way one individual session from one computer to another will have a limit of one gig but overall we've doubled the network capacity by combining our interfaces together so that's just something to bear in mind in terms of expectations here now what i've got is two interfaces but you can have more but you can't mix and match different speeds so i can't have a bonded interface made up of a one gig interface and a 10 gig interface i could have four one gig interfaces but that's just because they're all the same speed if i have a mixture of one gigs and 10 gigs then i need one bond that's using the one gigs another bond that's using the 10 gigs so just bear that in mind you can't mix and match speeds and like i said you can have more than one more than two interfaces that i'm using here but try to keep them to an even number because that's better for the algorithm it'll still work if you've got an odd number of interfaces but you're better off using an even number of interfaces when it tries to balance traffic over an odd number of interfaces it doesn't uh work as well but it does potentially work in the sense that if you've got four interfaces and one breaks it'll still work you just won't see that even balance and in order to be able to get the most efficiency out of this you want to be able to see an actual even balance of traffic over all of the interfaces you've got now what we've done is gone with proxmox's recommended settings for load balancing so that is setting the bond mode to 802.3 a d which is an industry standard known as lacp or link aggregation control protocol these sort of settings are pretty much the industry standard if proxmox are recommending them i would suggest going with them myself i have used other settings for other sort of network scenarios but this is what proxmox suggests so i'd say stick with that if you want to change the bond mode by all means do so just bear in mind that whatever you've got on your switch needs to be in sync with what you've got on your computer so for example let's say we put lacp on the network switch if we configure adaptive load balancing in the as the mode on proxmox the two don't work they've got to be similar at least so in this case they're suggesting put lscp as the bond mode that you're going to be using for load balancing in which case we want lacp on the switch fortunately even most of the typical retail switches out there will support it anyway so we should be fine doing caveats is that both of these interfaces have to be plugged into the same network switch so what i can't do is take this switch plug it into one i can't take this interface plug it into one switch then take this interface and plug it into another switch and use lasap it doesn't work unless that is you've got high-end enterprise-grade data center switches which are capable of supporting what's known as m-lag multi-chassis link aggregation protocol that's a pretty sophisticated switching environment which allows you to take two physical switches for example plug your computer into two separate physical switches and still take advantage of lacp like this the configuration for proxmox would still be the same but the switches requires a lot more sophistication so for the typical user if you've got say like your normal retail switches small business users lab users this is perfectly fine you would just plug your two interfaces into the same switch and that's it the downside is it means that if your physical switch breaks proxmox has no longer access to the uh the network which to be fair if you're only you know using one interface in the first place that would have happened anyway you would have lost your network connectivity if that network switch had gone down so just do bear on that in mind you don't get network redundancy out of this you're getting access to that throughput unless you're using these high grade enterprise capable data center switches to support that feature now the transmit policy that we're using here is layer 2 plus layer 3 which means proxmox is going to look at the layer 2 information and the layer 3 information to decide how to balance traffic over these two interfaces that i've got so look at the mac address look at the ip address and i'll do some comparisons to look up for differences so where it sees different traffic flows between computers it'll then start to split those traffic flows as best it can over these two interfaces that i've got but that's as far as it'll go we could go into something more sophisticated like three earlier three and layer four where we even start looking at the application level and start splitting even different applications over these individual interfaces uh by that i'm saying if we've got a virtual machine connecting to the sim computer on there on the physical network and that computer is acting as a web server and an ftp server this policy can't distinguish between web traffic and ftp traffic with layer 3 and layer 4 it could boot what proxmox are saying is that has a problem at least for them using 802.3d in which case they're suggesting stick with layer 2 plus layer 3. now that is the default on most switches and typically whatever you put on a switch you'd want on your computer as well so again since proxmox suggests sticking to this along with lscp then yeah i would go with that as well so all i need to do now is save these changes and then what i can then do is reboot proxmox itself for these changes to take effect now although proxmox has been rebooted and it's back up and running unfortunately i can't get remote access to anymore but that makes sense really because we've reconfigured the networking on that damn actual computer and it's completely out of sync with the actual switch proxmox is now expecting to see lacp to buying two interfaces together it's also expecting to see vlan tagging now how you reconfigure your network switch really depends on well not just the actual switch vendor but even possibly the family of switch that you've got from that vendor it can vary that much so there's so many possibilities out there so many different ways in which switches get configured but i'll go over how i would configure it on this particular cisco small business edition switch that i'm using here and hopefully as you follow along you'll have a better understanding of how to configure your switch so the first thing i need to do is to actually set up lacp now for this particular switch it's really easy i'll click on the port option i'll then click on where it says lacp and then i'll tick these two boxes because the computer that i've got that's running proxmox is plugged into ports one and two so i've enabled lacp i'm not gonna bother setting a specific admin key i'm just gonna leave it to create one itself and then i'm gonna click on save and that's basically it it's now enabled lacp if i go to the lacp status option it's saying that the protocol is active but that's pretty much it that's the best i can get out of this now it might vary for your particular switch so if we refer back to proxmox for example we actually created a logical interface called a bonded interface bond zero and you might have to do something similar with your actual network switch it might create its own logical switch uh using nacp and then you use that for your configuration going forward in my case i just had to enable lacp on those two ports but i just wanted to point out you might actually end up creating a completely new port and then you start configuring that port going forward so as an example this switch supports static link aggregation if i'd used that so let's say our computer was plugged into ports one five what i would do is create a completely new interface called lag one and then going forward i'll do all my configuration changes on this new lag one interface so you might be doing something similar with your switch in my case as i said i've just got to configure lacp on the two ports and that's it and for me it means i've got to now make changes to the actual physical interfaces whereas for you you might be making changes on a new logical interface so that's just something to bear in mind but the next thing we need to do is to enable vlan tagging now the next thing to do is to configure vlan tagging on the switch how you do that really depends on the switch that you've got i'm going to make some changes to this one and hopefully that'll give you some guidelines as what you need to do so i'm going to click on vlan then i'm going to click on vlan port settings now if your switch has either created a new logical interface or it's assigned your interfaces to some built-in um aggregation interface then you'll be wanting to change the configuration of that logical or aggregation interface whereas for me this switch doesn't do that at least when it comes to lacp which means i've got to make changes to the actual physical interfaces and i do have to do same changes to both physical ports these two ports port one and two which proxmox has plugged into have to be configured the same there might be another setting on your switch which says that the port's an access port or a trunk port the trunk port is used to connect to a computer that supports vlan tagging so if it's set as a access port then you'll want to change that to be a trunk port i don't have to do that this particular switch only has trunk ports all of the actual ports can support multiple vlans and there's nothing uh for me to tick no box to change or anything to say oh we're now an access port or we're now at a trunk port they're just trunk ports anyway so at the minute what's going on is the switch is set up to not send tags to proxmox because the computer that i'm using here is patched into podiate the primary vlan id is basically telling the switch port what to do with untagged traffic so when the traffic comes into port 8 the switch is putting a tag of 100 effectively onto that traffic it then makes its way through the switch and then it would go out to port one when it reaches port one it has a look and see what the primary vlan id is on that port and then it sends it over to proxmox now because the vlan id or at least the primary vlan id for this port is 100 and the traffic that was sent to the port had a tag of 100. what the switch is doing is it's removing that tag and forwarding onto proxmox without the tag and that's not going to work because proxmox doesn't know what to do with it it needs to see a tag of 100. so what i need to do is to actually change this primary vlan id to be something else so again for your switch it might be referenced as the default vlan it might be the primary vlid it might be the uh the untagged vlan it might be the native vlan it'll vary but you'll see something similar to this now i need to use something basically that's just different to what 100 is and that's going to be called the native vlan so what i really want to do as a best practice is to have a native vlan that isn't actually getting used anywhere so what i'm going to do is go over to vlan settings in my case i'm going to select what will be my native vlan 4093 click on modify and tell it to allow it on those two ports then click save i'll then come back to my vlan port settings and change the primary vlan id to be 4093 then i'll click save 4094 incident b is my unused uh vlan so these ports here three to six aren't being used to all been assigned to 4094 4093 will be the need of vlan for me or i can reveal any id if you will so that's why i've set them to 4093 and what that means is that when this computer sends its traffic to 0.8 it'll still get tagged with 100 when it gets sent over to port 1 or port 2 in this case going forward the switch is going to have a look at what the primary vlan id is now this time it doesn't match so it'll actually retain that tag of 100 when it gets sent out to proxmox but i need to make some other changes while i'm here i'm going to select these two options here now this setting here it might vary a bit for you it's it's the acceptable frame types and it's to decide what type of tags or unknown tags if you will that will accept from the actual computer that's plugged into the port by default it's set to all on this switch now your switch might have an option for untagged only and tagged only and all mine doesn't have this concept of untagged but because when i'm setting this up as a trunk it's a good security practice going forward to only accept tag traffic so i'll click on save and again i'm making the change in my case to both physical interfaces i've got to keep these two ports in sync what it'll mean is if proxmox were to actually send any traffic across that doesn't have a tag the switch will drop it so that's a good security practice to have another thing i'm going to do is deselect these two options here the reason for that is what this does is it forces traffic out the port to be untagged that makes sense if it was an access port and when you've got a computer that's just plugged into the port which doesn't support any tags there's no point sending tagged packets across to a computer when it doesn't understand them anyway so i had that set up because originally these ports were actually behaving similar to actual uh access ports in a sense they had no concept of tags there's no point actually sending any so i've now made some changes to this it might be slightly different for you i mean you might be able to add an interface add a vlan under the actual trunk within this page for example in my particular case i had to go out to the vlan settings page and i had to modify the actual vlan itself to tell it what ports to put it on as with some switches you just do it on the actual interface itself you just say oh you can now accept these vlans so it's it might vary but hopefully that gives you a bit of a better understanding so now we made those changes we're going to see if we can get access to proxmox again so at the moment it can't so i'm just going to stop that and hit refresh so this one has connected in if it hasn't sometimes what i've found is just click the cross to stop the the session refresh it sometimes if you ping the ip address it gives it a bit of a shove but looking on the network configuration now i mean basically we've restored connectivity when we look on the network configuration page you'll see we've now got a buttoned interface bond zero and there's the two interfaces we've got our linux bridge but the ip address isn't assigned to that anymore and the actual interface it's using is now the bonded interface we've also now got a new sub-interface or a linux vlan interface as it's now referred to and that's where our ip addressing has gone to and also the bridge is now vlan aware so what this means now is um i'm actually in a situation where proxmox has got access to two one gig interfaces which means we've got access to more network capacity we do actually have some redundancy here because if one of the network interfaces breaks if we get a cable break we do actually still have redundancy what we're missing out on the was that if the switch itself breaks well that's it proxmox doesn't have network connectivity at all so we're still getting some benefits for network and connectivity but the real benefit here is we've now doubled our network capacity effectively uh so our virtual machines and proxmox can send essentially double the traffic that we could before so just to demonstrate that actual redundancy if i start a ping test to this computer so that's 192.168.1.10 and i'll just set it to a permanent ping you can see it's pinging away and it's getting a response so i'm going to go over to the port settings on this i'm actually going to disable interface number one so now there's only one interface working now it always varies depending on the result you might actually see one packet get dropped for example and then it'll recover in this case that didn't happen but you can see we're still working even though we've only got one interface up it's still happily responding so i'm just going to restore that because obviously if i disable both interfaces it'll completely stop working anyway so i'm just jumping from one page to another because sometimes it can take a while to refresh so i'm still getting responses back so i'm now going to disable port number two click on save because the response you actually get really depends on which interface we're using but you can see here the request timed out so it dropped one um packet essentially when it was sending or packing out and sending a reply it must have gone over that second interface or it was trying to send it back on that interface you can see it quickly recovered so again now we're just operating on one link but we're still being able to access proxmox so we still do have our network redundancy to a certain degree we can tolerate a link failure we can tolerate a cable cut and so on but what we can't accept is a complete break of the actual switch itself if the brake if the switch itself has a hardware failure if it gets you know rebooted as part of a maintenance upgrade and so on proxmox would lose access to the network you can see it's pretty quick to recover and the real benefit is the fact that we're getting double the bandwidth more capacity for our virtual machines to use now the most common reason for binding two interfaces together like this is to give us network redundancy in other words even if a network switch or cable actually fails we still want proxmox and its virtual machines to have access to the network so to do that i'm going to copy and paste two lines into this configuration just to finish off the config for bond zero so this line here is setting the mode to active backup and what that basically means is one interface is the active interface in other words it's actually the only one that's operational and then the other interface will be the backup interface and its purpose is to just basically sit there and if the primary interface or active interface actually falls over or loses connection to the actual network switch for whatever reason then this secondary interface or backup interface takes over and it restores network connectivity but we have to decide which of these two interfaces is going to be that active interface so as the configuration here refers to it as the primary interface so the wording can be a bit confusing i must admit in some vendor solutions what you have is an active passive or an active backup a primary and a standby and there's a lot of different ways they're referred to these but as far as this configuration goes the mode is active backup and then you just tell it which one's the primary interface so basically which one is the active interface so what this means basically is that once these changes take effect this interface here enp0s25 is the interface which should be used for proxmox to send traffic to the network and the network to send traffic to proxmox this interface here won't be doing anything it'll just be sitting there in the background just waiting on the off chance it's maybe a network switch or the network cable breaks this one won't be able to get access to the network anymore in which case this one will take over and it'll restore network connectivity so i'm going to save these changes and then in order for the actual changes to take effect i need to actually reboot proxmox now a lot of proxmox has been restarted and it is now back up and running the problem is i can't get more access to it i mean i can click on links i mean it looks as though i've got some connection but the reality is it's it's just gonna eventually time out i mean if i hit f5 for example or control f5 eventually i'll actually get told that it's just you know timed out now the reason for that is because proximox has been configured to support vlan tagging but the switch doesn't and the reason for that was that as part of the initial installation we needed to be able to get direct access to proxmox without any vlan tagging at all because there's nothing within the actual configuration as part of that default installation to actually set specific vlans for the management interface for example this is something you have to do after the installation is being done so in my case i'm going to have to reconfigure my switch to actually support vlan tagging now how you configure a switch to support vlan tagging is going to vary it's going to be different you know depending on the actual vendor of the switch it might even be different if you've got a different type of switch even from the same vendor i mean anybody who's got a career in cisco networks for example knows this looks nothing like a that's like an enterprise switch by cisco for example but this particular one i've got it's a cisco small business edition switch and it's very easy to set up it's it's geared towards well obviously small businesses home offices and so on so it's pretty straightforward to configure in this case we just want to change you from what is an access port which is intended for computers that don't support vlan tagging over to a trunk port which is for computers that do actually support vlan tagging so what's going on at the minute is that this computer which is plugged into port 8 is sending its traffic into the switch and because the actual switch has been set up for an access port in this case as i'll refer to it as it's actually putting a tag of 100 on that traffic as it goes through the switch and then when it gets to port 1 it has a look at this heading called the pvid the primary vlan id it sees that it matches 100 and then what it does is it removes the actual tag now what i need to do is to actually change this to something different i need to actually retain that tag but how you do all this really depends on your actual vendor so for example you might actually have a column in here for instance on your port which actually says if it's an access port or a trunk port if it's configured to be an access port you need to change it over to a trunk port that's how we'll be able to support the actual vlan tagging going forward now for this particular switch that i've got here it doesn't have that concept of access ports or trunk ports everything is a trunk port you just keep adding vlans to it now it refers to this this column here as the pv primary vlan id some switches are referred as the default vlan some are referred to as the vlan this one just calls it the primary vlan id the principle is the same basically it's it's the vlan that you actually use for traffic that doesn't have a tag in other words the intention is like in this case for this particular computer for example for example it doesn't support tagging in which case the switch needs to actually add the tag and likewise if you're going to be sending traffic then back out to a device that doesn't support it we need to know which vlan id to actually remove at the end so like i've said that the problem i've got at the minute is that i've got a computer in vlan 100 i've got proxmox supporting vlan tagging and as far as the switch is concerned any traffic that's destined for port for vlan 100 should have the vlan header removed but i need to keep that so i need to change this setting to something other than 100 now to do that i need to actually add in an extra vlan so it's going to vary depending on your switch with some you just reconfigure the port add vlans as you go along this particular switch i'm going to actually modify the vlan itself and assign it to port so i'm going to go over to my vlan settings here and i'm going to pick this vlan 4093 this is going to be my native vlan the idea is this is an unused vlan i i don't want to be actually getting used i don't want any computers using it so i've picked out this specific vlm it's also why i've got proxmox not using it 4094 incidentally is for my unused uh port so any port that isn't in use will get assigned to 4094. again the idea is other computers out there if they accidentally get plugged into a port that's not in use there's nothing actually to you know connect to really but in this case i need to assign 4093 to port 1. so i'll click on save and that makes a change there i'll go back to this port and i'm going to change the primary vlan id to be 4093 so the id now is that when my computer sends traffic to port 8 the switch is going to add on a vlan of 100 and then when that traffic makes its way over to port 1 it exits out the interface and then the switch will say all right well 100 doesn't match the 4093 which is the native vlan the deep bold vlan the primary vlan id or whatever your switch refers to it as in which case it actually retains that heading of 100 so it comes in without any tag but exits with a tag another thing i'm going to do just for security reasons i'm going to set this setting here which is the acceptable frame type to tagged only and what that means is that going forward will only accept uh traffic from proxmox that has attacks that's a security feature we're expecting tagged traffic so we should set this to say we only expect tag traffic on some switches they'll have a setting for all untagged or tagged only in which case you've got more options and i've got here for me it's just either all or tagged only but i do want to set it to tag only because i don't expect any untagged traffic from proxmox therefore i should actually be dropping it another setting i need to do is to disable this feature here which it forces traffic out of the interface to be untagged it makes sense if it's an access port because it's not expecting any tags but now the proxmox is expecting tags i need to remove that feature so we'll go back to proxmox click around you can see i've actually now got access again and that's just because i've i've updated the actual tagging and so the switch is now supporting tagging now what i need to do is i've actually got two switches because we've we're setting this up for redundancy so i need to repeat the same process on switch 2 here because that's not supporting that feature one thing i'll point out is that these two switches are linked together the need to be and that gives me my redundancy so the idea is interface one plugs into this switch on port one interface two plugs into this switch on port dare one but i've got a link between the two switches so my computer which is on switch one is getting access to the actual computer here uh through switch one going over to switch two for example to get access to interface two so that's something to bear in mind is my computer doesn't have redundant access but uh proxmox does but you would have to have that link in between the trunk link between your two switches for all your computers to have access to each other regardless of which interface they're on so i'm just going to finish off the configuration um on switch 2 here and then we'll test out our redundancy well switch two has now been reconfigured uh so they're both configured in exactly the same way for port one because port one of proxmox has plugged into port one of switch one and it's port two is plugged into port one of switch two so we'll just have a look at the network configuration now now if we go to system and then to network you'll see we've now got a bond zero interface which is made up of our two physical interfaces and the mode is active backup if i select that click on edit you can see here where it's been configured so that enp 0s 25 is the actual primary and then we've got our two physical interfaces our linux bridge is still here but now it's vlan aware and it's referencing bond zero now instead of originally enp0s25 and then we've got a new interface the sub interface or vlan interface called vmbr0.100 so this is our new management interface so it has the ip addressing that was originally assigned to the bridge itself and what that means is that going forward proxmox is tagging traffic for this management network with a vlan of 100 and it's expecting traffic for this management interface to be tagged with a vlan of 100 so we've now got a vlan setup all our vlan tagging is now working and proxmox itself is vlan aware we can actually assign virtual machines with vlans we can create new interfaces for proxmox and new vlans but we want to test out next is actually the actual redundancy does it actually work so i'm going to open up a command prompt and then what i'm going to do is i'm going to start a continuous pin because at the moment this interface is the only interface that's actually doing anything the other one's just sitting in back up that's that's the expectation so i'm going to ping proxmox in this case it's 168 100.10 and i'm going to set this up as a continuous ping and that's how we do that in windows and just by putting on a t as a parameter hit return and you can see we're getting responses back from proxmox so i'm going to go over to switch one uh i'll go up to port and then port settings and i'm actually going to disable interface number one from the switch so this is sort of like the equivalent of a network switch failure basically or a cable failure now i can't literally feel the actual switch itself because then my computer wouldn't be able to get access to proxmox at all so this is like the closest i can get as far as proxmox will be concerned that interfaces suddenly disappeared so we can see it's timed out sometimes it does sometimes it doesn't it just depends on how quickly it can recover but it's basically what it's done now is that this interface here will have gone down and then what it's done is then brought this interface up and made it more active essentially so what we've now got is we're a situation where instead of actually sending traffic backwards and forwards on this interface we're now using that interface but the key thing to point out is only one of these interfaces ever active at any given time that's that's what we want really otherwise it causes a lot of confusion there are other modes you can use um switches just and network administrators they don't like it if you've got uh two active interfaces on two different switches the mac address starts jumping around and it causes all sorts of problems for management and it's yeah you just don't want to do that so if you're interested in network redundancy then you want one active interface and one standby interface or as proxmox calls it and one active and one backup anyway what we're going to do is we're going to restore this interface back again so i'll click on save i just need to click away and then click back and then eventually the link will come back up again so that's now back up again so you might get up yeah we've got a bit of a disruption again so basically what it what proxmox what then does is it it then disables this interface and then restores this interface so like i said there's only really one interface is actually being used it says they're both active it just means as far as uh the system's concerned yeah they both have the potential to be used but as far as linux concerned it's a case of one is actually there's only one interface actually being used at any given moment but it means that we've got full network redundancy so if this actual interface were to lose connection to the network so it could be a cable failure it could be a switch failure maybe the switch is undergoing maintenance getting a software upgrade and it's rebooted for whatever reason if this particular network interface is actually losing access to the actual network the operating system linux here is actually detecting that and it'll switch over to the other interface now like i was saying i mean although i've got two switches i've only got one computer which is plugged in when i'm doing my testing with and it's only plugged into one switch so i can't genuinely simulate a complete switch failure but the principle would be that all your computers would all be plugged into both switches and it wouldn't matter if a switch failed they'll all be able to still get access to each other so pretty straightforward to set up and the next thing is to start looking into providing proxmox with its own vlan interfaces in different vlans so it gets direct access to other networks there's also how to actually do the vlan tagging for the virtual machines now it doesn't really matter which of these methods you're using whether we've got two interfaces that are bound together like this as an active backup or we've actually got multiple interfaces bound together to give us higher throughput what we do going forward is basically the same if we want to be able to give proxmox access to another network so for example let's say we want direct access to a storage network uh we don't want to be sending out traffic through a firewall because it slows down so we have a dedicated vlan which gives direct access into banas asan whatever that storage shared device is it means we need another interface for proxmox there's nothing beyond that actual network it's an isolated network all the actual proxmox will get access to is the actual storage device but we need to create a new interface on proxmox now you could actually go back into the configuration file if you want but you can also do it through the actual gui here so what i'm going to do is i'm going to click on create up here and i want to create a linux vlan now i've got another vlan which is vlan 200 so i'm going to keep in sync here so it's vmbr0 if i can type properly so that's our bridge but this time i want to reference vlan 200 and you can see automatically it's populated the vlan tag of 200 so it's saying that the raw device is vmbr0 in other words the linux bridge that's vlan 200. now i'm going to keep things in sync here so its ip address will be 192 168 100 and 200 rather ten you don't have to do what i'm doing here which is to keep the third octet in sync i mean once you get into uh actual you know large networks with which you've got you know thousands of vlans it's just not practical you can only go from zero to 255 for example so it's just not practical but in this case it's a small network so i get away with that so that's its ip address and then we've got to give it its actual subnet mask or side or length in this case how many bits make up the subnet and then that creates a new vlan interface so i'll click on create now as you can see it's not active we actually need to apply the configuration so i'll click on apply and say yes and then off it goes now this is just going to reload the configuration uh the network settings in the background which means we don't have to actually reboot proxmox itself for this to take effect one thing i'll point out though is that i did not assign a gateway to that interface and you don't want to especially when you've got firewalls in your network you would only ever have one interface that has a default gateway like this one here otherwise it causes confusion it'll start to load balance traffic between multiple interfaces if you start hitting firewalls you'll start to find things break you get intermittent outages and so on it it just causes all sorts of pain basically so all you want to be doing is setting one interface with a default gateway like this one here for example everything else do not give it a gateway in this particular case this is fine let's say 200 as our you know storage network there's nothing beyond that vlan anywhere so it doesn't make sense to give it a gateway anyway so we've now created our actual vlan interface on proxmox but we need to actually allow access to that vlan because what we did when we set this all up was we allowed pretty much every vlan on the actual trunk for proxmox but not for the switches so we actually need to update the switches so that the actual proxmox server actually has access to 200 because at the minute if proxmox would have tried to send any traffic tagged with 200. the switch interface isn't actually configured to support it and it'll just ignore it and that's a good security feature so i'm going to go over to my switch go to vlan settings i need to log back in again and what i need to do is to actually allow vlan 200 to that specific port because it's a good security practice only allow access to you know vlans that a computer should get access to and by my doing that on the actual switch it just makes the administration easier i only need to update the switching up the switch and proxmox but in this case i'm going to have to do it on both switches because if i forget to do the second switch the problem you'll find is that if um if the primary switch failed or proxmox just lost access to it and the secondary switch didn't support access to vlan 200 then things wouldn't be working anymore so you've got to remember to do it for both when you've got an active passive interface like this likewise if you've got a switch that's got multiple interfaces being used by proxmox again you've got to update all of the physical interfaces if you're doing it on a switch like this if it's a logical interface it's easy just update the logical interface but in this particular case if i go and have a look at the list here we've got a range which includes 200 if we go to this one that one includes 200 one thing to point out is i'm not changing the primary vlan id that's just the native vlan once you've set that you just leave it as is now this particular computer it's 192 168 200.10 so i'm just going to stop my ping actually because i don't need that anymore assuming it'll let me so if i now try to ping 200.10 you see i'm actually getting a reply back and the reason for that is this computer does actually have two interfaces uh one is plugged into port eight which is assigned to vlan 100 the other is plugged into port seven which is assigned to vlan 200 the computer's actually got access to two vlans but it allows me to test and just demonstrate how i can set up a an extra interface on proxmox in a another actual vlan here in this case 200 and show you that in my case i've got a computer on the physical network getting access to proxmox and you can do it in both vlans and now that we've set it up like this so the next thing to do is to go over how to set up the actual virtual machines to get access to vlans now because proxmox and our network switch are vlan aware in other words they're exchanging vlan tags we need to make sure that our actual virtual machines are actually tagging their traffic with a with a correct vlan so what i want to do is i'm going to select this virtual machine over here and then i'm going to go to hardware now we then need to pick an interface that we want to change the vlan tag on because by default it doesn't do any tagging at all now in this particular case this specific virtual machine has only one interface but if it's got multiple interfaces you're going to have to put each of them into their own proper vlan i mean typically you might have like a server for example running as a virtual machine which has a like a management interface for example or a storage interface maybe it's got a completely separate backup interface a front-facing production interface and so on but in this case the principle is the same the idea is just select the interface click on edit and then where it says vlan tag we put the actual vlan number that we're going to tag the traffic for this interface with so in this case i'm just going to use 100 i'll then click on ok and then i'm going to start this virtual machine up and basically what this means is that going forward this virtual machine will start tagging traffic on this interface with the vlan 100 so it'll be able to get access to other virtual machines within proxmox for example that are also in vlan 100 it should be able to get access to any device reachable through the physical network on vlan 100 as long as the network switch supports vl900 on the interface i mean the way we set up proxmox itself it supports anything from two to 4092 for example so got a lot of flexibility one thing i'll point out though is that your virtual machines can be in different vlans to proxmox i i don't necessarily have to add a new vlan interface to proxmox just to get a virtual machine access to it so for example let's say i wanna an interface in vlan 300 i don't have to create a vlan 300 on proxmox i can tag that traffic as vlan 300 on the virtual machine i just need to make sure that the actual switch supports vlan 300 on that interface as well or rather all interfaces or in this case both switches would need to support vlan 300s that's that's just something to bear in mind we've set proxmox up to support a wide range from 2 to four thousand and ninety two or it could be two to four thousand ninety four if you've gone with uh proxmox is owned suggestion but it's a case of you do not have to have proxmox in a vlan that your virtual machines are there's no specific need for it anyway i'm going to go back to my virtual machine i'm going to see if it's up and running now so i'll just click on console and i'm going to log into this one because i need to find out what a type address is because the idea is i want to see if i can get access to the physical network from this virtual machine and likewise if i can get access to this from the actual physical network as well so i'm just going to select the menu option there just open up a terminal session find out what its ip address is so this is dot 50. so if i ping 192.168.100.12 i think the switch was so let's get your response back from that switch and there isn't a itself is 10 so it's getting a response facts back from proxmox so if i try to ping 100 so my computer gets a response back which is to be expected anyway because if this computer can get access to something on the physical network something on the physical network can get access to this the reason being is that this out of the box there's a linux computer it doesn't have a software firewall active on it windows does so for example i can't ping my windows computer from this linux computer just because it's got a software firewall and it will drop the traffic that's just something to bear in mind but now that we've got the tagging enabled i mean i can create other virtual machines on proxmox and put them in the same vlan i could have other computers anywhere that's got connectivity through the network on vlan 100 and they'll all be able to talk to each other so it's again pretty straightforward you just need to update the vlan tag but just remember that you need to do that for every interface if your virtual machine has multiple interfaces well thanks for making it to the end of this video i really do hope you found it useful if so then do click the like button and share because that encourages youtube's algorithm to suggest it to other people who might find it useful as well if you're new to the channel and you'd like to see more content like this then yeah do subscribe just remember to click the little bell icon though that way you'll get notifications when i send new content out if you've got any comments any suggestions if you want to leave any feedback at all please post that in the comments section below and if you'd like to support the channel i've left links to both patreon and paypal in the description below but above all thanks very much for watching i'll see you in the next video [Music]

Channel: Tech Tutorials - David McKone

Views: 24,393

Keywords: proxmox vlan aware, proxmox vlan bridge, proxmox vlan trunk, proxmox vlan aware linux bridge, proxmox vlan interface, proxmox networking, proxmox networking tutorial, proxmox networking vlan, proxmox, proxmox ve, virtualization, proxmox tutorial, proxmox network, proxmox network vlan, proxmox network bond, proxmox ve network configuration, proxmox network setup, proxmox bond, proxmox bond nics, proxmox vlan, proxmox vlan tagging, proxmox redundancy

Id: nIip66Rzt4I

Length: 65min 13sec (3913 seconds)

Published: Sun Apr 17 2022