Don't Use A Firewall, Use 2! OpnSense High Availability Guide

hey everybody and welcome back to Jim's Garage in this video we're going to set up open sense High availability that means having more than one firewall in this case open sense if you watched my previous videos you know that I run High availability with sofos XG and whilst this process is a little bit more involved we'll have you up and running by the end of this video so you might be thinking why do I want High availability and that's a good question if you're using something Hardware base say a UniFi dream machine or maybe some X Enterprise kit then you likely have an independent firewall on bare metal but the same principle applies what happens if that device fails most likely your internet's going to go down and probably your network that's where High availability comes in and you can do high availability with physical kit and that comes with the benefit of double the price but how about in the virtual world what does this mean well there's probably a fair few of you here who are using a virtual firewall and the same principle applies what happens if that virtual machine goes down it could be a hardware failure or it could be quite simple you're rebooting your node I.E your proxmox host where you have this that has issues because it brings down your network it will bring down internet for the rest of the network and it can also be a whole heap of problems if you're trying to remediate something and you don't have internet access so how does this fix things well by the end of this video you'll have two firewalls on two discrete machines so this means you could do some admin work on one of your proxmox nodes and hopefully your internet and networking will fail over to the other node so you shouldn't notice really any difference in activity and network connectivity this has saved my bacon more times than I care to mention and I'm really glad that I've gone down this route so let's dive into how this works and first I'm going to show you some Network diagrams with how you can set this up but there's a couple of gotchas that we need to talk about as well once we've done that we'll jump into the deployment about how you can configure this you are going to need two firewalls set up two virtual firewalls but this will also work with bare metal now the first key issue or problem we need to resolve with open sense is that it uses CP the common address resolution protocol and typically this is designed to have two actual dedicated internet links or at least two separate IP addresses and this is a problem for most residential isps as you only get one IP address if you do get more than one you can skip this next part and you can jump straight into the configuration later in the video but I'm going to assume that you've got one IP like I do so what can we do about that well as I said it expects two different IP addresses because C requires access and control over the interface now thankfully and this is the way that I'm going to have this configured in my video we can kind of trick open sense to thinking it's got two IP addresses how do we do that well I'm going to jump into some diagrams now to examine the problem and then give you a solution so if we look at this first diagram here this is the proposed setup on the open sense website you'll have your internet coming in you'll have an ISP router you could have two if you have discrete connections but typically this would be on say a /29 or something like that where you have more than one physical one IP address this would either then typically go into a switch with vlans and be connected to the respective firewalls or you could have dedicated connections from here into your router if it's configured such the important bit here is we have a link between the two firewalls and this will be the same for all of the diagrams I'm going to show you this is a physical connection between the two albeit you can virtualize it like I do and what this does is provid provides a link between the two so they've got high availability it can copy configs over it can do heartbeats and all of that stuff it actually uses virtual IPS and we'll come on to that later now out the back of these firewalls you'll have the Lan which will go into a switch and then all of your clients will connect to that switch so you've probably got this down here this here this here and this here already so you can see that we're basically adding another file W on and doing some clever rooting now as I mentioned this is what they recommend and it's probably more akin to an Enterprise setup where you'll have multiple switches what I actually have and how I'm configuring in this video is this setup here so very similar however I've removed that second switch that's because the respective Lamport on each of these firewalls actually goes back into the switch on separate vlans and then all of my clients connect to the same switch on those respective vlans so now let's solve the problem so one way you could do this is to have your ISP router still in router mode not modem mode now this does create the double net issue so things like firewall rules you'll have to create an at rule on here first and then again on your firewall and some Services might have issues with that but broadly speaking you should be okay now what does this approach do typically your ISP router will have a small switch baked into it usually four ports now what we can actually do is we can either directly physically connect both of these virtual firewalls into more than one port so effectively each firewall on its one port will pick up a l Port from here and just skipping ahead here I have the old firewall we set up and you can see that this is 1.1 and you can see this is 1.100 but but if you actually look at my one IP addresses on these machines they've got a seven subnet now that's because instead of my ISP router I've got sofos XG and it sits on one of sofos xg's networks that's effectively simulating what I'm proposing here with a dedicated router so just to simplify this I recommend for this video to keep it simple if you wanting to do this and you're wanting to start out from scratch make sure you've got your ISP router make sure it's in rooting mode for each of your virtual machines and we'll have a look in proxmox in a moment you want to connect the one to this router here if you've got more than one port if you don't have more than one port you're going to have to do what I do here and create a VLAN on the switch and connect it up to the router but provided this still has DHCP both of these firewalls will get different one IP addresses and that's the critical part we need here out the back of this on the lamp ports you want to put those into the switch and you also want to create an ha link between the two firewalls so either a physical Ethernet from one port to another port or you could actually do this with a VLAN again over the switch and that's how I have this set up just because I'm currently running two firewalls both in ha it's pretty complicated I don't want to go down that in this video but it's been a lot of fun trying to get this to work and then off the back of that you want all your clients down here and so if you're thinking what does this look like well if you remember from my previous video or videos I set up the open sense test this is the one over here which has the dark interface and I've created a new one which is the light interface now this one is here and if we look in the hardware remember I said we want three Nicks now these don't have to be physical they can be virtual I recommend for starting out you you have three physical if possible just to get yourself comfortable with some of the concepts that we're going to cover so in this instance this one here is the onean this one here is the Lan and this one here is the ha you can actually see that I've tagged this with a funny VLAN tag um that's like I say just because I've got a bit of an interesting setup with running two firewalls in ha at the moment but if possible try and do this physically once you feel a bit more confident and More Adventurous you can go down this route and you'll also note now that I've got a new one which is open sense 2 and again this is pretty much the direct replica of the existing one so three Nicks here we have Wan Lan and ha and now if I go on to my switch you'll see how I have that connected so over on my switch I have this one here which is the one so I'm running this with a 7 do IP address for my w this is on the second one that I've just created and the first one that I created in the first video if you remember should be this one here yeah windev 7 again fan 7 so this is the one on the master firewall the one we created previously and this is the one that is on the new one that I've just created for this video and both of these will plug into the Upstream router which is my sofos XG but what I'm saying is plug that into your ISP router and it should pick to a DHCP address next I've got the ha link so here I've got open sense ha it just happens to be on a random VLAN and this is actually replicated over here on this one where it accepts tagged vlans on 654 now this is getting complicated and it's just because I'm running multiple vlans on the same virtual Nick in proxmox what I recommend you do if you can do is directly connect these machines or use three Nicks just to keep it simple out the back of that I've then got the respective land ports so here again the default on this the native is this one here and on the other one so the new machine is this one here open sense land 350 so what does this mean well this means that all three firewalls are now physically connected to the switch they both have a one IP address they both have a lan on the same network and they both have a ha direct connection to one another now none of that's going to work at the moment because we have to dive into the configuration so we're going to jump now into open sense we're going to need to have access to both of them and then we're going to start the configuration process now as I stressed earlier this machine here with the dark interface this is the one that we created in the previous video this one here with the light interface is the one I've newly created just for this video to get high availability up and running one thing I recommend you do is unfortunately strip back everything we've done in the past couple of videos especially if you've been doing things like the open sense and the wire good it recommends on the open sense website that you do this with a clean installation and then you create new interfaces once you've got ha configured it kind of makes sense because a lot of the stuff we were doing last time was breaking some of that fundamental routing pushing it over a VPN for example so it won't see the local network so recommend you take this back to basics but it probably can work with a bit of jiggery pokery open sense tends to be a bit more involved with networking than most firewalls okay so let's jump into the bit that you're probably here for and that is configuration so the first thing we need to do on both the firewalls and there's a common theme here most of what we're going to do applies to both firewalls albeit in the final steps there's just a couple of exceptions to that rule and that's simply because one is going to be the master and one's going to be the backup and by the end of this video I'll demonstrate that failover to you so by default let's go back to the original firewall we're going to call this one the master and your interfaces tab should look something like this so you want to go to interfaces and assignments and remember we've got three virtual Nicks so this one here we need to add this so I'm going to call this one PF sync that's just what the documentation recommends you can call it anything you want it's just because it runs PF sync and I'm going to add that now once you click add you can see that it's here and it's been mounted as opt one now before we do anything we're going to do exactly the same thing over in the new firewall so here we see vet 2 again and I'm going to call that one PF sync and I'm going to add that great we've now added both interfaces on both firewalls the next thing we want to do as always with open sense is we want to click on them and we want to enable the interface now this is the first bit where we have a slight difference but it's only a values difference so on the first one again this is the master we're going to go down to the ipv4 configuration type and we're going to change this to be a static ipv4 when you do that it gives you additional options down the bottom here so now this is the PF sync and I'm going to use the IP rrange of 10.08 so I'm going to do 10. 0.0.1 in this case and I did just say eight I'm actually going to change it to a 24 so as always this gives us 254 usable IP addresses now the key thing here is that we're going to put this one as 10.01 and we're going to hop over into the other one in a minute repeat this exact same process and we're going to call it 10.0.2 so let's save that one and then make sure we apply it so that should now be applied we're going to hop immediately into the second one we're going to click on PF sync we're going to enable the interface we're going to scroll down make it static and on this one it's 10.0.0 do2 and again crucially we need to change this to a sl24 space so we're going to hit save we're going to hit apply so now that we have both of those interfaces created enabled and set with a static IP we can move on to the next phase and just after that phase I want to go and test this connection before we go any further just to make sure that it's up and running and behaving as expected before we can test it because of open scent being default deny we need to add some rules onto that new interface so we're going to hop into firewall we're going to go down to rules and we're going to go to this new interface which is PF sync you might need to do a refresh here so let me just do that yeah and there we go you can see PF sync so we need to add a new rule here and this one's going to be really simple I'm just going to say that on the interface PF sync and any any rule you can restrict this to just use the carp protocol which is used for ha but because this is a dedicated connection between the two firewalls I'm just going to keep it as any any that should be okay so let's scroll down and just hit save then we want to apply and hopefully we've got that here yeah anything on that interface will be accepted so we need to do the same thing now over in the other one so firewall rules PF sync and then we're going to create a new rule on that interface PF sync and then it's any any and I'm going to click save hit apply and now it's a good idea to go and test that both these interfaces can be reached so over in proxmox you can connect to either of your firewalls Now log in through the console and then once you're logged in you want to choose option seven so in this case you can see that I'm 10.0.0 do2 so I'm going to test ping in one so I'm going to put in 10.0.0 do1 and hopefully yeah there we go we can see that we've got a response we had three hits everything's fine so now we know that we've got the two interfaces correctly configured and that they can send traffic to one another brilliant we can move on to the next step so the next step is where it gets a little bit more interesting and where carp comes in and if you've been watching any of my kubernetes videos you'll be more than familiar now with virtual IPS and that's exactly what we need to do here there'll be a floating virtual IPS that are shared between these two firewalls and that's good because clients can connect to the one IP address and it doesn't matter which one's responding that ha link will make sure that they fail over seamlessly and the right firewall is responding to the right requests and I'll show you a demonstration of that in real time later when we do some pinging and I'll shut down one of the firewalls I.E the master and it should seamlessly fail over to the backup so handily we're already in the right place interfaces and virtual IPS and you're going to have to probably pause a video a couple of times on this CU it's quite involved so we're going to hit settings and we're going to create two new rules now again the really important thing here is you replicate this process on both of the firewalls so we're going to click add and so we need to change not an IP Alias we want to change it to carp now we're going to create two rules one for the onean and one for the Lan so I'm going to go ahead and populate this now and then I'll give you a quick explanation of what's actually happening here so the finished setup for my configuration looks like this it's on the interface of Wan with the mode of carp the network address so this is the virtual IP that you're going to choose now this needs to be an IP address that's outside of DHCP and is not used by anything else on the network exactly the same as in kubernetes and it's in the SL2 24 because this is a sl24 network so I've chosen 7150 and now remember this isn't a public IP address this is because it's internal in my network this is being handed out by sofos XG most likely in your case this is what's going to be handed out by the router if we look on here your ISP router which is typically 1.0 or 1.1 so you'll have an IP address from there it might be a good idea to make a static reservation in here for both respective firewalls so you might have firewall 1 on 1.2 and firewall 2 on 1.2 so in this case it's 150 for me on the VLAN of 7even you'll need to change this to whatever your Upstream router network is next we've got the vhid group and so this is a special group that it puts these commands in and I've given it the description of the VIP one and just as a reminder you need to put a password password in here that's because this communication is secured with a password so I've just put in the default value of open sense but you can choose any password you want here just be sure to reflect it on the other machine so when we hit save you can see that this is now here and you'll want to click apply to make sure that this rule applies now as I mentioned we need another one so I'm going to click the plus again and I'm going to skip ahead and then we'll go through the same run through process so in this instance again it's CP this time the interface is Lan and then I've created basically the same IP just on a different subnet because remember the default Lan for open sense is in the one dot range so I've made sure again that 150 is not taken by any IP address on that Network and you want to make sure that that's true so put it outside of DHCP lease I think it actually makes sure that you do and you can't set a DHCP if you've got a VIP in in place I've just given it the same password for this link but again you can choose whatever you want just make sure it's replicated on the separate machine I've given this one A different vhid group it needs a specific group for each VIP we're using and I've just given it the description of the VIP Lan so now we can click save and we can click apply and you can see that now both of those rules are here now we need to do exactly the same thing over on the other firewall so I'm just going to replicate this on here and I'll show you what that looks like when I skip the video forward in a minute and so here you can see that I've replicated the exact same thing on this firewall as on this firewall here great we can now move on to the next step the next thing we need to do and again this is on both firewalls is to create some outbound natat rules so we click on firewall click on NP and we click on outbound now you're going to need to set this to hybrid and hit save and then when you've done that just over here there's a plus sign we need to click and then we need to create two new rules so the first rule that we need to create is the PF sync interface from The Source address landet and we want to translate that to the PF sync address once you've created that rule scroll down and hit save and then we need to create another rule you can see that it's here but it's not applied yet we'll do that once we've created the second address so click add and then this time we do need to make a onean so for this rule we want the onean source address Lan address and we want to translate that to the VIP one address so let's scroll down and hit save and then we need to apply this rule so those are now applied and we need to replicate these rules now over in the secondary one so exactly the same again and once you've saved and applied them you can see here that this rule is exactly the same as this one here great we can move on to the next step and you'll be pleased to know that the next step is the final step and this is where we do the ha configuration so if we go to the system and we go to High availability and we hit the settings we need to make some changes here so the first thing we need to do is to tick synchronized States so this turns on the ha function now now we know that the synchronization interface isn't the Lan we've created PF sync and we know that the pier IP address isn't this one here it's 10.0.0 do2 because this one is1 remember we're going to specify one on the other machine after this the next thing we want to do is the synchronization config to IP now this will be the same IP as here because it's the one that we want to synchronize so we can just paste that in here the username name and password this will be whatever you've set your secondary firewall to be so for me it's just the default of root and open sense and then it's going to ask you well what do you want to synchronize you can choose what you want the default is just a few of these so DHCP VIPs Etc I'm just going to click the whole thing so I want everything to be synced so I've ticked that and then I'm going to scroll to the bottom and I'm going to hit save so that's now saved but it's not quite working you guessed it we need to head onto the other one and do a similar process but not the same so system we want to go to High availability and settings and we want to enable this one and we want to change the peer IP address to be 10.0.0 one now because this is the backup we don't want to synchronize this config to the other machine we just want to leave everything default so again we don't want to take any of this stuff here we just you just want to scroll down and hit save and now hopefully this should be up and running so let's go and validate so back on the previous machine let's hit status under high availability and if this has worked yes you should see all of this here you can see both firewalls and both statuses and so what I do recommend you do now is head to your dashboard and you can actually click on widget and in here you can select carp and when you do you'll see down here that it adds carp now I've just dragged that over but as you can see this machine is set up as the master and it tells us that here and if we do exactly the same thing on the separate machine it should show us the backup and yeah over on the secondary machine you can see that this is the backup so excellent let's do a test just to make sure that this is working so if I run these two commands this is just going to Ping inin ly Cloud flare in this instance and on this machine here it's just going to be pinging the VIP of the firewall so as you can see I'm rooted through open sense at the moment in ha mode and both of these are working so I'm going to jump now into proxmox and I'm going to turn off my master firewall here which is this one so if I now shut this one down you can see that it's starting to turn off hopefully we go back to these to here and we're not going to see any disruption ah there we go we saw a request timed out but it picked up straight away so there you can see ha in action it really is that quick so now the traffic is being rooted through the other firewall and you can see this one in the background isn't even available again we got another timeout probably the final thing but all those pings are still going through and remember this one here is the internet this one here is on the local network so we failed over from the master to the auxiliary so I can now start this up and if we check now on our dashboard if I give this a refresh you can see now that this is the master and this one here if we do a refresh it's not even available CU it's not turned on so let's see what happens now when I start up this new machine so this is really handy for example I don't know you've just had to bring this prox MOX node down to do some maintenance upgrade some Hardware do some patching and you want to automatically start this back up so this is now booting in the background and hopefully as this comes online again we're going to get all of those pings still going through so now that machine's coming back up there's a slight request timeout and that's probably now because it's failing back over to the master the master is taking control of my connections so as soon as this is finished we're going to go back into open sense and just check the the original one now so yeah this is all up and running so hopefully we can refresh this one here and we'll be asked to log in again and now we're logged back in and we can see that we're on the do we're on the master and this is now the master if we go back to this dashboard here and we hit refresh we can now see that this has become the backup and if we look at the pings all during that time we still had access to the internet and local networking excellent so thanks for watching and hopefully if you followed that guide you now have highly available open sense this was a ton of fun to get up and running I say fun in inverted commas um getting this working on my setup behind an already existing ha setup with sofos XG was a challenge but you won't have those issues and hopefully if you follow those step-by-step guides you've basically just got to replicate each step on each firewall and test as you're going along by checking in the console with pings Etc but hopefully follow those steps and you'll have all the benefits that I demonstrated with ha so now you can shut down one of your proxmox nodes or you could have a virtual machine failure or Hardware failure on one of those nodes and fingers crossed all that traffic should fail over to the other firewall and all of your routing and all of your networking will stay available let me know how you get on in the comments below and as always give this a thumbs up hit that subscribe button and I'll see you on the next one take care [Music] everybody [Music]

Channel: Jim's Garage

Views: 11,850

Keywords: opnsense, how to configure opnsense, opnsense setup, pfsense vs opnsense, opnsense install, opnsense wireguard, opnsense proxmox, proxmox, opnsense vlan, opnsense firewall rules, opnsense port forwarding, homelab, vlan configuration step by step, vlan explained, inter vlan routing, vlan configuration, wireguard vpn, dhcp, ddns, firewall configuration, firewall in network security, unifi, ubiquiti, ubiquiti switch vlan, vlan switch, high availability, opnsense high availability

Id: I5n3QXOlxmw

Length: 28min 29sec (1709 seconds)

Published: Wed Dec 13 2023