Cisco SD-WAN Design Series: Firewall/NAT

carpe diem vpn seize the network hey there it's uh tim with carpe dmvpn back for another sd-wan uh design video and i'm joined by a good friend of mine david who works for verizon as a principal engineer and um we've been doing this for a long time haven't we it's been a while yeah yeah we were both uh pretty much since the since viptela became a household name if you will we've both been slinging packets with the viptela stuff so um everybody knows me but uh and i'm sure pretty sure most people already know you but if you would do me the uh the honor of uh introducing yourself well maybe some people don't recognize me because i comb my hair today but but well yeah i'm i'm a very active member of the community at least for saying things in well in twitter so you would hear me or read me talking about well pretty much sd1 or routing or design or just recruiting some morbid joke because that's also another thing i like to do but well it depends what which are you're interested in you know when you get into twitter and uh well other than this yes i work as a consulting engineer so in general i just get to know a customer talk to them then discuss exactly what they need what what's it that they want to achieve with sd1 and then talk them through it okay so help them to get there i've been in situations in which the customer just bought sd one they have absolutely no clue about it because for them it was just replacing one box for another so they many customers still have this thinking about let's make the new thing work the old way and that's actually horrible because it is it is well it gets complicated because then you you get to this world cognitive dissonance in which do you believe something should be done this way and then the customer asks but why and then is this a ping pong about no dude don't do that well part of being a consulting engineer is being a good listener sitting with them and then we'll convey the right information stalk them through it and we'll show them that you can get better you can get to a better place and achieve some well business outcomes as if you understand where you are and where you want to be also it involves well you can do this in another way you can in you can integrate these two with another like sd1 umbrella like sd1 using advanced security features like sd1 and maybe something with the api because you want to have a different dashboard showing you different things but some people don't know this some even some engineers are no things that you can do by integrating tools and but only knowing the fundamentals so this is well this is my job now so just tell people how they can do it better and well in the process of it actually learn much more because sometimes customers come with some random question and then i have to investigate this amount of papers so i so i can justify my no but it's part of the the position i love it honestly i love to talk to people i i love the the human connection that you can put into technology when you talk to them and it's just about hey look at these comments and look at what they do but rather what do you want to do let's talk through it and putting this this humanity where we need to have it because it's not just about ones and zeros is it it's how do we interact and how do we get to a common agreement and things like that and well this is the way anyway that i met you or i met some other people this is true a little this humanity into technology that's what we do yeah no i couldn't could have said it better um so yeah i mean you hit the nail on the head and honestly that's why i started doing the this design thing is because i wanted to get out of the ones and zeros and start moving towards well why do we do it at all like why do we use this solution what does the solution do for me and then of course what's the best way to use the solution so or is there or is what is the quote-unquote best practice which of course we know best practice varies wildly on the network and the requirements and all of this right so um but anyway best practices are they they have in general best practices are these 80 20 rules that they will catch 80 of the cases that's right but the other 20 which is unsurprisingly the most of the cases that you would see [Laughter] that's so true you're like this is a corner case and suddenly wait but that was a courtney case on the other customer and in the other customer uh damn it yeah what's the number at which this stops being in a quarter case like do you tick the box enough times oh no absolutely just stop calling them corny cases because it makes no sense yeah yeah 100 um so we are here today uh to talk about firewalls to talk about nat and firewall rules and just generally how firewalls and security appliances um get both both enable and some cases maybe disable or or make departure the business of putting together an sd-win fabric um and as a as a rubric if you will i have my lab that i've been building um that i talked with chris last time about our control connections we touched briefly on the firewalls specifically we were talking about control connections in that case we were talking about how we can get mpls control connections or whatnot off the mpls over the internet through a firewall and and we had a couple other things but i wanted to have a whole session where we just talk about firewall the nat rules and just the considerations that we have to be aware of um for it and so i'm so glad that you could join me for this um because it's gonna get i'm sure we're gonna get in the weeds a little bit i'm gonna try to keep us somewhat focused on design but some of the stuff you just got to dig in and you know with the very specific use cases where the firewalls can get in the way so uh without further ado um let's go ahead and i'm gonna and you and i let's just go over to the diagram and just start talking through it let's go so let's go ahead and take a packet walk and just and just kind of remind ourselves of what the problem was last time so last time again the mpls router could not reach the s2n controllers so we added our mpls underlay router to provide that connectivity and what that means is that when branch one sends a packet over the mpls to the controllers that packet will go over the mpls and i'm originating a default route into the mpls from both underlay routers in the data centers so it'll pick one of the two but let's just say in this case it's going to go this way so the packet will go to the mpls underlay router which will route it to our firewall and then our firewall will perform the nat the network address translation that will change it from a private mpls address to a specific overload nat address and then that will then go over whichever transport it wants and off to the controllers so that's the change we made from the last video is the ability for the mpls to hit the controllers and by the way that's going to happen at the other data center because this was the no net but yes oh that's a good actually actually no no that's the no no you actually pointed out something very important right so so no you're absolutely right so actually what happened here uh i apologize i i forgot to talk about this this is a good good very good catch very good catch so what actually happens here i i said that that was here it's not it's here so that's in this data center in this data center i put the nat here so that's a very good catch um i put that out here because of that reason because we don't have a nat in the in there so that's a no nat dmz now over here is where i did it where the control packet will go through the mpls rep retain the private address and then hit this firewall and be netted so thanks for keeping me honest on that that's a that's a very good point no it was funny when they say that oh it's gonna be inevitable they say no not the guy's violating all the things but no you're absolutely right um i so full disclosure um after we talked to chris after i talked to chris it's been now two weeks i think right uh i did all this work and then i promptly forgot exactly how i did it so very good catch being there all right so um having said that uh having said that that's just one of the considerations that we have to to think about when we're talking about what are bringing firewalls into the into the uh picture so what we haven't really talked about and what i'd love to talk about with david today is among other things is this idea of each data center has two sd-wn routers and two sd routers are connected to the nat or known at dmz so the firewall and then the firewall is connected to multiple internet transports let me clean up this diagram a little bit so we can talk about it um give me a second to pick the eraser here let me erase that a little bit so so we can mark it up better and i'll let [Laughter] yeah so so now now there's a there's a bit of a lie here on the diagram the diagram makes it look like there's actually two interfaces going to the firewall this is not the case this is actually faces right this is actually sub-interfaces so what we have here is each s28 router has one physical interface going to the to the firewall and it's a sub interface i think actually how we've done it is like biz internet is vlan 11 and pub internet is vln12 or or 1q encapsulated and then when we hit the firewall uh the firewall is basically just routing the packets for us and so i would like to talk about a little bit now that's how i've that's how i've designed it i'd like to i'd like to ask uh david given given this setup just that we have two sd-wan routers firewall no nat dmz firewall connected to both transports you know like what what advice what would your suggestion be about how how you might do a design like this from a underlay perspective the first thing i would add is what i would ask i was going to ask is do you have a t-log extension between those routers is it true i do not oh okay and there's a reason for that right it's because each of these routers is connected to both all three transports yeah yeah so yeah it depends on how you want to deploy but that's correct you could have the extinction easier not connected to both transport but if all of them are connected to all the transfers then the table extension is doing nothing there so some people put a switch in front of them but i would say that just that just that's a single point failure to the design and i wouldn't go an hour out honestly so it really depends if you want to take the risk or not but again that was for tila extinction so i just went around the whole topic for the firewalls though let me just try to zoom in a little so uh i mean i mean oh you may have to uh i might have to zoom in for us since i'm the one presenting this hold on we can zoom in a little bit and remember that of course those are two not too physical it's more like a logical right the physical is that black line and then the logical no that's okay anyway if it's logical it is correct because you would have to in the connection to the firewall respectively being a t-log x i'm sorry a sub-interface or not it's okay okay what i generally see in in this type of design or where or the common design fc is that the firewall would be connected to some sort of internet switch usually so this is facing the the edge and the sa1 router is going to be connected to that switch so it's going to be the residue one router then the internet switch and then the firewall no that's a good point so no that's actually a very good point i i want to for those who might not be able to visualize it really well let me let me kind of and you could draw those two actually you've got certainly access to it but but let me make sure i'm understanding because i agree with you this is this is the more common actual implementation i'm doing this because i'm in a lab it's not actually that common to connect your firewall directly to your transport right because it's normal that your transport actually is going to end up in some kind of aggregation a wind aggregation switch and then you're going to use vlans or or something to break that out to send that to the firewall on the outside and then and then bring it back in on the inside or set up a dmz vlan where which is what we're talking about here right yeah and i'm going to tell i'm going to stop annotating i'm going to let you annotate a little bit that's yeah that's the common thing in general some people even do in this interconnections between the sd1 boxes and the firewalls so here i just don't want to make a mess of it you can erase what i've drawn by the way and annotate on top so people in general what they do is that they use some slash 31 range and then they just they just simulate that they are just connecting towards the switch ignoring totally that there's a firewall behind or or just ignoring that the switch is there and they just try to connect to the firewall instead so the switch is just doing the well but the switch is switching but they want to ignore it and that's okay it's up to whatever you're gonna do on it the trick in this is how do you then perform the not because what becomes complicated when you have nothing scenarios is the ranges that you're using for nodding if it's a slash 30 or flash 29 or whatever you get assigned doesn't matter and to which provider does that belong to if it's a provider independent or if they say provided a sort of provider assign range because if that's the case then do you want to understand that sd1 has was born from the premise that the space that you would use in general will be yours and you can fumble around you can play with it but if it's just depending on the provider then it will play some restrictions although in general those restrictions are going to be healthy let me just put it another way if you're using a provided uh provider sorry i'm bringing parting here provider independent range yeah then you can you can get into shady scenarios where things don't work and you don't know why because you would have most probably would just advertise this and branch in both of the clouds and that's bananas because it's funny that you bring this up because literally when chris and i were talking last time about the the control connections this is exactly the scenario we were running or that we started talking about was the exact same thing you just observed right if i'm using provider independent space and i'm announcing it to multiple providers i can't be deterministic about my traffic flow for the colors right so you know that's that's that's great that's a great observation it becomes complicated and then well the other thing is if you have a provider design range then then you have also to understand how do you make that work with mpls who is the provider giving you or allowing you to use the mpls service and also if you can then use a public range an mpls and then put it or advertise it into these internet but the default free zone if they allow you just to put that and then you know okay um i see you gesturing a lot which is which is great um let's help everybody else out erase erase yeah yeah let's let's because i think i think it's important i think what you're saying is extremely important and i want to make sure we for the people who can't visualize it themselves like that we can kind of show them a little bit yeah my apologies let me just try to find the eraser oh oh i got chang on i got i got the uh i got the tablet and the pen and all of that so it's a little easier easy for me right so okay there okay okay so let me just move around here so yeah i think i'm sorry i'm gonna go back in so if you have a provider uh let me see how soon there are okay so maybe this was gonna work if you have then a provider independent range it means that it could be advertised this way or this way so it means that your traffic is going to come inbound this direction or this direction because it doesn't belong to any provider so as it depends on you advertising it through one another so it's gonna depend of what you have here in the in the firewall and just yeah are we are we doing something like uh like primaries back in uh secondary where we're advertising specific uh something specific to well you could but it depends on the range because in general you cannot do anything smaller than the slash 24. right exactly so irrespectively of view advertising is slash 30 29 27 if the provider is gonna activate it into slash 24 then well you're doomed there's no difference no you're absolutely right so if we are um if we only have a slash 24 and it's provider independent then there's no we can't do uh traffic engineering in a way where we send like say more specific advertisement to one provider yeah absolutely true absolutely true so we'd have to do something else and and i've seen and i've done this in the wild and when i was an enterprise engineer and it's maybe like 60 40 like works some of the time is you could do something where you asp pen to one provider over another but like the truth is that there's so many cops out there on the internet that some people are still going to see that as the shortest path it's a little ugly right um well yeah it is an undeniable consequence of the internet being an and an uninstructor where i would say and not a non-hierarchical network right right so if the internet has no hierarchy then it will it is unlikely that some things would work so we will have to stick with the well the principle of either slash 24 or if you have something bigger or smaller we can play around that but if not then well you have to find some other ways yeah agreed it's and it makes it really complicated um using provider independent space makes it really complicated for your underlay is what i mean for your own really it makes it complicated uh in multiple ways right like now for example if i send something out of my biz internet color so so my sd-wan router thinks it's gonna you know sends uh some provide if i'm using provider independent space for me from sdn router and i send a packet out and i'm saying it's color biz internet and even if the firewall even if it even if the firewall forwards it to the biz internet color like that provider whatever att or verizon or whoever it is we have no expectation or understanding that it can come back the same way yes the path is complicated and also that could break some applications that could because the path is not the same one you might have a difference in quality of experience as well so it becomes tricky trying to regulate the internet which is the wild wild west so so well that's one of the downsides or having provider independent ranges in general you would want to be conscious of this now let me just can you help me with the eraser i just oh i don't seem to find it so uh now we should be then jumping into provider uh assign ranges yeah so let's talk about let's talk yeah so let's talk about provider assigned which is probably a lot cleaner for this purpose for this sd-wan yes yeah well now talking about provider assigned ranges it becomes easier and at least for our case it's it's a better way or at least a clear way to do it why well because in sd1 if you have several transports specifically internet which is going to be the i think the most complex one to try to influence then you want to find a way to make each one of those transports independent so what what i try to convey with this is that if you can have then a provided assigned range that would then tie whatever the traffic you're going to use for those or sources or or simply the traffic is going to be related to those ap addresses to these only transport then that's going to help you also tend to to segregate base right so if you get it coming back the same way right yeah so if you have a particular range that you advertise these ways out of the firewall for each one of those providers then there's no way that if i come out this way i am i am going to use pop internet i would rather just come these way most probably get to well it depends on how you have your policy if it goes to the firewall and then back or if you just do something like this but you would not then use the other transfer in the wrong way or at least you wouldn't simply mix them and that would make it easier to handle though now on top of this then the other thing to consider when you have well this type of senators in which the nothing is being performed then who is doing the knot is the vh doing the nod is the mpls router doing that is the firewall doing that so where's your visibility over those three cases and and why did you want to have visibility over those so assuming you have a an mpls router and this is a common deployment it's just not that cheaper not as cheap right yeah that's that's the thing is that a lot of people don't want to buy new routers or they don't want to so i get that right but it definitely makes it a lot of a simpler a lot more of a simple deployment it simplifies the deployment significantly right yes we talked a little bit yeah if you do on-prem you can do it you can just have a massive server that anyway is going to be running your controllers or something and then you would just put a csrv there doing all the magic and that's the that's the case i have seen most mostly honestly when it's on-prem of course so what do you do you have this mpls router and the mpls router is going to have three routing tables so it's gonna have the global routing table the writing table that you might use for management and then the routing table that you would use for the internet connection actually fourth because then you would have the one for mpls so are we talking are we talking about so you're talking about like when we oh okay so we're talking about if you were to do an on-prem yes controller installation yeah okay actually if you would do not on-prem controller bro if you would then use this router to not again okay and use a bunch of well if you have a massive uc server doing all this then you can use you put a csrv and then put some jigs of your of all of your i would say processing power and then just run a virtual device you don't need to put a new box for this you would just use the cards you have now when talking about this mpls router here then what is this router going to have i'm just way over in my apologies so yeah several routing tables and you have the mpls routing table um i apologize because i'm just using the mouse you have the internet routing table you have the global routing table and you have a running interview that you must probably would use for management now what the writer is gonna do is that it's gonna in general what i see is this term that for some people still causes shivering is vassi interface seems like nobody likes it or not yes it's it's not a mistake i never understand it on a stick type of discussion but it's hilarious so what you would have then is these most probably you have an ap address that comes out of one routing table lands into bossy and then bossy will do the not on a stick translation and then it gets advertised to the internet and then the rf yes and then in this internet vrf would be the one connected to the not to the firewall here so so it would depend on how your deployment is but in general this is a common deployment on prem you would have in a csr doing all the magic with a bunch of routing tables and then not in a stick if you don't have the nut in the firewall because then that is going to happen on the mpls underlay router this guy here if it is if it happens in the in the firewall then we're talking about this guy yep which we're gonna we're gonna go over there and we're gonna definitely dig in on that as well for sure so if it happens in the in the firewall then in the right then well first you have some pros and cons one of the pros is that you would have the visibility over the firewall the con is that well it depends depending on the firewall and whatever the deployment you have then how many of these interfaces can you test into these how many sub-interfaces are you going to use can you use sub-interfaces or not can use physical interfaces and also then in which way you're going to advertise this out then if it's uh if what you have with your provider there's some default route maybe if you have bgp if you have any other way in which you would advertise and and get this connectivity towards your provider cloud and you would need them to understand or to know to which of them you would advertise they or use a particular range that belongs to their network and also another of the considerations and i think it's the the last one that we would have to touch on is not necessarily internet but how mpls interacts with each one of these transports because ideally in general it doesn't it doesn't but it depends on how your controllers are deployed and that's we are getting back into deployment and controllers but in general then how do you deploy your controllers if you cannot deploy the controllers in the cleanest way which is all of them get a public ap address and then you would advertise those 40p addresses that you would use into in the internet and then the question is in the into the internet cloud of which provider do you want to use one on top of another do you want to use both so these are the things you would want to consider that's one thing another is maybe you don't want to put a public ap address into all the controllers you want to put it only in vbond which is actually the only one that is mandatory to have is the only one that is required all the others don't need a public ap address you can just put anything but then we are gonna you gotta have a nat so you have to have a nap for them somewhere though right because somebody on the internet colors has to be able to reach it exactly so the thing is that each time we involve that we get into into the discussion of which flavor of knot you're using which ip addresses and ports you're using and if it's going to work or not because simply some flavors or not don't worry simply when you mix them because there are several variations is it about eight of them maybe you have full connect restricted cone then you have i think is a dynamic nat and it had another fancy name that is related with a cone as well i apologize i don't remember the names but it's a cone so this is a wonderful one or whatever this is a this is a good uh this is actually good because most people that uh come into viptela actually are familiar with the cisco specific nat terminology and not the viptela i want to say viptela but i think it's not even necessarily viptela specific i want to say it's more like non-cisco or or more open industry uh the the restricted no you know symmetric nat or symmetric cone and asymmetric cone and full cone and and all of these are terms that most of the people you know watching probably don't aren't familiar with um so let me show you how i have it how i set it up because i wanted to get them on board i wanted to on board my my routers and and then you could just tell me like how how badly i did basically uh or or you know or how you would change it or how you would change it so how i did it here i already i already kind of oh here let me change this back to pen here so i don't erase everything um so how i did this was of course in this case i used sub interface and i um gave an ip address on each color to the you know to i think i actually how i did this is this uh just for so we can actually have something to talk about like so i think i did 119 1910 dot zero slash 24. i did dot 11 and 12. that's how i how i numbered it if you will right so here you know for this color i gave this uh you know like 12.x and i gave this 12.x right so so public ips on that color right so when it goes to the firewall the firewall is just routing it's just routing and it's doing um access control which we'll talk we should hit on as well the access control pieces and what needs to be allowed and the pain of you know infosec teams that don't allow certain types of traffic right um so so and then i did like you know.11.x here so so so that's how i did in the no nat dmz and the the firewall routing table is actually very simple right so it's literally uh just saying you know iprout and then for these subnets for the which i think i actually i think how i did this was i packaged them so for each transport i used a slash i think i used a slash 28 or maybe it's a 27 i have to go look um but the point is so it would be contained contiguous so like 12.x 12.x these would be contiguous subnets so that i could package them and then advertise them to the provider that way this so this is the same way provider assigned would work as well so provider assigned they would assign you a block of ips and then the provider wouldn't actually need you to advertise their own space to them but this being a lab i that's how i did it with bgp or no i didn't i did a static route here because of that because the provider that assigns you space is not going to need to have you announce their own routes back to them via bgp right yeah they're gonna be aggregated later anyway right they're gonna aggregate it in their network and they're gonna say the next hop is your firewall or your or whatever it is right so that's so that's how the ipsec and control connections are built in the known at dmz the firewall is literally just iprout xxx sending you know biz uh pub traffic that way and biz internet traffic that way now there is a interesting conundrum if you will which is where is the what is the default route point to on your firewall if you have multiple internet providers and and if you have multiple colors coming into that firewall how are we or can we even determine to which color we're going to build or ipsec if we're using a default route pointing at both providers right do you see the concern i understand that's a tricky one because i think that i would uh i think it would be then a tough call from inbound or sorry for an insight so from the firewall i think it's going to be easier to see it turn outside so it would depend on which one is the source i don't know if you would then have to well put some particular policy in which you would send traffic through this link if this conditions because if it's outbound to towards the internet usually you don't care usually except when you're trying to build ipsec tunnels over a particular color right so when you want to preserve the color and the reason of course to preserve the color is so that we can get our telemetry and get our bfd probes and our quality measurement across that color right to see how how how nice is pub internet having a day or how nice is the day for pub internet today and if it's half of the packets are going over biz internet you're not getting a good picture of the color right so no that's exactly the that's exactly the concern is is if you just load balance via zero routes you know it's it's it's a little harder right you almost have to be much more granular in your routing table uh to do that so that was that was something um the other thing was uh let me think so that's how i did it in the no not dmz it's actually very it's very simple and then at the end z to do it this way there's another thing i went and also to bring up okay uh uh because i mentioned that it would be also depending on how do you see it from outside so right in which way then are you attracting the traffic towards the controls are you using a dns name that is mapped to either of the ap addresses that you're advertising out in the internet and one transfer the controllers yes oh okay so the controllers actually yeah it is the controllers are actually in the cloud right in this lab in this lab at least well i didn't mean i didn't mean in this particular oh okay what i meant is in general how do you attract the traffic towards the controllers it is very common to use a dna's name and then there's the the dns is going to resolve to either of the ap addresses that they would use if you have one transfer or another but then yep then it's going to be tricky for you to simply determine which one is going to be the destination if it's a dna because it can be either yeah it can be load balanced between both of your public or however many public ips that you have agreed the good news is the good news there is at least for control traffic you don't tend to care that much like what you know as long as you can reach your controllers and it's a color that you expect to be able to reach the controllers it's the ipsex setup it's the ipsec setup that really matters here right so so bringing it back um just to i want everybody to kind of understand the the challenge and then maybe like some of the solutions that we have the challenge of course is what we've talked about from the beginning when we started talking about provider independent space being problematic um just making sure that we can pin our ipsec source and destination tunnels to the same color at least as much as possible through the internet right as much as possible to the internet you might not ride you know if you're if you're connected to cox business or something you might ride cox business all the way through to the other side whereas if you're connected to some third party you know small town isp you're guaranteed that they're not going to reach all the other way to the other side you're passing through four other internet providers to get there you know you're just doing your best to to measure the quality of experience or quality of the applications through that you know those two endpoints right so just wanted to kind of tie it together a little bit about what is the actual problem we're trying to solve here and that's that's the problem and then when talking about data playing then for ipsec then we get into the the the topic of what exactly are you allowing me because for the solution to work then you would need you need quite a few number of ports and protocols to be allowed not just dns but you will need tls dtls net conf stun then ipsec via the evidently icmp so it's quite a lot to consider especially because the solution uses a different set of ports for the port hopping it uses one two three forty six and then it does a poor hopping of a with an offset of twenty so then it will be one two three six is six one two three eighty six i think let's talk about that for a second because that not everybody might not know like not everybody who's watching me understand the port hopping thing so we port up when we can't make the connection otherwise right like that's the that's the it's an automatic feature correct and and and if we are unable to set up a tunnel the the the c edge v edge whatever the the wan edge assumes that that port might be blocked and so it automatically jumps itself up by 24th to try a different set of ports right is that yes in general you could see then the let me just try to write it down here bro if you have a vh here i'm just writing a hockey puck here sorry for the quality but i'm doing i'm using a mouse so when they try to establish this data plane tunnel so this is a tunnel or i'm trying to do it but i tried so then you would start with one i think is one two three forty six it is it is one two three in general stays there that's the common ported you would see now if it doesn't work for whatever the reason then we start with offset the offset is 20 by default here so if it doesn't work the first time we will try it again but it's going to be 66 then if not it's going to be 86 if not it's going to be c to 6 here and this is going to be a four if i'm not missing no sorry this yeah and yeah if not then we're gonna back into the first one so we are gonna offset four times till we are back into the first board we used correct it's gonna be a cycle and because of that because that's possible um what we need to be aware of for the firewalls i think is to either allow this or simply make sure that we're allowing some subset of the ports that because if we're if we're allowing one two three four six to go out on the firewall we shouldn't ever actually need to hop right because that that traffic should be allowed outbound on the firewall and we shouldn't run into a case where we can't set up the data plane unless um unless something else is broke somewhere else down the line right like yes but also it would depend who is doing the knot because if the nut is happening in the firewall here then you might have a problem with the number of ports being used oh sure sure sure absolutely we should and we should get to that you got a very good point there um i was just thinking of the firewall rules but you're absolutely right we need to talk about how does that fit into what ports we're allowing exactly especially uh because we could start stepping on each other we start napping and adding to the same ports right you would just yeah you would use uh i think is was it poor 500 for happy sec my apologies for that it's 500 4500 for nat traversal oh so that's good i still have memory yeah yeah so you need to get uh get the reports uh also you need to uh net conf because of the policies that you'll be pushing through assuming that the controls are not in on-prem they are on the cloud well that's not in the that's not in the clear though right we do that over the we do that over the controller tunnel you still yeah i would be then over uh let me see dtls or whatever it is yeah yeah so you allow just the the product building yes but there are deployments that do that i'm sure that that have private completely private transports where they can do that right yeah it is it is well it becomes tricky then to to tying all the required ports that you use simply because the solution although it is automated and it has a lot of well dust uh fancy things on top you know the sprinkles and all that it's it is a group of many protocols working together and well one being a trigger of another so then it means that you will have to consider ipsec bfd icmp and if it's bfd then it's gonna be if they are allowing in general should be a fully vfd section but some people would argue that it might be just echo blah blah it is complicated then to to determine properly it is you would have to sit with your influence like people and then well get to a middle ground yeah yeah exactly so at the very least right at the very least we need to allow our dtls port one two three four six that's just so we can generally get out now when we start natting we're gonna have to actually do more than that we have to set up port offset we should talk about that in a minute but certainly to get out you at least need one two three four six because that's what it's gonna try off the or or if you're gonna change this that's fine like some infrastruct teams are like well that's a well-known port we'd like to do something different right and so okay whatever like so let's set our port offset or something but but generally that's it right that's to set up uh both of our both our dtls and our ipsec are using one two three four six uh if it can right and then and then uh or it's trying to use it and then uh but you're right like tunneled within or like all these different protocols um and then you know something think about that so that's both for control that's for ipsec you need ibsec we need if you're doing dns lookup like you said if there's some reason you need dns requests to be able to come in you know to resolve your controller name or something obviously you need to allow dns unless you're hosting it outside the company or in the cloud or something right so there's a lot of a lot to consider there then http or https depending on yes should be https though then see well that would be then four for three but i'm writing just all these random ports here but yeah we're not gonna we're not gonna i actually have a doc i need there no there's a great doc that cisco published specifically talking about firewall traversal lists all the ports that's everything you have to take into account i will make sure to link it in the in the comments section of the video um but but before i forget about it i do want to talk about the nat thing because uh i think you hit on an extremely important point and something i've actually been struggling with myself as i've been testing things like cloud onramp and whatnot and and you know infosec uh you know is a so so okay let's let's let's think about that for a second so when you when we're natting so let's just let's step back a second so when we're adding we're going to say like in this case let's say it's a port overload let's say we're doing that we're doing pat and we've got two and we're padding and here it's actually a lot easier to control the underlay being built or the controller which transport we're using because you can set up the net to say you know you come in on here that's right and you know you not to this specific interface right so like you're going to come out as you're definitely going to come out as this color because i'm going to knock you to this specific interface right so that actually becomes a little easier we talked about the how do we make sure we're going out and coming out in the same way like this is this is one of the ways to do it is to we can pin the nat and say you're going out this interface if you come in this you know you come in this interface you're going out this this interface anyway but the problem is the first router that tries to build a control connection data plane whatever say let's say this is like router 1 or router 2. router 1 starts first it's going to come out it's going to see it's v bond address or dns or whatever and it's going to try to make a connection one two three four six and let's say we allow it let's say that firewall rule is allowed it's gonna happen right it's gonna let it go out and go to the student controllers but what's going to happen when router 2 boots up and tries to make a connection using a source port of one two three four six you have to change the controller so so right so what what will happen for those who aren't like super familiar with nat is that gen especially with pat with port overload or not translate uh matt overload is that it's gonna the it's gonna multiplex this the and use more or less eat this port and say this port is in use now nobody else can use it so if another port if another interface comes in trying to use the exact same nat and trying to use the exact same port on that interface it's going to say no because i've already reserved that port for the traffic that this guy sent out so what will happen is this guy will just never be able to reach the controllers until he tries to do into like we got to get past this thing where we call routers guys or or or whatever i i keep catching myself doing it and i see it other people do it and it's just it's been such a thing in the industry for so long to refer to like you know devices as mail or something i don't know i i'm trying i'm trying to be better about it you know but anyway so let's say this router try you know it fails and so what you're saying is true automatically after about what three minutes or something it tries the port offset where it will or the port hop the port hop is actually different than the port offset so it'll try 66 and it's assuming that nobody's used 66 that will succeed agreed yeah but we can actually get ahead of this right we could actually tell this guy in his config you want to use one two three four seven we can turn on port offset in the config and he'll never even it'll never even try to use because we know we know that our setup here right and and tell me if you think i'm wrong or if you want to add into that but you can actually say you number two in the template we'll always use port offset of one so it will always start with seven and so they will never step on each other and that should work that's a that's a thing we could do right it should but again as the upset is going to be a limited thing of five different occasions because it's made for five pops then then uh it would depend on how many routers you have how many routers are trying to access using that board and you may get to this race condition in which it works now for me but it doesn't work for you and as soon as i drop it it works for you but not for me yes yes absolutely absolutely true you have to play with this offset in in a way that well this guy's going to be one the other guy is going to be two or five or as long as you keep the opposite in separate spaces you should be able to handle it we can only offset up to 19 because of the reason of the port hopping every 20 right so if i'm going to set up upward offset i actually can only use i can only do i'm not i'm trying to think of some situation where i would ever have 19 routers or something 20 rounders that need to use the same hat right that's the point it's gonna be very random honestly so yeah so we we are just here while trying to yeah talk philosophy about this you might not get into this situation unless you're trying to emulate it in a lava but i i have gotten into this situation believe it or not i've actually i've actually hit this once and it was and it was it was actually during when i was creating a d-cloud demo actually uh d-cloud cisco yeah yeah the demos that we but the problem was and i just wanna i meant the situation like what the hell is that oh yeah yeah most enterprises will never run into this right no most enterprises will never do this but it is possible to run into it and uh in my case the problem was i had one public nat to work one public ip to work with and like four routers and so i had to and this this was the problem is that the infosec team was not did not open up the extra ports so we only had one two three four six to work with so i couldn't make it work but you know so i could set up the port offset and what happened is it would get as far as reaching the controllers which again i had hosted internally but when it came time to set up uh connections to the cloud con to the cloud to the csrs i was using cloud onramp for iis and uh it could make the connection because the v bond was like okay well i told them this is where you need connect but of course it didn't work because the the port was not allowed right the one two three four seven eight whatever was not allowed so i actually did manage to hit this once uh but you're right most 90 at least of enterprises would not run into this condition i think i wouldn't say this is even a corner case i mean but if you're going to show it it is you could if you ran into a situation where you had multiple routers trying to use the same nat you would have to take some precaution to make sure that they're using different control connections or different uh ports for the connection and then you would just open those ports in the firewall that's yeah that's the that's the caveat right you have to open those ports on the firewall but you could also decide which one of the colors from the sd1 router perspective you wanted to prefer establishing a control connection over because i'm talking about data plane now so i'm just thinking a data plane but you're right you're right you're absolutely you're absolutely correct right the control connections you're absolutely right we could just say you know what this is too much work let's just use always use this color or something right and and then you would just have to you would still have to do something though because you would step on the same port you would still have to do some kind of offset but yeah for data plane especially this would be a this was a problem i can say for sure that it is a problem because that's what i ran into and and so if you had multiple routers one public net you would have to do the port offset or or something you would want to wait for port hopping right it works every 20 every three minutes you would jump another 20 ports but who wants to sit and wait 10 minutes for their as soon as they went to work right nobody does you do clear to clear and then just you know you just you feel like your face and then oh exactly where there's a request command that you can say request port hop and it'll do it automatically or whatever but or not automatically it'll do it manually rather um so yeah no absolutely you start smacking the box till it works oh look at it it's working yeah look see i told you it would work we just had to wait a little longer so it will depend on the situation this is a rare instance of it but it is it is rare it is good yeah it could happen it just but just haven't thought about it because well i think it in general you don't go down these routes you try to well you you yeah it's it's a it's kind of an el cheapo route really if you think about it right we're we're trying to reuse the same public ips we're trying to use the same internet connection or all the same connections we're trying to put all the work basically on the config instead of instead of uh the gear right so yes no i agreed agreed but i i know there are customers that actually do some flavor of this because of of of whatever it's a small site or something like this right they don't want to spend the money um but no i so before i know we've been talking a while i i don't want to go too far over because i'm trying to keep these somewhat concise we've talked about a lot we've bounced around quite a lot you and i um hopefully we we brought our listeners hopefully we brought everybody along with us oh man i didn't leave anybody in the dust right is there are there considerations that we didn't get a chance to touch on that you think are like really important that we should talk about a little bit oh i'm honest because i feel like we bounced around quite a lot and we hit everything a little bit or at least to some degree yes we're touching everything but i you cannot think of anything else honestly at this point it was just about when do you want to not when you don't and what happens when you do either and what kind of that what you point out what if you do a one-to-one nap by the way all this bit about one two three four seven and port offset so none of that matters if you do a one-to-one net you're specifying like this connection always goes to this router you don't have to worry about it right it's a lot cleaner in general it's a rocket leader and you have to recommend it to just do one event and it's easier because then okay this is your only entrance point is the only access point well if it's only one aisle and you don't have to look around then it's very simple right but but in general this is the recommended way and you're being told all the time this is the easiest cleanest and fastest way to do it no if you're going to suffer this is also supported but it's exactly exactly if if you want to trade man hours for like uh money yeah yeah no this is this has been great i really appreciate you coming along uh this with me uh just having a discussion again i hope everybody uh followed along i know david and i both tend to bounce around a lot so when we get together we probably bounce around twice as much and twice as fast but i hope this has been interesting uh informative i would like to know if anybody has any unanswered questions in the comments specifically any um any feedback did we go too fast uh i hope not uh is there something we didn't cover you that we should have covered please just let me know down below i'll put david's uh twitter handle in the comments below let me um let's go ahead and wrap this up all right well dude that was super fun uh again i feel like we i feel like we were like bowling a china shop and they're like going over i see david pointing and he's like he's i get what he's saying perfectly and and i feel like we're leaving everybody behind so yeah i don't know dude that is like dude you die again [Laughter] no no that was great man i really appreciate you coming along with me this was this has been excellent i mean it's such a complicated topic it's hard to make it honestly interesting it's hard to make it engaging and but it's so important to talk about like man it's like one of the things where if you don't have to deal with it in your deployment hey good for you like congratulations somehow you managed to get around it but if you have to deal with it man you really got to deal with it it's you know you have to know how it works once i had one deployment in which we had an ip address assigned put it by a provider it was internet provider and it wasn't working dude it wasn't working at all and this was just like related to this when we check turns out that they didn't allow uh dtls by default in this circuit and you had this issue that is okay i have connectivity i can get the weave on everything is cool and then wait what it's not working i had to do debug and c and then suddenly you seen the log that hey the port is not working this is and then suddenly wait hold on and then you're back to are we allowing this protocol or not and suddenly they come back oh this wasn't requested that's pretty true um one thing we did you reminded me actually one thing we didn't actually talk very much about but honestly there's not much to talk about so i guess we could probably just do it real quick if your infosec team doesn't allow dtls they say we're not going to open up that many ports like because that's like 100 ports or something that you should be allowing right for the the port hopping and port offset and everything if they just won't do it you do have the option to force tls but it's i mean yeah but it makes it tcp based at that point like there's some other things that go on but hold on all the controllers like vivo and support the ls but v1 is dtls only but you will notice that's right anyway actually is that oh i think you're right no yeah that's a very good point so yeah at some point you're gonna have to reach out to v-bond over dtls and you're just gonna have to do it like you know so that's a good point oh okay so so that's that yeah and i just i just remembered that's i think that's true um so in that case people if you run into this you got to be able to at least make that they at least have to let dtls open to v-bond like if nothing else they at least have to let details then start bringing chocolates and convincing people in other ways that's right slip a 20 under the door walk away oh man um no that's this has been awesome i really appreciate you coming along with me um i'm going to i've done some work in the lab already just to get basic connectivity up and working so i can like start working on my wine edges but based on what we talk about i'm gonna kind of revisit implement the stuff we talked about i did mostly what we i mean honestly i think what we came up with is very similar to what i did i'll go look and see if i even have to change anything um but that's because of this lab it's because it's a slab not because in your lab you can play with things anyway and it doesn't need to be perfect or aligned with everything unless you're trying to demonstrate a particular use case so yeah best practice for this lab for example all right like another thing is that we also talked uh when we were off camera of course about some things that we could be talking in another another amp so oh yeah like doing this nothing or doing uh or doing service insertion and then redirecting the traffic to a particular firewall it could be even the same firewall just another completely different villain and but doing it on the overlay not in the underlay it gets nastier sorry funnier but well that's a good that's a good point um service insertion with firewalls is is completely supported it's it's a thing it's something i hadn't considered uh with this lab i was focusing on the firewalls for the underlay but you're absolutely right firewalls in the overlay could could exist and could be a thing yeah and service like not oh you haven't you haven't seen something nasty though you have to do that and okay i caused an outage while learning about it but but it worked it's after one update that's that's it you're like it's been this many days since we caused uh an audience yeah most probably the last object had my name so yes the funny thing is that it it happened in and well anyway i don't i'm not signing the company but it happened at the cio's office at the hq dude i have done so many things in many other offices but i had to screw up the one where the guy who is one of the most important people in the company is browsing true i don't know maybe facebook and he was of course frustrated because he couldn't browse turns out that ball for several minutes as the policy was wrong because i overlooked this the default action accept reject don't do it always put accept very rarely do you play reject yeah but for the data plane policy you in general just put it as a accept because the logic is different it's not like when we try to deploy something using a maybe a a uh when the logic doesn't matter with the doesn't match with the rod mag it just gets kicked out yeah now in this case it just gets totally rejected so that's what i did and then the traffic was being added to this subset but not for the rest of it so you could imagine boom the whole site down and then somebody sends me an email can you explain me why the cio's office was offline for some minutes and you uh during business hours about that oh man we should we should we should do another one uh we should do something um i've got a i have a short list that i'm trying to work with work through but after that like these are like the the the shortlist i'm working through on the design series here is the 80 20 right like what are the big common use cases um we should definitely revisit once we get through that if people like it if you want to see more of it we should definitely make a whole other one about like okay what are the cool things like the features the corner cases the extra stuff that we can talk about we should do a whole other one on that so thanks for joining me today um again i'll put all of david's contact info uh his his uh birthday his uh phone number what was the other thing maiden's mother made his maiden name his main name i'll put that in the in the context the comments below and uh yeah if you guys enjoyed this please let me know oh here i am just like recording bad things uh just messing up by clicking on the anyway uh if you guys enjoyed this uh let me know in the comments uh below let us know and uh yeah thanks for joining us

Channel: Carpe DMVPN

Views: 656

Keywords:

Id: Xq0VE0TYA0M

Length: 60min 53sec (3653 seconds)

Published: Tue Jul 20 2021