115 IPExpert MPLS Troubleshooting L3VPN Examples

[Music] I want to show you a couple of problems that you might encounter when you are troubleshooting MPLS VPNs now this is not by any means going to be an exhaustive list of all possible problems that can happen in the MPLS VPN scenario but I'm just going to show you a couple of really really dangerous ones now I'm going to do basic troubleshooting things here and just I want you to be able to see how those symptoms look but keep in mind that you could have many different problems that at first glance look the same but they have very very different root causes so the first thing that I'm going to do now is I'm going to go to r2 and I'm going to do a very very unusual thing I'm just going to turn off stuff now another thing that I'm going to do here is I'm just going to go to our five and I'm going to say clear IP bgp star all so I want to clear all bgp sessions from r5 and from r2 I'm going to say clear MPLS LD brief neighbor star now the way that there is a good reason why I'm doing this now this is probably going to be the state in which you are going to if this was a troubleshooting ticket in which you are going to find your routers so your troubleshooting task is going to say something along the lines of our sevens loopback cannot ping our eight loopback right and your status on routers is going to be like this so this is this is the situation that you're most likely going to encounter your routers in so you're going to go to our eight and you are going to say ping 1000 0 8 and of course you're not getting any responses so the logical next step in this troubleshooting would be to do show IP route okay I'm not seeing any routes other than connected routes ok what is my pece routing protocol it's erp show IP eigrp interfaces do i have the neighbors and here you can see that you actually do have the neighbor so you have one neighbor okay let's move on to our 8 is everything okay on this side which is the routing protocol it's BGP I should run show IP BGP summary okay mental note I'm receiving some prefixes here now which are those prefixes show IP BGP this is actually wrong so give me just once I clear IP BGP star I shouldn't be actually receiving this these were a little bit stale so if I do show IP BGP summary here oh wow okay let's say that this is the state in which one it should be really zero that I have no idea where six is getting this front but anyways on our eight let's say that I see these prefixes and I do show IP route V RFA so just to show IP route so I'm seeing these prefixes so let's see on our five what is the situation on our five here I would say show IP route V RFA and in this fear of here I'm getting the TIG RP prefix but I'm not getting any prefixes from our six now okay I do apologize now this is what I was waiting for but I thought I'd actually cleared the session but I cleared it only from from our five so now this is the situation that you are actually going to encounter on Rho IP BGP summary zero prefixes of course you're not getting any prefixes so now we have checked what is happening on our seven we have checked what is happening on our egg now let's move on one step in towards the center of our network now I'm going to check what's going on on our five and what is going on on our six mind you I've shown you what the root cause of the problem is the root cause of the problem is safe on our to being disabled but you see how it manifests itself by routes not actually being in our seven and all right that will become clear soon so the next step that I'm going to do when I don't know what the problem is I'm going to move in to our five and I'm going to say here show IP route v RF here I'm actually receiving the routes from our seven and then I'm going to go to our six and I'm going to say show IP route DRF a and here I'm going to see that I actually received the route from our eight so I know now for a fact that this communication is okay and that this communication is okay so I have problem somewhere between here so the next thing that I'm going to do is I'm going to start troubleshooting that so I am now completely ignoring the presence of my CA devices here the only thing that I'm looking at right now is the relationship between r5 and r6 so show IP route so not vrf I'm not focusing on the main routing table do I have the route for our sex I do can I paint it I cannot paint it if I cannot think it it's very unlikely that I actually have bgp session running show bgp evpn before a unicast or summary this confirms my suspicion I do not have bgp session with our six-hour five I'm done with you going to our six what's the status here show IP route do I actually have the route to our five I do have the route for it can I think it I cannot pin it okay show bgp evpn before a unicast all summary do I have the BGP session configured at all I can see that I have the neighbor and I can see it's actually active okay is it configured properly maybe it's the wrong source interface maybe something now is the time to actually take a look at the BGP config so I'm still what you it is something there is something wrong with our five and something wrong with our six so taking a look at here no BGP default ipv4 unicast okay don't care about that remote is update source loopback zero activates and community both no passwords nothing of the source configured this looks okay mental note redistribution is in place going to our six show run section router bgp what i'm expecting here to see is the update source loopback remote a s 56 that matches the other side I'm using the correct neighbor I before it's activated there should this matter you know maybe spend a couple of seconds here pondering shouldn't matter VPN before activated self community both nothing special mental note I can see the configuration between our six and r8 as far as I'm concerned r5 and r6 are properly configured it is something else so now I have to troubleshoot yr5 cannot communicate with our six they are configured properly so let's take a look I'm going to say trace route 192 168 to 0-6 and it stops immediately okay that's bad let's go to our for show IP route I seem to have the route for our six cannot ping it okay our to show IP route I see our six ping our six I can ping it so between our two and our six brain works between our four and our six thing doesn't work okay it could be some access lists so run interface 0 0 1 0 206 no access lists 204 no access lists so nothing is configured to filter the trunk in the transit traffic so now I'm getting a little bit paranoid so show access list I'm going to say inspect zone just to see if I have anything that filters the traffic no okay from our 6 show IP route - I actually have the route to say our to ping 182 168 0 - that works 0 4 doesn't work ok time to take a look at the labels show MPLS forwarding table on our five what is the label that I'm using for our 6 is loopback 402 going to our two showing pillars forwarding what is the label that I'm using for our 6 is look back it is 202 going to our to show in pillars forwarding BAM I have nothing this is the problem why don't I have anything in my MPLS forwarding table these MPLS running at all show em pls tell DP discovery so this is what I'm going to look at show em pls show em pls LDP bindings so I do have the control plan in place I have the MPLS Discovery running I do have the labels in the binding database I don't don't have them in the forwarding table at this point I want to explore what is happening with my forwarding table and I see what the root cause of my problem is so now I know that ITF here is going to probably solve the problem so I'm going to confirm that with show MPLS forwarding table now the forwarding table is populated so on our five if I go back to our five I see this session coming up between five and six on our seven show IP route I should be getting some routes right now I am getting them so if I go here and I think between the loop packs I know that my configuration works save the configure all the routers maybe ping from the other side as well and done so this would be a troubleshooting approach that you would kay take here but you see that so I was obviously pretending I obviously knew what the problem was but you saw the steps that I have taken in troubleshooting this so I started from the edges right I first tried to reproduce the problem I was pinging from one side to the other side okay that's not the case do I have the route here do I have the route here don't have the route so let's move on to the next step I'm moving closer to where the problem is do I have the BGP do I have the route do I have the routes no bgp session is down I'm ignoring now the fact that this is MPLS VPN now I'm troubleshooting the reach ability between r5 and r6 in the main routing table so I was working on the assumption that there was something wrong with the IP traffic I was going there and I saw that the route was there all over the place and I was narrowing down where the problem is and finally I narrowed it down to r2 so it was something on our - and that kept me digging a little bit there so you see how the problem IPSec turned off you saw how it manifests itself now the real manifestation the real symptom that I should have picked up on and I was playing here down my capabilities little bit what I should have picked up on is the fact that on our to the communication to and from our to works but at the transit traffic doesn't work this was an immediate pointer that this was a safe problem but I didn't want to jump on it right I wanted to show you how you could or how I could when I was you know when I knew a little bit less than I know now now I can actually read much more into the symptoms having you know 80 plus bootcamps does help with with that capability so this is how I narrowed it down to say it was a process of elimination I was checking the control panel and finally when I reach the forwarding plane it didn't exist on our toe if the forwarding plane doesn't exist what does the set table say oh there is no self table so this is how we actually pinpointed what the problem is now let's take a look at another issue here so I'm going to go to our six now and this is the loopback of our sex now this is running OSPF so the main routing table here R 5 r 4 r 2 and r 6 are running OSPF let's change the loopback address a little bit so what I'm going to do here now is I'm going to change the loopback address to be actually / 24 now if I go to our five oops if I go to our five I can still ping back of our sex okay if I do show IP route OSPF this loopback is now still showing up as slash to return right if I go to R 2 if I do show IP route OSPF this loopback is still showing a slash Torito why is it showing a slash 32 because remember by default OSPF will advertise all loop bags as / 32 let's see if our ping now works but it obviously does let me up do one thing before we proceed so clear and P les Feldick actually I'm on our six I just need to up to force LDP to uh to actually uh reset and this is the fastest way to do it okay so let's wait for things to converge a little bit in order to so I'm just going to give it a couple of seconds so going to our seven now now I'm going to try to ping our tit so this is the scenario that I have right now okay I'm now starting and this is the state in which I'm very likely to find my devices if I start the troubleshooting section so from our site are seven I cannot be all right show IP route do I have the rats yes I do have the rat okay maybe our eight doesn't have the route back show IP route so you see I'm always going to be working from the edges I don't know what the problem is right and here I see that I have the routes you know what at this point I am pretty much sure that this is not a control plane problem because the router making it across there is just one thing that I need to verify here on our five I'm going to show IP route the RFA I'm just going to make sure that this next hop here is actually reachable from the main routing table if this is true my control plane is very likely okay so here I'm going to say show IP route DRF a so I'm looking for the next top of this route and in the main routing table I'm trying to pin it it works so this tells me that my MPLS VPN control plane is okay now this doesn't mean anything for the MPLS control panel I'm talking about MPLS VPN control plane is very likely okay why because my pece routers my pece routing protocol work and my PE routers actually exchange these routes and those routes ended up on CES so that's done that actually works what doesn't work is getting the traffic from A to B so now this is clearly a data plane problem well at least I think so so let's take a look what is happening so I'm going to go here I'm going to do the tracer let's see where the problem stops now take a note this is wrong thing to do so I'm going to do the trace route to our it and it tells me that it stops on five now this is the reason why I'm telling you this is the wrong thing to do right so this is now not me the troubleshooter talking this is me the CCI instructor talking you can see here how trace route is lying to us now we know exactly where the problem is the problem is on our set or maybe on our to but trace route if I use the IP troubleshooting tool to troubleshoot this problem is actually going to show me that the problem exists in the wrong place so don't use this for troubleshooting MPLS VPNs follow the labels so what we need to do here is we need to figure out what is the label that is actually so sure IP route vrf a what is the actual label that is going to be used to forward this traffic now if I do show MPLS forwarding table I am NOT going to see this label here the reason why I'm not going to see this label here is because this is not LDP derived label this is the label that is learned from MP BGP so if I do show BGP EP and V for unicast all or actually say unicast vrf a I'm missing number four here so this is going to be the label that is going to be used so this is the VPN label but they also need the transport label so what will be the actual transport label that gets used for this well if I do show IP surf vrf a I will see the transport label in use so this is the label stack so this order to be this will be the VPN label and this will be the label that gets used so what I need to do here identify in this output here what is going to be this label so this is label 402 and I can see that it is the label used for the next hop so now I need to follow this label across so I'm going to go to our for show MPLS forwarding table and if I take a look at label 402 it tells me use label - oh - that's very good so if I go to label tour to show MPLS forwarding table label 202 means send traffic to our six with no label well that's okay because this is PHP right well no this is not PHP this no label means destroy the label that means no matter which labels were actually on the label stack do not send them labeled that means that this label here 6:08 the one that we saw here in the label stack the packet that our two cents sorry that our four cents - are - will contain the label 202 followed by 608 but when our 2 is supposed to send traffic to our 6 it's supposed to send it labeled with 608 but no it won't do that it will actually remove that label 608 well this is the problem now why is this a problem well let's take a look do we actually have the label in our label database in our Lib so if I do show MPLS tell the the bindings what I'm looking for here is 100 to 160 8:06 now let's take a look I do have the label locally assigned 202 and we know for a fact that our 4 is using this and then I have the remote binding from our 4 but where is the binding from our 6 well here it is our 6 actually in its routing table so if we take a look at this show IP route in its routing table actually has slash 24 out it doesn't have / 32 route that everyone else has it has the loopback that is configured with slash 24 now mind you it's OSPF that advertises this route as / 32 so our 2 here receives the label locally assigned by our 6 4/24 but our two doesn't have this route in the routing table but it doesn't it does have / 32 from our 6 but it doesn't have our 6 is the label for it because our 6 never assigns the label 4/32 because it doesn't know about the existence of this route because you see LDP looks at the routing table not at always be f database it looks at a routing table so there is now discrepancy between our IP routing and our MPLS database and this is why our 2 is sending unlabeled packets to our sex now if I get just a trace route from our 5 I would see the exact same output that I had before no differences in output because in IP traffic in regular IP traffic we cannot see this because when our two needs to send an unlabeled packet to our sex are to help the route for it but it cannot send the labeled packet so our MPLS traffic is what suffers not the regular IP traffic how do we fix this problem well with this problem but made by making sure that OSPF actually advertises the correct mask on the route so now when I do show MPLS LDP bindings now I will see that for days / 24 Network I actually do have the binding from r6 so now this is in place I also see that I have the local binding I'm sorry this is the wrong network I do apologize where is it it's just one second where is it where is it oh of course it's going to be at the top sorry so now I see that from our 6 I actually have the binding now I have the local binding for this flash 24 as well so if I go and take a look at my MPLS forwarding table now I see that for this network here we are using the correct label and on our - as well show MPLS forwarding table where is it 0 0 now take a look at this it's no longer delete label it is pop label so now when I go to our 7 and when actually do the trace route now it works or if I do ping now it works as well there is one thing that I want to show you about this so if I did show MPLS LDP discovery now we are going to get this message here on our neighbor statements now I would really like to meet that person who thought that this was a good idea to put this in the output because this here indicates that oh my god there is a problem actually no this means oh my god this is good because now it works because before when we actually had the host route there was no indication of a problem let me show you that so here if I remove no IP ospf Network point-to-point when that route actually gets advertised so let me say you show IP route let's wait for there we go now there is no indication of the problem and now we actually do have a problem but now when I actually correct the issue so let's wait for that to be advertised now when it's going to say that there is no house trout now actually everything works so I would really really like to meet this person who put this in iOS and asked him what exactly were they thinking now there is a situation when not having your house trout is actually a dangerous situation but I mean there shouldn't be warning like this in this case it shouldn't really really be there

Channel: CCIEORDIE.COM

Views: 7,983

Keywords:

Id: 3qIGndjzHIU

Length: 24min 45sec (1485 seconds)

Published: Fri Feb 02 2018