Troubleshooting MTU Problems With Wireshark

thousands of people turn to the internet for health because they have a burning computer and network problem that they cannot solve if you have a problem if no one else can help and if you can find them maybe you can hire the pack and a team Carrie Rogers back Obama calm we've got a case study here from reddit.com the networking subreddit person says slow network throughput for the first 6.5 seconds yeah yeah yeah blah blah blah blah iperf blah blah blah I've been trying to figure out the problem for over a month any thoughts sure I got some thoughts if you were able to take a pee cap and do a basic bit of analysis you could solve this in five minutes that's my thoughts so why don't we do that let's look at the P cap that the person provided after people asked to provide P cap so we can solve it here's a big gap of said problems somewhere in here I don't know what I'm looking for whatever but let's here's I'm gonna start let's go to statistics and conversations I'm gonna do TCP there's 20 for TCP connections why don't we sort by bytes the highest one is far and away the highest one there's nothing close and oh it is poor 5001 which happens to be iperf this is probably one I care about so you can do follow stream here alright and so here we are with this iperf stream so I'm looking at this and I'm thinking great I like to see we have the three-way handshake because there's some valuable information in there we have this first 24 byte packet which is pretty typical from what I've seen from iperf and of course these TCP brand retransmissions about me we'll get to that if you see my videos you know I don't usually have a particular layout that I use the columns that I use in Wireshark come and go the ones I use most often I'm gonna probably have TCP lengths up here I'm gonna have sequence number next sequence number and ACK especially when I'm troubleshooting performance because I want to see the data flow and be able to do sequence number analysis so what jumps out at me is this TCP length of 89 48 that's very large typical him to use 1,500 bytes so maybe this is offloading if you know if you do read my site there is a something I wrote up called how can the packet size be greater than the MTU and that covers TCP offloading with knit cards somebody that's what that is one way to look if we go to the syn a three-way handshake syn packet if we go open up TCP go down to the options look at the maximum segment size that is allowed and it's actually 8960 so allow 20 bytes for the TCP header 20 bytes for the IP header that's 9000 bytes that's a jumbo frame how about the server side go down open up the options same thing 8960 so that tells me this client in the server are configured for jumbo frames and if we look at the latency between them it's microseconds so this is on a LAN it's a great most likely this is an actual jumbo frame it's 9,000 14 bytes 14 bytes for the Ethernet header so that's a 9,000 bite into you so this is this is a demo frame so we send one there's an ACK the ACK is for that 24 byte packet up here because if we look at the acknowledgement number of 11:29 that's acknowledging bytes up to and including 11 28 so that is this packet and then we send another 89 48 byte packet then the next thing that happens is Wireshark says it's a retransmission it is in fact sequence number 1129 which is maps back up here to this first large sized frame that we sent of 1129 now blue for me signifies the push bit the orange is a delta of 200 milliseconds for more actually I set those a few videos ago and they're just Dillon here but I find it useful so I've left it so let's let's change I've got just the time let's go to view time display format second since previous displayed packet of course you can add a TCP Delta or a delta column if you want in here so this one comes 200 milliseconds from the previous packet which these are micro seconds in between so this one comes about 200 milliseconds after it was first transmitted 200 milliseconds is a pretty typical around trip timeout generally that's calculated on the fly 200 milliseconds is pretty pretty typical and then we send it again the same sequence number same data 400 milliseconds and 800 milliseconds and 1.6 seconds this is exponential back-off folks and after we transmit it 1 2 3 4 times we see this same sequence number but look at the the TCP link is 512 bytes and you can look at the next sequence number has changed this is obviously a lot less this is reflecting the fact that we've only sent 512 bytes the next packet is an ACK and it is acting 1641 this next sequence number of the one we sent is 1640 1 that means this packet was received and is being acknowledged so this 512 byte packet got through when these other ones didn't and then if we look so this was sixteen forty one the next one is sixteen forty one and then the next sequence number is 2153 it's down here I don't know what that is 89 that actually actually comes from up here the next sequence number ninety 25 this is 925 but the next act we get is this one apparently did not get through but we got 21 53 which acknowledges this one and from that point on we send these small you know mostly 512 byte packets as we can see this is the next thing with some around this one is 31 49 the AK is 31:49 so they're getting through and then you know so we sent these first two here sorry this 189 4889 48 it's like over 17,000 bytes so then we send all that data we retransmitted these are all real retransmissions as 512 byte chunks and then around here we actually send them all we're all caught up I guess you could say and that took let's find out so if we go back to let's change this to view time display format time since beginning of capture that's at a time reference on this particular connection do to do and so we recover after we tried to send huge packets jumbo frames I didn't get through but then we chop them up in little bitty 512 byte packets and send all that data it takes right at almost 6.5 seconds to do it tada that explains the you know as he saw 6.5 second delay at the top so what's happening here is we're trying to send a jumbo frames some something between the client server is not configured for jumbo frames and it's dropping it and then this particular sending TCP stack must have some kind of heuristic to say you know we tried to send this large frame amount of data it didn't get through so let's try again and just try really small sizes and see if that works and oh it does now what really should happen is it would either get fragmented unless the DF bit asset is the DF bit set the don't fragment no frankly there it is yes so it's not this should not get fragment it's what to get dropped and then whatever is dropping it because it's MTU is lower should send back an ICMP message saying that the MTU has been exceeded and we have to drop this and here's the here's my m to you so if I do or ICMP there's nothing here so whatever device dropped it did not send an ICMP message back doing path MTU discovery to the sender so either some device drop the ICMP message like a firewall that's configured to drop ICMP or the device that dropped it is configured with no IP and reachable and it just doesn't send it to start with so those two things can break path MTU discovery and we're just lucky that the sending stack decided to try smaller packets otherwise this connection just would have broken and died a horrible horrible death so you need to go through figure out the devices in between fine the MTU of the interface that is not set properly and fix it or I mean course you could use a lower rim to you and your in devices but then you know if you're on a LAN and you're using jumbo frames and great you're gonna get better throughput so there you go that's it that's what's going on that explains the six point five second delay thank you problem you know spot it if someone can do this problem like hey I have this problem that give me a peak amp and you spend five minutes like oh here's a problem and then you go look and you find the device that you track it down and fix it or you could spend a month and then you can go to read it and then I'll fix it for you alright till next time you

Channel: PacketBomb

Views: 20,496

Keywords: wireshark, packet analysis, mtu, ptmud, jumbo frames, case study, retransmission, tcp

Id: XemWuiCorKQ

Length: 11min 23sec (683 seconds)

Published: Wed Jul 22 2015