All right packet heads, welcome back to the
channel! So in this video what we're gonna do is take a closer look at retransmissions.
You don't have to look far in a pcap to find one. But how can we understand it and
what do we do about it? Stick around. All right so in this video we're going to
take a closer look at tcp retransmissions. No lie, as a packet analyst, I spend a lot
of time taking a look at tcp retransmissions, and no doubt you as you've looked through
your pcaps, you've run into them. You're running into different flavors of retransmissions,
spurious retransmissions, fast retransmissions. So I thought we could start with a basic tcp
retransmission, dig into it, and we could look at also what to do about it. What does that
retransmission tell us about our flow of data and the network that's carrying that flow? So go ahead
and hit the link in the description down below to download the pcap and you can follow right along.
Let's get to it. So here in this trace file we can see we have a simple tcp thread - SYN, SYN/ACK,
ACK - this is just a single tcp conversation. We can see that the client is sending out
a client hello, going out and talking to our server. Let me just adjust this a little
bit - just going to blow this up for you guys just a little bit, and i'm going to readjust
my columns over here, my time column this is set to - if we come to view time display
format second since beginning of capture. So this is basically my running total of time.
Next to that I have my delta time. Now if you're not sure how to set up that delta time you can
click this video up here, where I go into this basic setup of what you see here on my wireshark
profile. Okay so it doesn't take us long to run into problems, right. We see a lot of black lines
with red letters - AH! Is our network falling?? What can we learn?? Why do we see all these tcp
retransmissions?? Well let's analyze. So here we can see in packet four that the client's trying
to send this client hello over to the other side. And this is where for me in this profile,
something that I like to have is my tcp segment length. You see I have 214 there. So instead
of having the full frame length including the ethernet frame, like seeing that full length up
there which is the standard profile in wireshark, I like actually to see how much payload is there.
What that does is it shows me what packets are just empty acknowledgments and which ones are
actually carrying data. So to get that if you just come down to tcp - pick any packet - and you
come down here to tcp segment length, right click you can apply as column to add that as a column.
Again I just like to see that just because it's easier to spot which packets are carrying data and
which ones are empty ACKs. So here the client - we can tell we're capturing client-side because of
our handshake our SYN goes out, SYN/ACK comes back and we're capturing on 101. That's where our
delay was perceived. So our client hello goes out, 214 bytes, and here's the deal, we never get
a response. We never get an acknowledgement. We don't hear anything from the other side. So
what does the client do? Well, when it sends these bytes out - the tcp sender - what it does is
it actually starts a stopwatch internal to itself. It goes "You know what? I sent out those
214, I'm going to go ahead and start a timer, and when that retransmission timer expires, I'm
going to go ahead and resend that data." Well how long is that retransmission timer? Well that
depends upon the type of stack that we're running. What some of its default settings are. But as
a general rule of thumb, what you'll find with a retransmission timer, it's typical to see it
if you see the network round trip time and the initial round trip time in the handshake.
So since SYN/ACK is around 85 milliseconds, it's not uncommon to see the amount of
time for a retransmission timer to be a multiple of that, at least three or four
times the initial network round trip time. What that does is it gives the data opportunity to
get to the other side and get that acknowledgement back. Uh what if we took a different path? What if
the client on the other side was busy ACKing some other data, or some on some other connection
and then it finally gets around to us? So it does give some leeway time there before we just
fire off that retransmission. But as we can see here our RTO or retransmission timeout right
there, in this case it's 307 milliseconds. So there's our first retransmission. Well
after this around, 90 milliseconds later, what we see is that the other side goes
ahead and sends us an acknowledgement. Now that number is around the network round
trip time. So 90 milliseconds here I can see 85 milliseconds up here. So that was about the time
I would expect to hear an acknowledgement. What this does is it tells us that our data got there,
but we also see some other information here. We notice tcp previous segment not captured. This is
interesting. So if I take a look at that this, is the other side sending back to me, if I come down
here and I take a look at the sequence numbers. So i'm going to come down to sequence number.
Well the sequence number on this ingress ACK jumped. The last time that I heard data coming
in from the other side, the sequence number, if I go back to packet 2, it's the last time I saw
something from this direction. The sequence number advanced from 0 to 1 in the handshake, so the very
next packet that I should have seen coming in from that other side should have started at one, but
it doesn't. Instead we see it jump up to 4067, and that's why wireshark is flagging - whoa whoa
tcp previous segment not captured - the sender on the other side thinks that it was able
to advance the sequence number up to 4067. He's like hey, I sent you 4067 bytes
or 4066 bytes that's already behind me, I've moved forward in my sequence numbers. This
is why we see tcp previous segment not captured. Wireshark's going whoa wait a second, we saw a
jump in sequence numbers. Now since the other side advertised that it was at 4067, but it backs up
and resends previous data, that's why wireshark goes - oh this is a retransmitted segment.
Now there's a few interesting things that are going on here. First of all this is
what my eyes do. Two seconds hmm. Also low segment size 536. In fact we can look into the
rfcs and take a look a deeper look into tcp, but basically this is basically the tcp minimum
mss that it wants to try. It doesn't want to go lower than 536. This is basically the min.
That's why my eye catches it. This is a tcp stack that's saying you know what let me
just give this a shot to see if this works. And sure enough it does. This packet makes it
through and the client is able to acknowledge it. So what happened here? Well likely
just by studying these retransmissions, what we see is that the other side tried to
get data across, but the mss ran into problems. Likely see it was thinking okay, I got a larger
mss right, this receiver can handle 1460 why don't i go ahead and send segments of 1460 across
to the other side. Well it tried to send those. Those probably got dropped on the other side.
It probably tried to retransmit. Retransmit. Retransmit. And some mtu problem happened in the
middle and finally the other side says - you know what before i give up on this and just reset this
connection, let me just try the minimum mss that i'm allowed to. Let's just try 536 before we give
up. And sure enough that packet made it through so then we acknowledge and then the other side
goes - oh wow that packet made it through, great! Okay here's another packet that was
missing, here's another one that was missing, client says sweet we're good we're talking. Here's
another packet that was missing. So what does this tell us? Well first of all, our client hello
likely made it across the other side. This very first one but the problem is is that the sender on
the other side likely tried a larger segment size than the network can handle. There was probably
an MTU that it ran into. Now this would be a very interesting trace to take from the opposite side.
This is oftentimes why I like to do dual side captures, one in the client, one on the server,
and then we can see the server let go of 1460s or maybe even 1440s and then we get that feedback
from the network, maybe an icmp - I wanted to send this on but the do not fragment bit was set,
so I had to drop it - maybe that's coming from a router and that would tell me what link is having
the mtu the lower mtu than i'm trying to use. So this is just one form of retransmission we hear.
We saw the RTO expire, we saw a retransmission, and we also saw retransmissions coming from
the other side that showed us about a gap in data that likely was due to an mtu problem. A lot
more to say about retransmissions and I'm going to make sure to cover those topics in other videos
- spurious retransmissions, fast retransmissions, and even more. So thanks for stopping by
the channel, make sure to like, subscribe, forward, share, all the things to your friends
and i appreciate you viewing! Take care!
