How to Troubleshoot Throughput and TCP Windows

ah cleaver buddy this is kiri I'm not going to talk like that I'll kill myself or you'll kill me so this is Kari with packet bomb calm this is another video in the it's not the damn Network you guys series I'm going to show you some things you can look for that can impact performance but they really don't have anything to do with the network so I strongly recommend you read the write-up I wrote previous to this about throughput and TCP windows so receive window congestion windows send buffers if that doesn't make any sense to you go read it it won't take long so let's just jump right in I've got two captures one is going to be cinder bound one is going to be receiver bound meaning the bottleneck is the receiver on one and the sender on the other so let's jump right in if you recall the last video we added had a default layout to Wireshark but we added a delta column we added in the sequence number of columns so we've got this capture and you know what I want to know when I'm looking at throughput is you know what's the round-trip time if I'm working with someone I can just ask them but they don't know they can't tell me or whatever you can look at the three-way handshake so this I know was captured on the client side because the delay is between the cinesín ACK the client sent the syn and had to go across the network the server had to respond and it had to come back across the network so the round-trip time is about 98 milliseconds if this were taken on the server side then the delay would be between the syn ACK and the ACK just think about it okay so we know that's what we're kind of dealing with is around 98 milliseconds so you know that when I'm troubleshooting performance I like to look at the TCP stream graph so you need to because the stream is one direction you got to click on a packet that has data in it and the data flow so I'm clicking on one of those packets and I'm going to bring up the DCP trace graph okay so let's jump in down here at the beginning you can see what would appear to be slow start and then there's an abrupt change where we're doing you know what appears to be slow start and then it just stops and then it just shoots out a bunch of data so I want to see what what's going on here so I'm going to zoom in start the beginning of this I'm going to click on this packet and I'm not going to go over what this graph is I've written about on my site and I talked about in previous videos but you know this is bandwidth this is bytes over time so click on this segment yeah okay and we'll bring this down out of the way now so what I want to do is add a column I want to know about my bytes in flight so I can add that as a column under TCP under sequence analysis and remember the things in square brackets are just Wireshark informational stuff is not actually in the the data not in the header or anything apply as column boom okay so what this tells me is on this packet the bytes in-flight at this point in time is 1460 now I can see that the amount of data in this packet is 1460 so what that tells me is there's no outstanding data for all from this this acknowledgment we're all acknowledged up so this is the first data out on the wire because everything else to be acknowledged so I'm going to kind of follow this along you know from that graph we sent a whole bunch of it at one time so I'm scrolling down I'm looking I'm looking and one thing that catches my eye is this packet why because it's a power of two boundary that is exactly 64 K that that is something that should jump out at you additionally what else jumps out at you aha this push flag if we come over here to the length of this frame previously they were all maximum size you know for 15 14 so MTU of 1514 byte Ethernet header and then we have the smaller one so with a push flag and a small bit of data what that tells me is we reach some kind of data boundary obviously at 64 K so to me that's either the application sent 64 K to TCP or the buffer size the send buffer size is 64 K now there's no delay we don't stop sending data so our congestion window is open enough our receive window is open enough and apparently there's enough room in the buffer to buffer this previous data because we can't remove data from the buffer until it's been acknowledged by the remote side because we may have to retransmit it so we continue sending data scroll down and then here's the last one this is exactly 128 K it's a push flag again same thing we got full frames and then the whatever is left over so another 64 K so we sent to 64 K chunks back to back and then we waited full round-trip time for the acknowledgment so we would not send more than 128 K so that could be you know it could be congestion window or it could be the buffer size of the sender now if we look at say the receive window that's this packet it is a megabyte and we've not had any data loss yet so we're right now the slow start threshold no I'm sorry the the congestion window is set to the initial congestion window which usually is some arbitrary high number a lot of times is going to match the receive window so this limit is probably not the congestion window it's probably the buffer size so either the buffer is 64k while you tell you I know the buffer size is 64 K at least that's what the operating system reports this is Windows however clearly the bytes and the bytes in flight - this time is 128 K now when the application sets a buffer size this was sort of a suggestion to the operating system so what this tells me is even though the buffer 64 K or it says it is it actually has enough room that double that of 128 K because it's sent 128 K in-flight without acknowledgement so clearly it can buffer more than just 64 K but the congestion window is most likely much higher than 128 K so I'm going to say that this is you know bound by the the buffer size so we wait for all these acknowledgments to roll in we send like a little bit of data here it's like 2 bytes it's probably just to my perf stuff and then we start up again we start back everything's acknowledged how do we know well there's a couple ways one because the Bison flight is 1460 in this packet is what has 14 60 bytes of TCP data you got to look at the acknowledgement number scroll back up so it's what 169 scroll back up 169 was the last thing we sent so we know we're all acknowledged up and we do it again so this repeats itself this pattern over and over again we send 128 K we stop and then we wait for acknowledgments this flat spot down here is where nothing's happening we're just waiting for the data to make it across you know the first two because TCP is going to acknowledge every two packets and then the acknowledgement start rolling in one for every other packet did it did it did it and once we acknowledge they all roll in off we go again now we can also see from this graph that the receive windows way up here there's all the space in between the space between all this space is available buffer space you know this is time selfish of straight vertical line is one moment in time I'm not going to sing them so this base from the top of this line up to there is the amount of free buffer space on the receiver so plenty of room right so to me the issue here is considering there's no packet loss so the congestion window should be open enough to match the receive window the issue here is the send buffer is not large enough for the bdp of this connection okay is there anything else I want to say so you know I did I've mentioned you know what I think the congestion window is a couple of ways you can kind of estimate it is based on the bytes in flight you know but of course if you're bound by the size the receive buffer I mean the syn buffer that's not really going to help you a whole lot on Windows there's this little program called TCP analyzer that comes with the SDK if you down the SDK go digging in like a bin directory it's like a graphical netstat if you click in it you click on a connection and hit stat and I'll bring up another window it shows you like the congestion window the round-trip time out the number of fast retransmissions all kind of really cool TCP stack state information the only downside is as far as I can tell you can only start it on the established connection so if I want to see the stats from the very beginning of a connection I don't know how to do that because by the time you can click lick lick lick and find it you know the connections already been going if you're on Linux you can use socket statistics the SS command I think it's - E or maybe - I - give you congestion window and slow start threshold and all that I'll put something in the write-up under this post okay so let's switch over to the next one which by process of elimination must be receive window bound or receiver bound so same thing you know we start at the the mean just go ahead and add bytes in flight to this one as well so right click by it's in flight applies column I'm going to go to the beginning these are the same host actually running iperf so it's the same it's the same round-trip time so let's click on a data packet let's bring up TCP trace graph have a look so if we zoom in towards the beginning we can see we have the same behaviors before we got slow start starting up did it up and then the difference here is and you can see then it it goes back to shooting out this data all at once the difference here is you know this top line you know is the receive window we run into it or we get really close to it actually we don't there's just a few bytes left but we can't send another full packet and then over here we we start sending out our data you own the breeze when we did 128 K so this one if we click and go go there 1416 so this is the first flight out the door after everything's been acknowledged scroll down let's open this up a bit so we get we get to 64 240 so the difference between that and 64 K is not quite again a full size frame and it doesn't send so it's running a full ton or full round-trip time for an acknowledgement you know get an augment acknowledges two packets so we send that opens the window up a little bit we send two packets and then we just kind of limp along like this acknowledgement two packets acknowledgement two packets so what that looks like is if we zoom back out so we send up to the 64 242 here we have to wait for the round-trip time and then we limp along sending a couple of packets for an acknowledgement until we finish sending however much we need is in probably another close to 64 K and then we do another sending out this data and it's hard to hit that just right so there we see we actually hit exactly 64 K and it says TCP window full so we sent all this data just one after the other no delays until we filled the receive window if we look this acknowledgment the receive window 64 K so we filled it we have to wait for an acknowledgment you know it's a full round-trip time that that gives us two more packets and here we go limping along at two packets for every ACK to finish out probably another 64 K then the rest acknowledgments roll in the rest is knowledgeable in and we just repeat this pattern over and over we send 64 K we filled the window when the data here reaches the top line we've filled the receive window we can't send anymore we wait for the window open a little bit and we limp along a couple of packets at a time and we repeat this process over and over again until we send all of our data so why isn't the received window opening more than 64 K well if we go back to our beginning so the client says hey I can use a window scale but I'm not really going to use it so whatever I acknowledge and send to my window size value that's what it really is but this is the client who's sending the data who cares the server the cynic is the one receiving the data but it is the same thing it says yeah I know about windows scale cool stuff but not going to use it so it's limited to 64k and if we go to stats io graph as mike is burger chain just to advanced we will do this to say a maximum of TCP window size and I only want know both both sides always include a window size nor TCP header I only want to graph the window size of the receiver and his source port is 5,000 won so if i graph that let's change this to like this this is you know 64k and then these spots Inbetweeners when we're receiving data and not sending any acknowledgments if you go back here you can see this is probably slow start like single ax and then couple of acts and as we send one more data so let's say I want to graph the average of the bytes in flight TCP data analysis bytes and flight invites in flight is only in the header when it's someone who's sending data so I don't have to filter on any particular side if i graph that and that's not line that's f bar but whatever that's fine it's kind of cool here you can see how we start with slow start you know as we build our build our congestion window because it starts off at like you know - we can put more and more bytes on the wire at a time and it builds up until that almost hits our receive window almost and then it does so this is a point we're hitting we're sending 64k and then we have to wait you know round-trip time that we can eke out a few packets at a time and then that repeats itself over and over again so that's kind of a cool vision of visualization okay so to recap our first this one our first capture was cinder bound because the send buffer wasn't large enough to fill the pipe so we had a lot of unused space in the pipe we weren't using so for that one you want to go to the cinder and you need to either adjust something in the operating system or the application and then for the other one the receive window wasn't opening up if the application is specifically setting it to this value of 64 K which is true in this case or the operating system has it configured that way so if you see something like this then you can go back to whoever's complaining and be like look this is not the network fixture excuse me I get worked up sometimes so what do you think oh I didn't want to say on the one that this one do you notice these that sometimes we only send 64 K at a time and then we wait a round-trip time before sending the rest it happens a few times here's one other times we send all hundred for me k Havel I'll put this capture up have a look at it see if you can figure out why yeah cool email me let's talk about packets I talked to people all the time on the email about packets love it Kari and Pekka bomb comm join the email list you'll get some additional videos and those are my favorite people so yeah thanks for watching I hope you think this is cool stuff because I do and if you don't just watching Hey

Channel: PacketBomb

Views: 16,667

Keywords: Throughput, TCP, Network performance, wireshark, congestion window, receive window, send buffer

Id: qFWjugyKyrE

Length: 20min 7sec (1207 seconds)

Published: Wed Jul 30 2014