How to Use Packet Analysis to Prove it's Not the Network (or it is the network)

network engineering uh on a Jew rebooted look there is no are no alert in the monitoring system so the network has to be fine look I told him it's it's his crappy application ok the network just moves packets around no no nothing look look who has a CCNA I do right the network is my baby it's like the apple of networks ok it just works oh man we have all been there am i right you guys it's the network no it's the server no it's the application full of moon it's solar flares whatever you want french fries everyone has fingers to point like how can you narrow down the problem how can determine what is or isn't the root cause of some network problem a performance problem whatever it is packet analysis can get you a long way there it can help you eliminate things sometimes again you can find the problem you can nail it and that's what we're going to be looking at as I mentioned in the write-up I hope you read it because I don't want it to Yammer all day just part of the day now I'm skipping some of the questions that you'd really want to do some of the legwork upfront before you get to analysis we're going to assume we've already got a data capture of the problem and we want to analyze the capture so that's we're going to start let's jump in so you've got this capture file where do you start you've got data packets how do you find the problem so I've got a fresh and solve our shark default options there are some good resources for optimally setting up wireshark to solve different kinds of problems you can look at I think Laura Chapelle has one hon song Bay has got one great stuff good information I'm just gonna start with default and as I go I'll change things and I'll tell you you know why I'm changing them why it's useful for me let's make the font bigger so it's easier to see no there we go okay so just a couple things right quick I'm going to change under preferences the layout I like to use this this 1 2 3 whatever I layout first because I really want get this out of the way I want these you know the packet information this frame to have as much space as possible every now and then I do want to see the packet details so I want them to be available but otherwise I just want them out of the way and then let's change under name resolution let's disable transport names because I just want to see the numbers I don't want to see the random weird names that your random port numbers can be I just wanna see the numbers so that's the bare minimum I'm going to do for now let's jump in okay so what's a good words a good place to start some people like the easy button down here this little circle you click on that and it brings up your errors and warnings we've got you know some things here are they important I don't know duplicate ACKs retransmissions oh gosh you know maybe it is the network so you can you know click on these and you can go to these packets and look at that.look retransmission definitely so that's happening um that's useful you can look at was it I agree under statistics this is probably want to do something like that okay this is packets per you know tick here also good you can change it to bits look at the throughput okay you can zoom in and stuff let me show you where I start I use the time sequence graphs so what you want to do this is like I said this is FTP because I mean pretty much all cat people use FTP to get their pictures you need to find the direction of the data flow so an FTP we're downloading so it's going from the server to the client and this particular graph is just in one direction so if you click on the wrong one like an AK from the client and try to look at this graph it's useless there's nothing there so you have to click on a packet in the in the data flow so here's a packet from the server statistics TCP stream graph I like time sequence graph TCP trace because it has a lot of information I've written about this particular one on my site a packet babcom did like list with all the different little markings and colors and whatnot mean so I'm not going to walk through all of them so this is a graph of data byte sequence number all the same thing on the y axis I have relative you know sequence numbers turned on so it always seen this number will always start with one in my trace when reality is just some random 32-bit number is it random I don't know I'm not really sure how sequence numbers are generated I can probably do that for you I'm not going to but so with data transfer with TCP the sequence number is how TCP keeps track of what data has been sent what data is laws need to be retransmitted etc so for every single byte of data that gets transmitted sequence number increases by that same number one bite gets sent sequence number increases by one a thousand bytes gets sent is a superstorm increases by a thousand write your x-axis is time so we're just literally looking at a number of bytes being transferred over time so what you want to see here is sort of a smooth line going from the bottom left to the upper right and because I mean this is this is throughput the steeper the line at the where the data is being transferred because it takes less time to transfer that data right so when you look at this I'm looking for big wins I don't want to have to drill in to minutiae and really have to get into the the subtleties of TCP I want I when I open this graph I'm praying for some big wins and you know what we got them we totally got them you see these big flat spots that's a big red flag the flat spot means no data is being transferred we've got some amount of time I don't know how much it is where no data is being sent so we want to find out why that is so I'm going to zoom in you can click and drag zooms and hit I if you want where it'll zoom in where your mouse is all these little Eyebeam thingies those are actual packets being transmitted so the distance between them is time and then this line sort of gray line underneath are the acknowledgments for the packets so what we have here is going out and use this flat spot and then we have this little packet this one little single packet if you click on it it will magically go to that packet in the trace that's super handy so I've clicked on it this is the first packet after some pause and here it is now I want to know what the posit is how long is it so what I'm going to do is add I think by default the time here is time since beginning of capture you can change it if you want but I'm just going to add a delta column so there's two Delta if you expand is the word I want the frame you get time Delta from previous captured packet and previous displayed packet so if right now I'm not using any display filter so you're seeing all the packets in this capture but if I were using a display filter like filtering on a particular connection and I'm looking at Delta and I'm using say previous captured frame then it's going to show me the Delta from the previous captured frame even though it may not be shown to me at this time so if this were a filtered display right now like there were other packets in this capture that I'm not seeing there could be a thousand packets between this one and this one and it's going to show me the time from the previous captured packet that I can't even see so for this I don't want that I want to know the delta from the previous displayed frame so if I'm using a display filter it's going to show me a capture relative to the the packets I'm actually looking at I hope that made sense so I'm going to apply this column and it puts this ridiculously long thing up here that I'm going to edit because it offends me just call it Delta now we can see that let's do this get a little get that other way just to create some space around it we can see that this packet says the Delta is 182 milliseconds what that means is this packet arrived 182 milliseconds after the previous one so the client sent an AK 182 milliseconds went by we got the next data frame so I'm thinking sounds like it's the the server being slow but we also want to kind of know what's going on in terms of where we are with the data flow do we have a lot of bakit do we have a lot of data in flight currently do we are we all caught up how can we figure that out well we can do a quick bit of sequence number analysis and right click yeah so I've expanded TCP TCP bet here and find sequence number and then you can apply as column you can do for the next sequence number you can apply as column and the acknowledgement number apply as column now I'm I would normally shorten these but I won't right now let's pull this out now if I click on sequence number you can see the sequence number is in the TCP header and the acknowledgement number is in the TCP header next sequence number that's not on the TCP header anything in Wireshark that has these square brackets around it that's not actually in the data that's a Wireshark informational so calculation or just it's it's informational data to you that Wireshark is displaying but it's not really in the packet so the next sequence number is the current sequence number plus the amount of data that was sent because then that should be the next sequence number of the next packet so what we're going to do is go up here we've currently got that one the delayed one highlighted we'll go to the act before it and just talk about the last few digits here four to two to five that was 122 s for two to five that's saying I have received all the data up to and including for two to four and the next sequence number I expect to receive is for two to five the next packet is indeed for two to five and the previous data packet three 649 it sent 576 bytes so that makes the next sequence number for two to five we say yep we expect for two to five and guess what it the next one is for two to five that means there's no outstanding data after this act we've acknowledged everything then we get the next one at 182 milliseconds there's no reason why the server should wait so we say well what's the round-trip time perhaps it took time for the ACK to get there the server to send the next one to us fair question if you go to the beginning of the capture always always get the beginning of the capture you need you really need the three-way handshake it has a lot of information and in there that will cover another time but if you look at the Delta between the syn synack well this is like three milliseconds this is actually I did this in Amazon AWS so it's very little delay so 182 milliseconds that's not round-trip time so I'm going to go ahead and say definitively that particular delay was not the client might be the server potentially could be something in the network introducing latency but you know for now we can say that was for sure not the client now let's sort our packets by the Delta column and what this is going to show me is click it twice to get it in descending order is all the packets that have you know a delay like the one we just saw which is this one so these first ones up here starting with six seconds and three almost three two these are all FTP control protocol stuff like typing in a super secure password and a username and then quitting setting the type these things we probably don't really care about in terms of performance these are user interaction for the most part then we have a thin packet that's seven hundred and twenty seven milliseconds we also don't care about delay before fin packets that's at the end of the connection things are being cleaned up closed down saying bye-bye we don't care about delays before fin packets so that let's move on now we have the one we already found 182 milliseconds and we have eight of these guys that are all FTP data from the server from 142 millisecond delay up to 182 then we the next one after that drops to sixty four milliseconds basically and it's an act from the server to the client then we drop to 38 which is actually the client to the server that's the first one that we might care about from the client 38 milliseconds but these to me are the big ones that I'm looking at there's eight of them all you know roughly 150 180 milliseconds and if we go back to our graph and we zoom back out we can see these flat spots are 1 2 3 4 5 6 7 8 so I can tell you because I looked that all these flat spots they do correspond to these eight packets and these are big wins if I'm talking about later on digging into these like you know 10 12 20 30 milliseconds they later on if I'm trying to eke out more performance I might dig into them but right now I want big wins and these are big wins and none of them are the client machine I can definitely say not the client could be the server so what I want is a capture taken on the server and see if I see the same delays on the server and if so I'm going to go to mr. sis head man and say hey buddy we need to have a chat about your server if you don't see the delays there then well we need to dig into the network and try to figure out where the delay is coming from so that was a that was actually fairly quick I mean how long this video is going to be but that was real there's a lot more things I could dig into but I don't to make these super long for you guys right now we will definitely dive into other types of analysis things in a lot of detail going forward so what I'm going to do is I'm going to get another capture file I'm gonna put it up with this post I want you to have a look at it you tell me where is the delay is it it let's just say it'll be at a server capture is it on the server or is it not on the server so we can either eliminate the server or maybe we need to look at the client or potentially the network leave a comment what you think email me carry at packet bomb.com and we've got more content on the website I've got some that are only available to email subscribers if you really like this stuff then I encourage you to subscribe that's where I do all my chatting with peeps about the packets and thanks for watching and I hope you found this useful and if you didn't I guess I'm sorry mm-hmm the network is fine okay I mean look no I I don't know I don't I don't I don't know okay I just uh Hey Dude what's wrong with the router we need those man go plug that back in

Channel: PacketBomb

Views: 40,220

Keywords: wireshark, tcp, network, performance, packet analysis

Id: bR5Iuuxhg4o

Length: 17min 31sec (1051 seconds)

Published: Mon Jul 07 2014