TCP Performance Deep Dive, Part 1

[Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] okay hello welcome to the I okay I hear audio you can't be too sure I screwed up last time we spend five minutes around with the audio so welcome to the what is this the packet bomb livestream experience episode three happy Thursday 6:00 p.m. good audio yes thank you it was really stupid of course you know if you you guys if you've been paying attention you know that I've got a new PC so I keep moving it around and shuffling things I put the mic on here and when you unplug and replug things maybe in a different USB port windows isn't super smart to like know that it I don't know anyway I had to retell it which port that the the microphone was on but I couldn't figure that at the moment I was too flustered you guys stressed anyway it's fine so Thursday this week we've got some this is tequila soda some mango juice twist of lime and some Trader Joe's has this nice chili lime salt that I've got on the rim so um what are you guys sippin I'm a no for not everyone it's you know it's in uh it's six o'clock it could be who knows what morning for you thanks for spreading the word mr. Bruin daya preciate that getting some folks to come check out our study sessions for Wireshark so it's gonna be you know there are some good tequila's I I just went my go to you know that's if I don't want to splurge a bit my go-to is um Cazadores it's like 25 bucks and it's pretty solid right I like how amigos I think the first one I talked about the rocks tera mana it was it was good it was fine I had the Reposado but it's just it's too hot it's too hot like I closed my window so you don't get too much street noise but it's hot in San Francisco so I was gonna do bourbon and we're doing a more of a tropical flavor thing which it's okay it's the first time I've made it I did one last night with strawberry it was really nice so what's going on I am I got some stats up I've never watched the stats like frames dropped and all that I think it's doing okay the health of the stream got like a little command center on one side of the screen and then Wireshark on the other side of the screen having a good time um so you guys probably know about shark fest I've talked about it I hope you guys will go one day I've been wearing a different shark fest shirt each video this one is uh really probably one of the better ones I mean in terms of style right it's beautiful it's an instructor shirt I don't remember what year maybe maybe 2017 but um yeah we'll talk about shark fest more cuz there's breaking news if you haven't seen its virtual this year got cancelled it was gonna be in Kansas City in Gerald's hometown like right about now I think I think it was literally gonna be this next week I could be wrong but it was around this time it's been canceled for something I'm not sure what there's something going on in the world I'm not sure exactly what it is tribbles so it's moving to October and it is virtual I'll bring up the the website when we switch to the screen mode just so you'll see you can go and you can register I think it's free don't quote me but I think it's free Lynn Levin Modelo love all those choices those are beautiful choices there are courses that are in the days leading up to shark fest those will be paid because those are like smaller groups instructor-led deep dives you know like training classes not so much as a conference right those you got to pay for but I think the conference is online I'm gonna present some case studies I think it's gonna be free don't quote me on that so if you guys missed it I put out a new video my first video I'm a little embarrassed to say I've been distracted had things to do an actual Wireshark video maybe the first one in four years I think the previous one before that was packet bob happy little packets and then I did a shart Wireshark infomercial which is there's nothing useful in that it's just for your entertainment value and this is the first one that I put out I was trying to connect this new PC that I talked about to a really old readiness Netgear it just gave me a crappy error so it's simple simple fix it's not a great fix because you have to enable smb1 on Windows 10 but at least again the whole point of my videos is not like here's the solution a to be it's here's the process that I use to solve it and that's to me that's I hope that's the value that you get out of what I do so other things I wanted to mention I dug out some books that I have you guys might recognize some of them this is Stephens Volume one tcpi Ellis traited I don't know what Edition it is there's some debate around you know you know the various editions and how they're updated but you know it's a good bedtime reading of course this was heavier this is um volume two you get into like code level stuff honestly I don't think I've spent much time on this one but I think I bought it how did I get this I think I bought it cheap off maybe eBay because it's got a weird sticker on a complements of something marketing but you know the first one the first one I would definitely recommend to read and then I have I think Chris Sanders presented at a shark fest and I think maybe these went out oops sorry these went out to maybe the instructors this is the covers Wireshark 2.0 it is the third edition but he puts out good stuff the other two I have I think this one's pretty good this is Lara's troubleshooting the wash art book you know it's not it's not quite the Bible that this one is and this is an older version I think this is this is more like a study guide for the certification anyway these books are collecting dust I mean I thought maybe in a future one over the next however many we could do like a giveaway for but you have to be present to win and I don't know exactly how that would work you know to randomly select a Minh oh if you know a good way to do it like live select somebody who's in the chat or a watching that would be cool otherwise we can figure something else out but that could be fun okay I don't wanna spend too much time on the ramble chitchat let's go here so I mentioned shark fest this is a new website shark fest virtual wire shock dot org where you can go I guess and register if let me know if it's not free but I think it what else so what we're going to talk about today ladies and gentlemen boys and girls the last couple we've talked about like applications that were broken right we talked about it was the first one it was um streaming right it was streaming it see I've lost all sense of time because every day is the same like Groundhog Day and I've lost the ability to tell yeah so it was it was a to vendors you know talking to each other you know setting up new stories for a television station and you know it was broken and who's at fault right that's what we figured out second one was call of duty home network why isn't it why is it it disconnect you know and we figured that's broken but then we get to performance right where it's not broken we're just not getting the expected result that we want and this is where to me so much of the pain lies with finger-pointing you know is it the application is it's the server the network the band you know provider so many things in like the network what does that mean and the thing that sort of falls in between the cracks is TCP well who owns TCP right I mean that's just something that's riding on the network and it's behavior so if you're a strictly a network engineer network person you could arguably say if there's a TCP problem with your application or your server it's not my problem all right you could argue that if you're a server guys server admin and you're spinning up servers and installing applications do you want TCP if it's not working well what if it's working locally but then you migrate to the cloud and now the performance is bad is that the application developer is is that the the admin I don't know what do you guys think what is your experience I I think in big companies where there's lots of division of labor and you have teams focused on one thing firewall team you know the network infrastructure team the performance team this team then yeah I bet it's hard right to say who owns what and there's maybe lots of finger-pointing if you're a smaller shop you know right maybe you own everything or maybe there's my first job was just like the UNIX and network guys and the windows and network nobody'll know we're guys that's how long ago that was those were the two main teams he had DBAs and whatnot but those were the two main teams so we you know well we could work together if there were problems my opinion is it behooves everyone to understand the basics of what what is going on in the wire if you're a application developer if you're a server guy if you're a network eye it helps out so much understand what's happening and be able to pinpoint the problem and move on give it to the right people and stop pointing at each other have some data determine what the problem is right and that's why I think this is applicable for anyone not just network folks who is usually the people who are going to check this out so kudos to you right potentially this is TCP is not your problem boat but it helps you to be able to know TCP and be able to say hey this is a problem this is what's going on so to that end there's honestly this could be you know a series of of live streams or videos or whatever maybe it will be my initial thought was you know actually Fidel let me know about an FTP problem that turned out to be you know if you upload it slow but you download it's fast quote fast and he ended up figuring out that it was the receive buffer or the receive window on the server because normally the FTP servers you know I guess you think about downloading but uploading to it it was too small you bump it up problem solved it's pretty straightforward and we can look at that that's kind of like a 101 receive window is like a TCP performance of 101 but guess what we're just going to dive right in and maybe we'll pick apart some of these in future videos so what this is is actually the download that was fast but is it good enough I mean if no one's complaining I guess it is so we're not necessarily going to be going to like solve a problem but we're gonna start pulling apart what's actually happening what is the performance if someone says here's a peak app what is the performance you're getting because if you're looking at your FTP client or an application is telling you a throughput that's the application throughput right that's not the network throughput those are different things so what we're going to look at that that's my opinion on why you should learn this stuff you know people that just it's not my problem and push it away I've run across plenty of people you've run across plenty of people maybe you're one of those people in my experience the people who stand out to me who make an impression on me who I would hire are the people who just they want to solve the problem because they like solving problems that was me right if I can solve the problem I want to solve the problem I want to be the guy give me all the glory I you know at some point if I can't all right if I say hi this is a problem over there and I there's nothing I can do about it then let's bring those people in or hand it off but you know just pushing it back and forth I think it's always better just to dive in okay I think what we'll do is we've talked about TCP I've got some notes so let's just have a look so um this is a download peak app I have not put them up to download I apologize maybe I will after the fact and when it becomes a recording video I can put it in the description but these are not my peak apps I did scramble the IPS not that it matters it's it's you know private IPS and clearly it's just a test FileZilla thing but I left the data in because these are I forget how many bytes 300 megabyte pcap it was a little bit slower to filter and I was going to just cut off the TCP data but then it limits some of our reporting visibility so I just left it in alright so what is it what is what his fast performance look like right I mean ideally you know what the bandwidth you're working with is is it's a hundred megabits or 500 megabits or 10 megabits at least ideally you know that maybe you don't so just right off the bat you know the overview the file properties is gonna tell you quite a bit about what's going on here we can see that it's 44 seconds long we've got almost 400,000 packets we've got yet 44 and a half seconds almost 9,000 packets per second if we come down to the bottom we're getting about 72 megabits per second that's over the entire entire capture right so captured on the Left displayed in the middle I haven't applied any display filters but remember that that could be useful later so every packet in here over the tire thing it's just simple math right how many seconds how many bits how many bytes you get 72 megabits on average across it so okay that's that's not bad I suppose depending on what you're dealing with you can also look at conversations to get a view as well if we look at TCP there's only three there's two port 21 which you probably know is you know FTP control port right and then there's some random high numbered port with about 400 megabytes so there's your data transfer and this tells you for this one TCP connection right again just the whole thing TCP connection we're looking at about 84 megabits per second right so that's better than the average of the whole thing which is what was it 72 something like that so eighty four megabits for the one actual download connection but again does that count just TCP I don't know now that I think about it at the IP IP view of it there's only two hosts talking so we only get one flow here if you know between these two hosts yeah it's different so this is another way that you can look at did Frank teach of all this hello Greg you know Greg and I work together a long time ago Frank was our awesome boss our IP I wish I'd known a lot of this back then it would have been useful to be honest but it didn't I learned learned so another place you can look is protocol hierarchy and I think this one's interesting because you kind of kind of breaks it down for you so at the frame level right all the bits that we can see we're looking again it's 72 megabits right that's what we saw that's what we saw in the summary that's everything from the frame layer down if we come down to Ethernet well you know that's just a header right if you bytes header well that's right about a Meg for that overhead and then you have IP and you have throughput for TCP then you have the throughput for TCP data at 68 megabits per second so you could say that the oh so that's the application throughput right you have to think about again the different layers in the stack and then when you're talking about if you step back big picture and think about network throughput what you're probably talking about everything overhead everything how much can i push through this pipe I don't care what it is how much can i push but then when you're talking about application performance FTP HTTP well the through but it's different right because you have overhead so in this case we've got like at the TCP layer there's two two Meg overhead above it if you come down a layer under TCP to the application protocol it's two more Meg so the TCP headers account for two megabits per second of overhead right thanks so now we kind of have like a view of what what performers were seeing and we still don't know like necessarily if it's good or bad now if I didn't know what the bandwidth was of this but if I knew let's say it was a hundred megabits is this good well maybe what else is happening am I the only person talking is Imagi site with hundreds of users sharing the bandwidth you know these are things to think about is there QoS in place for this traffic now I want you to gather yourself for a minute I'm gonna do something I don't think I've ever done on a video maybe once maybe one short tiny video at the very beginning I use this tool it's not cuz it's not a good tool I just it's not in my normal workflow and that's shame on me i o graph i bet you guys use it a ton cuz it's a good tool right so the default view of whatever version I'm using 3 to 4 is graphing all packets and all TCP errors are TCP analysis not flags right that's if you click down and remember last live stream when I didn't click the button in the lower left corner and that would have told me the problem with the call of duty duplicate IP so right away and I didn't do that that's what that is telling you right well that's expert analysis flags but this is TCP analysis flags which is a subset so strictly about TCP analysis flags it's things Wireshark wants to tell you about you know what it's if it thinks it sees dewbacks or retransmissions or things 0 window things like that so we clearly see that it starts to ramp up it drops and then rule which shoots way up here and then it drops off and then it's like a rollercoaster ride and what do we see in the dips TCP analysis flags so that's like super useful right away we know ok clearly there is something happening on the network that is impacting our performance so I think this is useful if you want to understand the actual throughput let me turn these off I already had this before this is again looking at just the data throughput right graphing and I'm graphing bits so then you can easily zoom out and you can say okay it literally flat tops I mean it is going up like a rocket and then it goes and flattens out right around I don't know what is that 115 to 120 117 ish just flattens out it stays there for we can eyeball it about these are two-second intervals you know about 4 seconds right and then it drops off and we know if if I click this and graph it it won't be helpful because the numbers are too crazy you'd have to you can turn it on but I mean it's not that useful unless you turn on log scale and then it really flattens out your graph but you can kind of see it right you can see these little dips and you can see that those were the air show up so from this it looks to me like police we're hitting in the neighborhood of 120 megabits per second is the tippy-top so potentially that's what the throughput is and then I did I asked Fidel after the fact hey it looks to be about 120 or so what does easy yep that's it exactly so this tells you that we are hitting the max but then we have something happened we back off we ramped up we back off and then the average becomes around 70 megabits right so that's that's our performance I am I gonna use this again later I don't remember I'm going to close it so there are like file properties conversation protocol hierarchy IO graph there are four ways to look at the performance that you're getting in your P cap use your favorite one so now let's look at some packets this is thirsty work you so we have through a handshake always always always always always always always get to through a handshake why do we want three hinging one I like to look through a handshake to see what the round trip is to see where I'm capturing this from did I capture this on the client on the server in between well the client on FTP initiates the connection to a FTP server on port 21 which is exactly what we see here with the sin or synchronize flag set so if you think about it oh I added a downloader to look send flag okay it's not that cool the client sitting there decides I'm going to initiate an FTP connection so at time zero right over here at our time offset time zero the sin goes out now it has to travel wherever it's going could be next door could be across the world to the server the server receives that sin and then it immediately generally responds with a syn ACK and that syn ACK has to travel back to the client so if we see a delay between the client and the server I'm sorry between the sin and the sin ACK we know that we're capturing on the client if you see and then immediately after receiving the Cermak I should be looking at the delta immediately after receiving this sin act the client completes the three-way handshake with an ACK and that took you microseconds if we were capturing on the server side which I have will probably look at I should have open Wireshark for that the delay is between the cynic and the AK because I'm a server I'm minding my business people are going to talk to me but I'm just waiting around for people to talk to me at times zero thus in shows up out of the blue and I immediately probably within microseconds reply with a cynic now the cynic is the one that's travelling wherever it's got to go to the client and the client responds with an AK and the AG has to travel back round-trip time so that's where the delay is now the other reason why it's imperative is that you get information that you don't get otherwise in the TCP options we know that the maximum segment size is 14 60 bytes the segment size is the amount of TCP data that can be put into a packet into a I could frankwit a packet IP packet that probably means with overhead of IP of 20 bytes and or overhead of at least overhead of TCP for 20 bytes as 40 bytes so we're looking at an MTU of 1500 bytes for this host on this interface no op is just like you know these things have to bite aligned so that's just a space it's a nothing window scale again if you want to look at window problems in the host and most hosts use window scaling this is the only place at scene one time no three we handshake so that it can multiply by 256 every single receive window that is seen on the wire in this connection after this write in sacrament it's selective acknowledgement acknowledgments are permitted by this host the response same MSS same window scale sack permitted okay so now we know what's going on there we have a round-trip time of 65 milliseconds to our server and that's one right then we have another one here 65 milliseconds it was four seconds four and a half seconds later do we care probably not because this was just logging on and doing nothing and then we log on again and yeah it's 65 milliseconds round-trip time and look you can also if I click on a packet if I go to where do I go sequence sequence akan alysus it puts in the round the initial round-trip time scene for this connection it puts it in like I think every packet every every packet so that's good but again it has to see this through a handshake to figure that out no I don't know anything about IP VIP v6 to be honest Greg can you please learn and I'll bring you on as a guest and we'll talk about ipv6 and you can teach us because I don't know ipv6 to be honest with you so that is the round-trip time so then we have yeah yeah we log on we enter passive mode here's the command that I'm not going to tell you how this works you probably know if you don't google it how that we use this information to get the IP and the high port number to then connect to and we're retrieving a test file jumping in about TC ipv6 so then we have the three handshake will just double check it's still right about 64 6000 65 milliseconds and then here comes data so what we'll do first and this is sometimes a good idea is just kind of scroll through and see what we see human brains we like to think we're logical and you know ruled by ration and logic and it's just not true but one thing we are good at is pattern recognition our brains are always looking for patterns and trying to fit some experience in some information into something we've already seen our brains our natural pattern machines so I just like to scroll through or click through and just look and see what I'm seeing does it all look kind of the same pretty much scrolling through this is let me actually scroll it might be faster I don't think it is now we see something's coming up so there's some stuff that happened so we know we have something which we from our IO graph we kind of knew right um come back to the we'll come back to that so what we see we've definitely a pattern here right we see this 1460 14 60 0 14 60 14 60 0 and what this is is FTP data as it says data data acknowledgement data data acknowledgement and that is the typical tcp bulk data transfer behavior it's right down here on the floor in the stevens book tcp will acknowledge every other packet i think it's supposed to be every other full-size packet but the the the front the TCP stacks might do it slightly differently but generally it will acknowledge every other packet so data data ACK data data ACK is TCP doing what it is supposed to be doing right and because this is just a one-way data transfer I mean TCP is duplex but this is a download stream right the application is FTP we have a file of I think you know what it's probably up in the control protocol it'll probably tell you and here somewhere like how much data it is but you can also again go back to one of these things like a look at our protocol hierarchy UNIX is every two full windows will act every two if not there you go you know we transferred yeah around 400 megabytes right is about the size of the file so let's go to the tool that do use the most and because it's streaming where I was going with that the application has a file of 400 megabytes it is taking chunks of up data and shoving it to giving it to the TCP stack could say here take care of this and it might be giving it in small chunks big chunks it depends on how the application is written let's just say it's giving it in 1 megabyte chunks to TCP well TCP can't send a 1 megabyte packet it's TCPS job to take stream of data and break it up into segments TCP segments put it on the wire right so that's what we were just it's just a stream of data it's one layer up from TCP we've got a stream of 400 Meg stream down a one layer down or up down you know it's breaking it up into these fourteen sixty byte chunks now because this is a stream it's unidirectional we're gonna look at a stream graph and so you need to click on a packet that's in the dirt in the flow of the data so I'm clicking on a server packet and I'm going statistics TCP stream graph TCP trace you and what we're looking at is simply bytes over time the x-axis is time as in seconds y axis is bytes well it's sequence number but this is how t CP keeps track again stream oriented data transfer protocol the sequence number is how it determines where we are you know if I've sent 100 bytes a thousand bytes if I need to resend certain chunks of it the sequence number increases one sequence number for every one byte that's sent it's a one-to-one mapping so basically sequence number is bytes so up and to the right is good that's what we want we want as time progresses we want the counter to go up the steeper the line the faster the throughput right so generally you want to see a smooth line going up into the right and this the the slope of the line right would be the bandwidth the throughput so looking at this we see a few things right we see it's start out a certain manner a little flatter it ramps up and then it does the undulating sort of this wavy motion which is just a different representation than what we already saw with the i/o graph right so if it starts to flatten out that means it's less throughput and as it ramps up and gets deeper that's more throughput so what we can see here is that the throughput goes up but then comes down it goes up and comes down which is exactly what we saw in the i/o graph let's start at the beginning of the connection you and what we have really have to zoom and don't you just the first one I think this is the first one first set of packets we see we zoom in and these little guys these little guys are individual packets their lengths represent the amount of data that is in that packet so if it's a long line it's a lot of data if it's a short line and you have to look at the you know look at the y-axis to determine right we know these are 1460 right so that these are individually packets and when this green line goes up to meet the line at the same level that means these packets have been act there's a little bit of a space two more packets come in and the green line goes up which means these packets have now been act so at this point there's no outstanding data for packets came in they were all acknowledged give me my thing back then we zoom out and there's a big flat spot here we zoom in one two three four five six seven eight which is double what we had before right and you can see the green line going up meaning all these packets are now acknowledged and this pattern repeats itself another flat spot which means time passed how much time is that from like 0.1 the two huh point two one two point two seven is it maybe about 65 milliseconds this is round-trip time right so this is how t CP behaves in the beginning of a connection you may have heard of slow start we the beginning of a connection we have something called the congestion window sorry maybe you've heard it called the sin window this is what determines how much data can be sent in one go so at the beginning of a connection the number may be small so we're gonna send in this case it looks like four four packets went out the door right and then we wait did they oh my gosh did they make it I hope they made it if they come home safe right and when we get the acknowledgments for those four we increase the window the congestion window the CIN window we can send more and so we're gonna in this case it doubled right you've heard you know slow start typically has that kind of behavior so it increases for every akka ckets it increases the congestion window we acknowledged for segments we increased it by four behavior the exact numbers when you drill into these it's not always perfect but that's a guideline right the point is is going to sort of kinda double each round trip up to a certain point so you can see that this little set this little area of packets good that was not what I wanted this little area this one is longer and we go up and then it's even longer and it's even longer so our throughput is increasing with every single round-trip this is normal TCB behavior we're gonna back out and then something happens right in here right we have these all these black black red lines are sack blocks meaning we had packet loss so let's and the cool thing about this is that you can drill in wherever you click right there it's going to go in the capture so let's have a look so everything was fine man we were ramping up slow we're coming maybe starting to come out of slow start and we had a problem so I generally like to when I'm looking at TCP I didn't really like to have a couple more columns these guys I don't want to crowd the screen too much these are the sequence numbers sequence number next sequence number act number right so what we can do is say alright we're chugging along here's our sequence number 26 13 we sent we know we sent 14 60 bytes of data you add that together you get this next number 80 407 3 is that what we expect to be the next sequence number if we come down to the next packet it sends sure enough it's 4 0 7 3 the next one we expect to see 5 5 3 3 we come to the next packet sure enough 5 5 3 3 the next one we Specht expect to see is 6 9 9 3 and the next one we see is 8 4 5 3 I don't match previous segment not captured right how many packets did we how many packets did we lose well we can do the math you can subtract this sequence number from what was expected and you're gonna get 14 60 times 2 because I did it already I think I did it on this one it doesn't matter but you do that math divide it by 14 60 it's going to tell you how many packets were lost and in one of the cases I looked at it was two packets we lost two packets so what happens when we lose a packet so if you look at the acknowledgments these two packets came in and they're the client said okay here's my Act number 1 1 5 3 which means I've received all the data you've sent me up to and including 1 1 5 2 and the next sequence number I expect to receive is 1 1 5 3 what is what do we get 1 1 5 3 all good we come to this 1 4 0 7 3 so it's acknowledging all the data up to and including 4:07 2 and the next so I expect to see 4 0 7 3 next and that's what we get when we come here again these that you see this little sort of stare stop this matches this matches this this matches well this one matches this one matches this one so 6 9 9 3 will we get 8 4 5 3 they don't match so we received a packet and we immediately act and say no I want six nine nine three you know we were acting every other packet when we get an out-of-order packet we will a kit every single time we don't delay so we say no no I want in six nine nine three all those red lines on the graph are called sack blocks so we did receive a packet right we received this packet and if you go down here you can see TCP sac saying hey I received eight four five three up through nine nine one three which exactly matches this one packet here so we're saying hey sender I need six nine nine three but I got this one well the next one shows up now it's an order relative to the one that was received right nine nine one three we got nine nine one three but it's still not the one we want it so we again say hey look I hate to be annoying but I actually really want six nine nine three I don't have that one but if you look at the ACK now the sak block this is called the right edge left edge and right edge the right edge has now increased to include this one too this is called selective acknowledgments so we're saying hey yeah I got that one but I still need this one this happens for every single packet if we scroll through here right you guys catch something right there pattern watch see that again your eyes will learn to just pick these things up what changed this one little thing right here well other than obviously these numbers that are changing but my eye saw this that's what my eye was drawn to it's we sent a different size packet we've been sending 1460 this whole time now we got 648 experience tells me this is probably just a push flag and yes it is meaning it's the end of a buffer no big deal we're gonna not we'll follow that away but we're not going to worry about it so for every packet we get when that is not the one we want we're going to immediately act and increase the right edge saying hey I got I got all these packets I got of these packets eventually come down how many do we get 171 duplicate ACKs and we get a fast retransmission there it is six nine nine three hallelujah hooray we got the one we wanted and now we continue on as we were why are there so many duplicate ACKs is a question I hear a lot to answer this question I will answer in the form of a monologue from my favorite play we have to think about perspective we're capturing from the client right so every time packets come in we get we react right away two packets come in send out an ACK two packets come in send out an ACK on the server side we sent some data and we were waiting if we go back to our graph let me zoom out this line of packets altogether guess what they were all sent at the same time boom this packet train went out out the door it sent nothing we can do about it it's gone now it takes what half of 65 surround trip time is 65 milliseconds right so it's 30 something 32 and a half milliseconds to go from the sender to the receiver and these packets start showing up well one or two got lost along the way sorry the rest of the train cars are coming in can't stop it so as they come in the behavior of TCP is to act every out of order a packet and if we lose one or two technically because it's a stream oriented protocol right every packet after that is technically out of order now what happens is the sender so these acts so this data comes into the client and the the clients start sending these acts back so it takes a full round trip time for all those acts to come back to the sender once the sender receives the third third duplicate ACK it will go oh we lost a packet let me resend the one at once and it just reasons the one packet right we can probably even spot it because we know it was lost right here right look there's a gap all these guys touch each other there's a gap right here so I mean we can probably come over here and we'll see it you somewhere there is okay so that is the explanation of the retransmits sacks etc now what happens what happened when that happened you saw how we were increasing every round trip every round trip we're sending more and more packets our congestion window remember that's a something you cannot see a congestion window in the header it's not it's not even Wireshark doesn't even like calculate that for you it's not something that's no it's only known like in memory in in-state on the cinder there's tools you can use to that it'll it'll report it for active TCP connections but it's not something you can go look for you just kind of have to count the packets and figure out congestion window is usually it's counted an MSS so if it's 10 that means 10 MSS 10 packets well look at now look at what's happening between the round-trip times we're sending less and less packets that's TCP backing off it's doing what it's supposed to do we hit a limit in the network we don't drop packets well all right hold up back off but slow down everyone take a second how we doing and we sent again we start again we start increasing the amount of data we're sending per packet until we hit right in here and we really we really take off if I had to guess this is like still sort of like slow start behavior and there is a certain threshold of congestion window when TCP will switch state and go into I forget what it's called someone probably knows but now we're just really cranking along right so we know another way you can look sometimes I look at these and I go okay well what's what is it throughput so we hit another one up here right we hit another packet loss scenario right here sometimes I want to know what was the throughput before we before we hit that packet loss maybe that's informative for us so what I like to do is zoom in you know just right somewhere right in before it click on a packet before the packet loss let me be sure I'll click on that packet I go to that packet I get the frame number and I do frame number prepare filter selected oops selected and because I was at the high end I'm gonna do less than or equal right I'm going to go back and let's just pick a point down here right where it looks pretty smooth this is moving really fast and we'll click on that and we go to that packet and we get the frame number and we do prepare and and this we're going to do greater than right because we're at the bottom of the range and now we're only viewing that little segment of packets in that steep the steepest part of the line and we can go back to one of our tools even just like kalfa the the capture of file properties and we can see the overall throughput now not the application throughput remember the overall throughput all the packet all the bits and bytes and bubbles 117 megabits so if it's a hundred and twenty megabit link right and we're sharing it with other people and we're maxing it out yeah we might have some packet loss so I want to start to bring this home we've talked about talk about bikes in flight right that's the packet rain bytes go out the door they're gone they're in flight and you can see it in the capture however again this is where this is where your perspective matters so it is under I think it's under here too if we go you know bytes in flight we will apply as column because we're on the receiver side and we're acting as soon as the packets come in every - well we don't get above two packets of bytes in flight right if we get beyond that well then we're falling behind we're not processing the packets fast enough potentially if we were to look at this on the server side we would see the bison flight you know go up dramatically okay why don't we look at the server side we'll wrap this up and look at the server side which is this one so as mentioned this is the cynic through a handshake and you see that the Delta here is between the cynic and the act so or on the server side you scroll down hi these I don't think we need them I'll leave them for now you might notice some differences with this one let's put this guy say in the center and maybe it'll make it easier we don't see 1460 here we see much larger numbers so we know that I mean is this some kind of crazy jumbo frame no this is segmentation offloading right feature is enabled on this host that we're saving CPU cycles to do the actual breaking this stream data into MSS I segments we're handle we're giving that to the NIC the NIC is doing that for us and it's going off on the wire this is why it's important if you can to get a span or get a tap because if you're in some cases you're you're not seeing what's really on the wire if we go back to the for example you know sometimes slow start starts with just one or two segments but we September on the client side we saw four well let's find so here's the first data packet from the server and it's four MSS so it sent one which then wears broken up into four on the wire and it waited a full round-trip for the act to come in before sending the next one and it sends two and then it weights a round trip so it's sending one weight to wait but remember these are form SS each so it is still doing TCP like behavior but you know honestly it gets really can be really challenging like when you're looking trying to understand TC behavior on us on a host that's using segmentation offload if we go to you can also do like you know TCP people like to do TCP analysis flags sometimes right off the bat and look so this is that first that first drop we had here I'll just kill the filter you so we sent 32k another 32k at this point we have you know 262 K bytes in flight right those are packets out the door waiting to be acknowledged unacknowledged data and then these acknowledgments start coming in and we see you know ack ack ack and again these are acting what every two packets right because remember these are these big these big boys are broken up into MSS sized packets and the client is acting everyone so these acts represent two MSS sized acts packets payloads segments acronyms synonyms and then we see the the six nine the one we saw on the client-side and then we see the dewbacks now it's supposed to see three of these boys and send the fast retransmission you know you know that's that's ideal that's what it's supposed to do in this case we saw five before the fast retransmission well they all came in really quick right they all came in within microseconds of each other sometimes you're just gonna see that is that a problem probably not I don't think I would spend too much time worrying about it but the sinner goes oh I've got multiple dewbacks three five whatever I'm I need to I need to send this back I'm not gonna wait for the timeout because every every segment that it sends every data that it sends gets a timer right it sets a clock when that clock expires if I have not received an acknowledgment in that time I need to resend it that's called a timeout that's usually much further apart fast retransmission we don't wait like oh you told me something you told me you lost this I'm gonna send it right away it's supposed to result in better performance I have seen this vary based on the stack and operating system I've heard people say there's no penalty for a fast retransmission we know that's not true we saw the data it was doing this it dropped every time there was a fast retransmission that behavior is can is controlled by the congestion control algorithm on the cinder this is Windows 2012 r2 I think I googled it and now I don't remember maybe you guys know what the the default thing is I looked on this this is a Windows 10 right you can run this command net SH and TCP show global and it gives you a lot of good information so my initial round-trip time out which I was just talking about is a thousand milliseconds right that's how it starts there are some stuff in here I don't know what they are we could google them but there is supplemental and my congestion control provider is cubic um some there's compound TCP that's I think used on Windows servers there's DC TCP that can be then there's older versions I forget what the older one was that would show up here sometimes my point is I have seen say Linux for example which uses a another flavor the penalty for a fast retransmission is very small clearly on this Windows host it is so let's go back to the question let's wrap this up is there a problem well we know that we're while we're hitting max throughput that we can expect it's not staying there and the average is in the neighborhood of 70 megabits per second is that except what is anyone complaining I don't know haha I don't know but he's someone complaining do we need to look at it is is the link over subscribed do we have QoS in place that's quite can we improve it if we wanted to well maybe can we use QoS can we look for drops in our network can we change the congestion control algorithm on the host right because it's working and it's fast compared to the upload the upload was the problem it was crawling no one was complaining about download but we we we dug in deep and we found out actually we could improve performance here potentially depending on what the environment is like there was a there was a googled I thought this was interesting where was it this is from a book right this is specifically the version of Windows and I don't know what this is I tried to google it and I didn't find much information but frankly I didn't spend a ton of time modified fast recovery algorithm maybe that's a setting we can turn on that will help this perform better I don't know but but now that you know what the issue is maybe there's something you can do to address it or maybe it's just what it is okay so if if this was over your head I would recommend you come to my website go to read not over your head but if you if you want to start if you want a more structured approach to this like because I'm you know this is live it's all over the place I have one called understanding throughput and TCP windows it will take you like through each thing you need to worry about when it comes to performance the receive window this in with the same window congestion window the syn buffer and then I think and then I have a video that walks you through oh this is the clan one yeah it will walk you through that will focus on this stuff I think different you know showing you how each one can affect the throughput so so do that while you're here you can get a Wireshark little course emailed to you if you sign up I will send that to you there's a lot more we can do with TCP guys this is please let me know send me an email send me a jump on disk or if you want I got a little discord server it's in the description since I'm sitting here in front of my computer all day with work you know I can usually respond fairly quickly I'm happy to discuss with you guys about more about this stuff so we're yeah we are at just over an hour let me know what you want to see next time we can do more TCP I've got some more I don't know other stuff I am looking for SMB case studies if you have SMB questions problems performance let me work with you I'd like to help you out take a look looking for some SMB case studies to you so with that said thank you for joining I hope everyone has a great rest of your week and weekend and you know what we'll see you next time here on the packet bottom live stream experience I will hang out in the chat and see if I miss any questions and see you next time [Music] [Music] [Music] [Music] [Music] [Music]

Channel: PacketBomb

Views: 1,650

Keywords: wireshark, packet analysis, tcp, networking, packet loss, fast retransmission, dup ack, performance

Id: Mw7QsjuAjhY

Length: 71min 5sec (4265 seconds)

Published: Fri Jun 19 2020