TCP Fundamentals Part 1 // TCP/IP Explained with Wireshark

welcome to this session this is a two-part session as you know on your program we're going to start um start now all morning we're gonna go till lunch and we're going to be talking about TCP okay so this session is named TCP tips tricks and traces now intentionally the first session if you notice on your program it's a one fin so beginner level we want to go in and talk about some of the tcp fundamentals I noticed on some of the other sessions that even I attended sometimes the instructor might say hey you know it's just a window problem okay moving on and if we're new to TCP we're going would know what does that mean or no it's you know no big deal sack you know okay there's sac there's this bit there's that bit and it's like well it's rhyme time time back up hang on what is that what does that mean how does it work so we're not going to make any assumptions as far as your level of TCP experience and knowledge we're going to spend this first session one fin going through those fundamentals and making sure that we thoroughly understand them that way in the second part of this session when I start throwing some case files and/or I'm sorry some case studies and showing you some of the things that I've seen out there you can follow along and understand better what's happening okay so to just introduce myself you see my name there but okay my name is Chris career I worked for a company called packet pioneer so I basically do consulting using trace files in Wireshark and I also AM a Wireshark instructor so a lot of times people they'll be having a problem they'll find me through whatever means sometimes my youtube channel sometimes the website or different blogs that I do in the industry they'll have a problem it sounds kind of like what I wrote about and they'll contact me and say hey I got a problem can I send you a trace I just got a query last night for somebody in New York that's having a performance issue so I'll do my best to help them out and sometimes that'll turn into a Wireshark training class which is great I'll say hey that's awesome but how do I not call you again so the way I'm kind of training myself out of a job right with the training part but hey that's what I like to do I like to instruct as well as I mentioned my youtube channel go out there and check it out youtube.com slash pocket pioneer or if you just if you just search for Wireshark or filters or that kind of stuff you'll probably find it eventually what I try to do on my youtube channel is not do anything longer or much longer than 10 minutes because I mean let's face it I have a tough time sitting through a one hour or hour and a half full even the shark sessions it's just I have a hard time finding that time to dedicate my brain to what's really happening in that video so what I try to do out there is just show you throw I'll call them shark bites even though tomorrow we have shark bites technically but I try to show throw an idea at you here's a five-minute that's all we're gonna cover and then boom five minutes ten minutes so if any of you guys were out there have seen those videos please come from chat with me and one of the breaks or afterward I'm always interested in ideas or things that you might be seeing out there that would be useful to have a video for sometimes I create those videos on behalf of my customers so that they can better explain what's happening in their trace file using that video so for example I say oh you just have a receive window problem and instead of them needing to go to their boss and say hey we've got to receive window problem instead they can watch that five-minute video together and you know then I do the heavy lifting for you so anyway um I'm absolutely open to suggestions in that way as well okay what are we going to cover we're going to talk about why TCP is such a critical protocol why is it something that hey a lot of us are in here we're interested in TCP we're interested in how it works we're interested in how it delivers data we're interested in looking at one of those screens like we've seen in the last couple of days and the instructor stops and says okay you see the problems right there oh no maybe you like me when I first started coming to shark fest I'd sit in those sessions and I'm looking at the screen and I'm thinking on honestly I don't I don't see what's going on and they're just running through this trace and I'm just like wow okay maybe one day I'll get there well if that's how you feel sometimes I mean hey I feel that way too sometimes at these sessions you know it says especially when someone's moving through a trace really quickly so what I want to do today for you is to get you a bit more comfortable with what TCP does how it works some of those underpinning things that we should know to then be able to build on later okay so we're gonna be talking about the handshake options within the handshake TCP windows both receive and congestion we're going talking about retransmissions how those work selective acknowledgments okay so so some core things now you might know some of this pretty well you might know it cold but I think an important thing with TCP and one of the reasons why you see a lot of these instructors do what they can do here up on this stage or even with a trace file if they are sent a trace file and they can just like hand song he can just rip through and go 30-second you know he takes 30 seconds and bam I found the problem it's because he knows the fundamentals really really well he knows them cold right he can he knows how it's supposed to work that way when he sees a trace file that's struggling having some problems he can more quickly spot the problems in that trace okay so that's why we're doing this today we're gonna go over some of these fundamentals and then make sure we can really understand what they're doing again this is in session 1 as we get into session 2 we'll do a bit more with case studies also tips for isolating performance issues this is the TCP tricks part so creating a TCP profile in Wireshark how can we create a profile that can help us to to find some of these issues faster what filters can we set spotting delays and TCP streams so kind of handful of things there for you and of course case studies like I mentioned all right so first of all TCP why is it a big deal what's the what's the reason why we're all here why are we interested in TCP well important stuff uses it at least for now right so it's reliable it's connection oriented it's going to ensure that that data gets over there and we get a positive acknowledgment our stuff got there also TCP handles loss and congestion relatively well if TCP is smart it can figure out okay I just send some data on the wire I said it sent it at a certain rate I started to send some congestion I saw some loss let me back off a little bit on how much I'm sending it wants to figure out especially with the large data transfer it wants to figure out what's the best rate that I can send this at without running into problems so TCP is a smart protocol now how it does that and interpreting what we see on the screen that's what we're here to do today now today this I think the number one reason I get called anyway from my clients is because this stuff is slow right we all hear that I still haven't ever gotten a call from somebody telling me things are too fast this is probably why they point the finger at you it could be why you're here at shark fest no doubt this situation has led you to look at packets right as we're going to go on to talk about from a network engineering perspective a lot of times we just we just get the blame don't we and if we don't get that if if we are not to blame and a lot of times the onus is on us to show who is to blame right it used to be ten years ago we could just say oh it's not the network my stuff's great not anymore though right now we need to facilitate the next step of troubleshooting okay if it's not the network than what is it right so back to us back to network people by the way network people raise a hand if you are from the network side of the house hands up whoo okay server people application developers we got hey man raise up raise the hand buddy you're good two three four ish Wow notice who's coming to shark fest I mean it's great to have you awesome but yeah so Network people realize we're the ones that have the the feeds we were the people that pass the packets on where they're supposed to go right so we're responsible for collecting them capturing them and understanding them one time I was working with the one of my customers who grabbed the trace file he's a network guy right he's able to download Wireshark put it on a box he he gets the gets it in the path of packets which is a whole nother conversation gets in there grabs the trace file looks through doesn't find any retransmissions and he figures well my networks fine it's not me and then he emails that trace file to his application guy so I said okay what happened next I know you just sent it over this wall to your the other side of the house but what happened I mean what did he do with it he says he didn't know so he goes over there talks to the application guys so what do you do with it what you find in those packets would you find in that trace file applications guy I don't know what that stuff is that's all your stuff I deleted it pcap the heck is a pcap right so I was like well clearly that didn't have the desired result so a lot of times if we grab a trace and we can't quite figure it out or if we exonerate the network and then send that trace to somebody else sight unseen that's not gonna do a whole lot of good right Network people are here today that tells us something Network people were responsible for owning the packets collecting them capturing them and sorry understanding them now we come when a problem strikes okay OSI model on the right hopefully you have seen that OSI model before when it comes to network engineers if we're from the network side of the house which most of us are I grew up in the network side of the house if if a problem strikes our goal is basically and for all intents and purposes our responsibility is to exonerate the lower three layers we might put a little toe up into the transport layer with firewalls but really when it comes to those lower three layers of transport cables switches routers making sure data can get from there to there reliably and with low latency if they go down to HR and they pull your job description and there is a legitimate problem in those lower three layers that's me right hey if there's a problem with Wi-Fi if there's a problem with a switch fabric the infrastructure if I'm legitimately dropping traffic okay you got me it's the network so when a problem strikes think about what your knee-jerk reaction is what tools do you go to what how do you exonerate you're part of this whole thing maybe you go and get a throughput tool maybe you run some other tests you maybe you go over to some monitoring platform and you take a look for any and horrible indicators packet loss do you have an interface that's flagging drops discards so if we don't see that stuff we go up to our monitoring solution everything looks green it's SNMP based I got net flow over here I don't see high congestion I see low latency I see things are up looks good to me right then you tell them hey it's not my network well we still got the problem okay so we go to the other side of the house server peeps pull out their job description and let's just wrap those upper layers together let's call that just upper layer session presentation application there in the the server the app the virtual environment they go in there they check their resources they look at the make sure nothing spike no weird process no backups happening they go in there to their little world to be honest I don't have a ton of exposure into that world because I'm not a server app guy they go in there they check their stuff and they say hey resource looks good don't have any spike processes memories all right it's not me and what do they do well they dump it right back onto you this is a network problem my stuff's great do we notice a layer that neither side is really taking ownership of this whole conversation came because traveling around talking to people helping to troubleshoot issues this is what I see over and over and over again sidebar I'm not saying all problems are in the transport layer I'm not saying that what I am saying though neither side of this house I mean IT has become very siloed right you're this kind of expert that kind of expert your your app via network Wi-Fi you have all these fields of responsibility but where is the tcp guy [Applause] let me think about it your company is running mission-critical if anything happens these applications if they slow down drop can't connect we're in big trouble these applications are using TCP but who's the tcp guy i'll sit down in a conference room with a whole IT department so who owns TCP mmm no no hands like oh there's that redhead stepchild that no one really wants to grab and owned so the fact that you're here at shark fest and the fact that a session like this caught your attention means that that's now you you're the tcp guy or girl and you know what grab it own it live it breathe it become an expert at the transport layer why because you're gonna be the one that can fix some stuff you're gonna be the one that can answer the unanswerable you're gonna be the one that they go to like hey that guy got that thing real fast last time let's go back where are they where's the packet person right so for me personally in my career that's why I started focusing on the transport layer I started getting deeper and deeper into this protocol because that's what I saw it's like I go into environment you got the Microsoft guys you have the you know the app server people over here you have these like CCIE hotshot cisco guys over here and everyone's fighting and we can't figure out the problem and then these packet guys walk in and they're just like whoa boom there's the issue and it was a TCP thing so let inside your brain as far as what you are responsible as far as what you are responsible for within your environment if your physical Data Link Network let's reach up grab that transport layer and own it now where you're gonna run into one of the reasons why is network people we say you know what that's above me is because that tea P component and everything that it does is within devices that we don't own in many cases right where is the TCP stack it's inside that box there my network can't do much with it I don't have the responsibility for shaping it or changing it it doesn't it's not generated by in my network however to push forward as far as being analysts it's something that we want to own so if you find this in your your environment it could be this is something that we want to grab hold of okay so let's go ahead and start digging here first how does TCP start its conversation we've seen it so far today first thing we got to do is handshake all right we're going to spend some time talking about the TCP handshake what's in it what can we learn from it those first three packets which are so critical we've heard that from sessions previous to this one make sure to get the handshake make sure to get the handshake okay we're going to take a look at a couple of trace files that have some handshakes see what we can learn look at those options see what they mean and we're also going to look at a trace file that was missing a handshake and see what we are missing from that stream what can we not tell about that conversation because we missed the handshake okay that's where we're going with this now to put it simply for you this client wants to talk to that server it wants to get something from it it wants some piece of information before it can do that it's got to do a tan shake thing write three packets TCP syn syn ack ack and then it can go on and do whatever it was it wanted to do hey give me that picture give me that file give me that thing okay here you go so those three packet that's where we're going to focus now if this is review for you great if there's something you can contribute to what I'm doing up or hey I like this part of the handshake too cool but it's this is meant to be a review for you okay enough PowerPoint I'm I'm not a PowerPoint guy so in fact my entire this morning I opened up my presentation and I found that all every single image that you see on this presentation was missing it was gone just from the shark fin to everything you see so I had to rebuild it back there about an hour ago anyway okay let's go and go to our handshake so let me get set up here okay all right so first when it comes to Wireshark let me okay now something that I always need when someone else opens up their Wireshark as I need them to talk me through their screen just talk me through what you got here talk me through your profile so I'm not gonna assume you've seen all these columns and all that stuff but basically so for me I like to create can't see it down here it says TCP plain this is just my plain-jane default TCP profile as you can see it's not that fancy it doesn't have a lot of extra information in it if I need that all that other stuff cumulative bytes calculate a window size bytes in-flight up up up up up up up if I want all that stuff I can flip my profile but at first sometimes that stuff makes me dizzy all right and especially when I'm looking at somebody else's screen you know I see what all the columns they have and I'm like whoa okay hang on a second that's also for me sidebar sometimes in my classes people say oh you know show us your profiles like what kind of profiles you have and so I'll bring up my profiles know there's all my profiles how can we have those I'd be great know you can't why like Hanson said yesterday this is my way and in my world and Chris land it's my way or the highway right this is how I analyze my setup my configuration unless you built all of those profiles how do you know what they do right so that's for me I tell my students in my classes don't use anybody else's profile build your own because then you know exactly what you did right so for me I've got TCP plane I've got TCP Advanced I got TCP windows depending on what I'm troubleshooting I'll use one of those at least first right and then we'll go off into the specific protocol stuff now let me tell you a little bit about TCP plain now for me one thing that I like to do that's not default for me I just I used to hate the colors so I just turned them off but I started getting used to them and I they grew on me but you'll notice one thing for me that I like to do is I like to paint the sins green see up at the top so one of the first things I do go into my coloring rules I create a new coloring rule paint it bright green tcp dot flags dots in equals equals 1 okay that'll paint all of your sins one i want to paint them green I want to see them I want them to jump out at me that's a personal preference also as you've heard in other sessions Delta time okay so I like to add a delta time column I like to do that almost for every single profile I have hopefully that's something that you have in your profile as well now to quickly do that I know we saw that yesterday I just want to make sure I don't want to leave anybody out or behind or anything I'm just going to go to my preferences I'm gonna go to my columns I just like to manually add it when it comes to Delta time so I add a column called Delta Delta time displayed okay I move it up next to the time column that I can then change to running total or time of day or whatever but that delta time is an important one I also I'm not going to talk about this right now in fact I'm just gonna kind of close it up a little bit this up here it says time since previous frame in this TCP stream now for this trace file this is one TCP stream so that column and the Delta column are the same alright so you might be thinking what Chris why do you have a redundant it's the same thing because I'm gonna get there I'm gonna show you guys why you would want to add another time column and how that's useful for you in spotting delays but okay it's gonna kind of close that one up for now okay so let's do a little bit of digging so that's my basic basic TCP plain setup I also do have some buttons up here bad TCP no broadcast chatter slow slow HTTP I've got a handful of other ones but basically those are my quick filter buttons okay so sin let's take a look at this handshake and let's see what we can learn from it okay so the client has initiated a sin to the server this is port 80 let's go ahead and take a look at what we can learn now hopefully everybody can see this I'm gonna blow it up just a little bit more alright okay layer two and a jump it layer three for now I'm going to jump it we see our IPS let's just take a look at TCP and what we can find here all right source port 61 330 high client side port number dynamic client side ephemeral port I'm not too worried about that right now it's just a high number port great destination port the guy I'm gonna talk to stream index now Wireshark does a nice little thing you notice maybe you see heard this in a different session but anytime you see a value that is in brackets those hard brackets that has a Wireshark value that it done that's not part of the data it's something Wireshark does to help us interpret what we're seeing now we see here stream index 0 so this is the first stream of any kind that we see in this trace file if I saw a sin and a sin right after it on a different port number that'd be stream 0 stream 1 right so you can quickly spot how many how many different TCP streams do I have in this trace I mean go up and look at conversations if we want to TCP segment length how much data is in this packet that I'm sending as we see none sequence number 0 we're going to talk about sequence numbers and how to run through that and how sequence numbers work but basically what Wireshark does is it will start this sequence number off at 0 for me if I select that field let me go to sequence number I'm going to come over here to my hex so a sequence number just to boil it down a sequence number is a number that increments as I send data okay if my sequence number goes up by 1 it's because I've sent one byte of data all right if it goes up by a hundred then I've sent a hundred bytes of data I've got a positive acknowledgment back for that data it allows TCP to figure out what actually got there where am I starting from in this in the packet train as it leaves what's missing how do I fill that back in that's because every byte that I send has a sequence number assignment if you will we're gonna look at how to count that and how that works in a another trace file but Wireshark figures well by default as human beings we have a better time starting off at the number zero then we do some long crazy for byte number if I select the sequence number value here look at my hex that's actually a four byte number six 5b 494 0e is that zero no that's an actual big long scary-looking number but as a human being I have a tough time making that number my starting point right I like things to start at zero and then it's easy to count up and I can see how much data went and how much has been act for that reason by default Wireshark will do relative sequence number meaning don't worry about that long crazy number Chris I got you this is just gonna start at what we'll call zero now that's something that I can take off if I would if I want to see that original starting value there that's a TCP setting I can always bring it back if I choose I just right-click TCP come into protocol prefs and then I can come down and I can uncheck relative sequence numbers that'll make that scary number come back now some people would say that's the one I want to see boom for me a it's a long scary value that it's just more work for me to do I'm all about making less work for myself so let's go ahead and turn on relative again okay relative sequence numbers there we go all right so what is the client doing here the client is sent saying to the server this is my starting initial sequence number this is where I'm gonna begin I'm not telling you how much I have to send to you yet but let's just start from my perspective on starting here now the acknowledgement number if we look over at our hex values this is also a four byte value at this point in time that acknowledgment number is zeros why because the server hasn't said anything to me yet he hasn't told me where he's starting in terms of his sequence number on his side I'm just telling him where I am starting this is my initial sequence number value sir the question was does that depend on where you take the capture from great question it can yes it that can depend do I have a box in the middle that's changing it or taking over that part of the TCP header that's possible so that's also another reason why it's nice to have a client side and server side captures simultaneously because you can see what was let go how did it arrive how what was like on that side and how did it arrive I wrote an article back a couple years and I'm gonna show you guys the trace father that came from and it was like and I said hey network leave my packets alone because so many network devices now can muck with that stuff so that's a that's a good good comment but if I don't have anything on the network that's changing this stuff that's the whole point with TCP so he's saying hey this is where I'm starting from from a sequence number perspective there you go server you go ahead and tell me what you have on your end in fact the syn the whole reason why it's called sin let's synchronize this stuff all right here's my sequence number you send me yours or synchronized now let's come down here okay window size value all right now window size value this is this is an advertisement of what my receive window buffer is how much can you send me at once yesterday hand song used an example I thought this was really cool with the pipe with water in it and we have a bucket on each side if you're gonna send me a bunch of water how much can you send me at once through that pipe and I can catch all of it right so I'm telling you right now I got an 8 k received window right now at this second that's what I'm advertising okay can't send me anything more than that so this in some cases can limit the server on how much it can send like really Chris you're only gonna let me send 8k at once now one of the reasons why this is so low you might be thinking aka whoa whoa why is this guy advertising something so so low well first of all we do have some options down here we haven't talked about yet but when a client is first initiating something to the server it doesn't want to start with this massive receive window allocation because every time that I send a syn I'm committing resource to that connection I'm saying you know what I'm I'm going to allocate in memory 8k that resource will be earmarked and stamped just for this connection well what if I'm sending a bunch of sins to a server what if I send like 10 12 14 sins to that server and I start with like a one gig window that's allocated so my sin could say up one gig window BAM and there's only I only have so much resource right so I'm gonna kind of start small like are you really gonna pick up the phone server or you're playing hard to get what's going on over there all right I'm just gonna send out a sin if you respond I'll work with that window I can adjust it well we'll work some other magic so you can send me more than just a que Calculon with window sighs we'll get their options let's talk about some tcp options okay so this is also where I am basically laying down some ground rules in the relationship okay so I'm saying hey server I can only receive 14 60 bytes in the TCP payload don't send me any more than that I'm telling you how much you can send back to me in one packet okay that's at 14 60 we're gonna talk a good bit about that as well no operation no operation no operation whoa so do I just have a repeating myself problem here I mean what's up with all these no operation TCP options is that a problem is it an issue no WAP well basically explain that real quick so TCP no operation up here I'm saying that my header length is 32 bytes all right it starts there header length is 32 bytes so that's a it's a binary eight if I multiply it by 4 I get to 32 so that number has to be a multiple of 4 a TCP header in terms of bytes is always a multiple of four okay now when it comes to how I'm advertising options and making enough space for these conversations to happen sometimes I need to pad the options field out to make that minimum header size that I I'm the one who set it up there this headers 32 bytes sometimes my options don't neatly fit into these buckets to make it a complete 32 byte thing so what no op does it's completely Denine if I come over here I'm just gonna select no operation you see it's just a one option value 1 it just means hey my maximum segment size that's 4 bytes no operation is 1 if I come down here my TCP option window scale you see my window scale that's 3 bytes my scale options that's all the space that that takes in the header I've got to be a multiple of 4 so I'm gonna pad that guy out by 1 TCP no operation ok it's just packing peanuts if I come down to sack permitted that's a 2-byte option right sack permitted we'll talk about that in a second that's only to bite indication in the header so what do I need to pad myself out to for I need to packing peanuts boom boom no operation right so overlook it but that's why you see that there in the options part of the TCP syn it's just to pad you out to meet your header link and make sure that it's a multiple of four windows scale all right just the option window scale so this is where I tell those the server all right you know that window that receive window upstairs that I told you about that 88k thing I really want you to multiply that number by this is my this is the value I'm indicating thankfully Wireshark does the math for us it's a multiple two to the eighth so we got 256 all right so take that number I just sent you up there and go ahead and multiply it by this number that will be my calculated window size now why do we need that well if we come up to window size value look at the the amount of space that was originally allocated for the for the receive window in TCP that's only two bytes okay so the most this value can be is 65 535 if it's just that value that I'm using to indicate how much you can send I mean think about it the original spec for TCP the old-school the spec that was actually written as anyone know went what year that thing was written how old is TCP they were the old-school one its MIT it's gone through some change since but when 83 little older 81 RFC 793 1981 okay so in terms of outstanding data on a wire back in 1981 they probably thought boy those he'll never need more than 64 K on a wire going in one direction two bytes will do right back in those days all right so that's why we just have that two bytes that's why this this window size value the actual number can only be up to 65 535 well later on thankfully some TCP options came around to greatly increase the amount of data that we can put on the wire introduced window scaling so if both sides of the connection can do window scaling and we can add this option again we're just laying down the ground rules of the relationship here that's what the handshake does both sides need to be able to do it so here I'm saying multiple by be my 8 multiply me by 8 take that 8k or whatever number I send up there multiply it by 256 that's my real true receive window this is a big reason why it's important that we get the handshake because if I never got that window scale it's only sent once it's sent in the handshake and never again I won't truly know what that when the scale is all right moving through our options here sat permitted all right TCP stack oh we're going to get pretty deep in the weeds with TCP stack but basically what I'm saying I can do selective acknowledgment what does that mean that means that hey mister server if you send me a packet train if you send me 10 cars and I only receive 1 2 3 5 6 7 8 9 10 I can indicate to you which one was lost without you needing to send the whole train again or from the last point after that it's a way I can selectively tell you I miss that block I miss this block I miss that block and you just intelligently resend the stuff that was missing ok there's a quick TC piece Zak okay so there's our sin now there's also some other interesting information we can get from that first packet of the handshake one field that I like to use as well is going up to the IP header come down here to TTL okay the TTL is 128 why would that be useful any thoughts number of hops what does that tell me now keep in mind for me a lot of the time when I get a trace file I'm not the one that captured it I got to figure out how do they capture it where did they capture it what kind of device captured it I got to figure that all out you can do it from the packets largely sometimes I'll ask my customers say hey customer what you do would you have to get this but just to put myself a big part of analysis for me and again my way or the highway I got to think like the analyzer where was this captured in the conversate where was I what was I client in was I server end sometimes I'll ask my customer hey when can you talk again oh I'm busy all week I can't talk until it ended next week okay I got to do it all from the packets themselves now this is a number that can really help me figure out where the point of capture was okay TTL IP tto how does IP TTL work well it's not time as in minutes seconds right number of hops so this packet will die the next router that forwards it at once it hits zero so every router it crosses this number is going to decrement all right 128 crosses the first router 127 126 125 124 123 if it ever got to the point where it was one router decrements it zero die kill hey sorry sir sorry endpoint I had to kill your packet ICMP message sorry TTL exceeded our TTL expired now 128 hops that's a lot of hops you can get around the world and what 1516 hops but or less these days so that's a pretty good hop count that's that's I'm gonna safely assume this number can get all the way to the server now what does a 128 mean well TTLs typically I'll say most of the time start at these bit boundary numbers 64 128 256 right so by capturing a 128 either this packet has not been routed yet and it started at 128 keep in mind this number only goes down it doesn't go up so one of two things is true either this packet has not yet been routed or this packet started at 255 and I'm a hundred and twenty seven hops into that path and that's where I captured it one of those two possibilities is true well there's a third possibility it started at 1:30 I haven't seen that and so long I just this thing started at 128 okay what does that tell me I'm client side right if this was 127 clients one hop away from me 126 client is two hops away from me unless there's an unless I'll put in here unless there's a box between me and that client that's like intercepting the IP stack part of it and rewriting it or putting in its own stuff possible however typically most of the time you see a 128 on a syn your client side okay that's all so while I'll peek I'll peek up to that TTL alright enough on the sin you see how much we ripped out of one sin packet this is great yes possible see what question was doesn't that 128 mean it's a Windows server now this came from a client a lot of times Windows will use 128 it used to be back in the day the different operating systems would use all kinds of different TTL O's and you'd see them kind of all over the board and you can fingerprint the OS from that I just have it I've seen that steadily disappear as stacks have been more standard I'll say that a lot of times in network infrastructure devices will start at 255 maybe they think they have the farthest to go like if you sent a ping from a router a lot of times you can grab that and see a TTL at 255 but what I've actually started to see if anything the number of hops go down in operating systems I see more 64 today than I used to probably because 128 a lot of hops right if I'm going farther than that I wanted to die before it hits 128 so thanks for the question carrying on let's just assume we're gonna make an assumption here let's assume nothing messed with our TTL ok we don't know that at this point but let's assume nothing messed with our TTL this is the syn ack coming back look at our delta time 105 milliseconds that's my network round-trip time sent mice in got a syn ack back 125 milliseconds go to there to the TTL IP header I got this hundred and eleven thing okay now I can just make a guess it's an educated guess to absolutely know for sure I need the other server side trace but I'm okay with not having that for right now now when a TTL goes to the server the server does not say let's imagine the server receive mine I'm five hops away it started at 128 he receives it at 123 he doesn't cut and paste might 123 and then put his packet and send it back whenever an IP stack creates the IP header it always starts with a full TTL all right whatever its number is so on the server side that number started at something larger than 111 and on its walk to me across routers that was progressively decremented decremented decremented until I got it on the client side so now I can make an assumption here that's pretty I can make a pretty good guess especially with a round-trip time like that 105 milliseconds what number you think this numbers started at again assuming nothing's hijacking in the middle 128 it could have started at 255 unlikely I'd be a lot of hops to go through so how many routers are between me and that server 17 ok again absolute a thousand percent sure at this point no but nine ninety-nine point nine in yeah we're there yeah question is the TTL always gonna be a power of two no it can you mean from when it starts 64 128 256 T I haven't seen those in a while either I mean if you guys see them out there if you see starting TTL there's something else come give me a trace of it I love that I just I personally just haven't seen it know what so ok so there we go it's coming at me it's a TTL of a hundred and eleven great I can just kind of put in the back of my mind my server 17 ish hops away or at least it was for this path I've got a hundred and five milliseconds of round-trip I mean this alone can tell you quite a bit of info right now let's go to the TCP header value stuff let's actually go down here and just see what this server is doing all right coming from port 80 all right sequence number as we know this is a long crazy four byte value that's really hard to work with as a human being so Wireshark says I'll do the work for you let's start it at zero acknowledgment number one now that first packet the syn sent me was empty it didn't have any TT any TCP lengths to it it was zero maybe you saw that in the syn the receiver of that sin will increment the sequence number by one and it will acknowledge one why does it do that this is called our ghost byte in the handshake this is TCP x' way of making sure everyone's on the same page it's not a true byte of data but the client sends me an empty syn and I act him one it's a ghost byte not real data it's a false one if you will but it's a way of telling that that client I heard you I'll increment you we're cool by the way I'm sending you one here's my starting sequence number just go ahead and note that on the client side this is where the server is going to begin window size value 64 240 that's what the server is allocating in its resources for this connection come down to options maximum segment size 14 60 bytes servers saying hey we're cool 1414 60 I agree you can send me that I'll send you 14 60 that's our maximum maximum segment size for this connection window scale 0 the server saying hey I saw your window scale thing that option in order for you to do it I got to do it - all right let's do windows scale but what you see is what you get if I send you a window number window size value 64 240 what I mean don't multiply it don't change it you don't have to do anything with it that's my real number okay so the server it's allowing window scaling it's just saying I'm just doing a window scale zero and then it's act permitted okay so these options if the server did not say sack permitted let's just say it's an older stack which I have a trace file of older stack sorry we can't do sack terms of the relationship and I'm not open to that so the client can't do it question question was window scale to zero is that a problem no because the server at least at this point the server is just saying I'm not going to use a scale factor so the number that you see in window size value will always be my actual calculated window size now a lot of times clients aren't sending stuff to servers servers are sending stuff to clients in most cases depending on the type of server so a server doesn't like to allocate a whole lot of resource to this connection right I'm not I mean this is a receive window value so the server doesn't want to say hey I'll do I'll dedicate a gig to you that's way too committed for this relationship like whoa you want to get exclusive here huh I'm seeing other people out here so you get Friday nights so it's just a it's a much smaller value right it doesn't want to allocate that much resource for the justice connection now the client side it knows hey me and you we're exclusive here's a scale factor here's a bunch of resource I'll dedicate alright so is this a problem no in fact I'd prefer this rather than no window scale because if the server doesn't want to do window scaling I can't do it either so at least he's involved he's letting me do what I need to do on the client side does that answer the question hopefully yeah and in fact okay so I got a trace file in fact we're having so much fun that tyrant load our time whoa I only got 20 minutes I gotta move um we're gonna talk about zero Windows when that goes down to zero what happens all right so here's our terms of our conversation when I just park and get through this year okay all right so here's our handshake do you see how much we ripped out of a handshake that's why it's so important to capture them they tell you everything about the terms of the conversation okay so final act there's our handshake and now we start to move our data now keep in mind that because we know the scale factor wireshark saw the scale factor for the client side up in the syn any packet that you see let me just come down here and pick one from one at 170 216 okay 170 216 and cool all right I just grabbed a packet that was down a little bit I come down to window size value this is something that I get asked not quite a bit but I'd use the word often people send me a trace file and they say Chris check out the window size this is a really low window size what's going on here do I need to get in the registry and increase this window size on the client side in fact I mean I it happens quite a bit so window size value is 64 this is what the client is advertising back to the server but remember that since we captured the scale factor in the handshake we know he doesn't really mean 64 he means 64 times 256 that's the calculated window size there's our scaling factor we caught this now if we miss the handshake wire sharks like no 64 is what he's saying I don't have any context for that number now if you miss a handshake if you did not capture the scale factor and you're in a TCP stream somewhere down the road and it says okay I didn't see the scale factor and you see a 64 pretty simple to figure out if that number is scaled or not basically what you see what's the next two packets after this ACK this is data coming in from the server right 14 and 15 those are full sized packets that's coming in from the server so again I'm capturing client-side but by the time this this ACK gets to that server if I legitimately only had 64 bytes that server is gonna go whoa hey yeah this teeny little one I can all right I can only send you 64 bytes really okay ding right okay that it right so when you see data continue in flight the way that we see it here we know that that number is not the real number in fact we can go into our TCP settings until we're sure to do a manual scale factor if we if we had to okay so we talked quite a bit about this trace let me switch up here I want to go to yes okay let me go back there oops I opened up case study let me go back to the same one here okay alright so first of all the question was we see that scale factor in the sin so right here if he's advertising a scale factor how come we don't see it immediately off the gate right we see an 8k window he's sending a scale factor of 256 keep in mind that this is just a sin is establishing the terms of the relationship laying down a ground rules right right now I don't know yet if the server will let me do this I'm offering it to him like hey so we cool could we use this option so right now it's not applied but in the AK as soon as I get a scale factor back from him the three the the the third packet in the three-way handshake will apply the scale factor right so this is the AK the final act we're cool sequence number one acknowledgement number one we're synched now let's apply our scale factor so it is applied but just on the third packets and the reason again it's not in the sin it's because I don't know if I can do this yet and if that server doesn't do it then how I got is sixty five five thirty five so something just that I'd like to kind of kick out to you guys as exercises when you're capturing when you go home and look at your systems and stuff what kind of options do you see coming off of your servers when you hit something take a look at those options tells you a little bit about the stack is it a really old stack that can only do certain options you know a lot of some traces that I've seen from these IOT devices that are out there it's interesting to look at the TCP options with some of those devices they're pretty limited it's pretty interesting in fact if we have time I'll show you the trace or two of that but anyway um actually let me show you one that I not long ago okay here's my handshake okay we can see what the client is offering all right let me come down here let's focus down here so kind of similar to what you just saw this is a client talking to a server 39 milliseconds later we get a syn ACK so we're going out there we're sending our stuff we're starting off all small 8192 don't want to commit too much yet until I know what the ground rules are server comes back with a sin anything missing tell me about this server what do you learn from this let's just look here and the TCP options we got an MSS okay he's doesn't have sac okay so he's telling me that sack thing can't do it no fast free trans you send me dupe box I'm not gonna know what to do with them okay so that's interesting windows scale right there he's offering me 8192 and he means it maximum segment size okay interesting kind of a weird number to me little bit but all right we'll work with it he's just telling me 1379 that's what I can receive no it's possible let me throw out that maybe thing it's possible that something along the path changed that MSS all right he could have sent it at 1460 or whatever number he picks in flight a router could have gone Bing and kicked it down to 1372 or 1379 that's possible that's why I can't be absolutely sure what he sent unless I have both sides but this is what I want you to get used to when you're looking at these handshakes spend some time looking at the options what do you see what do you not see there what's missing it's establishing the rules of the ground rules of the relationship okay now if we don't have a handshake how we doing man we're just having so much fun look at us let me just pull okay sure I think this is one where I miss the head shake or I I was sent it without any context okay I have no handshake here this is just a single conversation between a client a server a big dump of data is going one direction because I don't have the handshake let me ask you this since I don't have the handshake what do I not know at this point I don't know my scale factor good job so if I come down here to TCP calculator window sighs okay here's the original one window size scaling factor unknown Wireshark saying look missed it don't know sack I don't know if I can do sack or not I could look at the behavior and determine that you know what I mean if I see do packs and faster each hands okay these guys do sack what else do I not know maximum segment size I don't know that but again that's something if I if I look at my my info and I see stuff going in both directions no retransmissions okay yeah sorta determine it right okay but again those handshakes are important to get they tell us quite a bit that initial round-trip time it lets us have a good picture of what's going on in that conversation all right so you know my problem is I'm just used to having more time with you guys that's my issue because like usually when I'm doing Wireshark training I got like 3 full days like hey let's shred but four hours okay um sequence numbers this slide came about because I was chatting with a few of the attendees over the past couple days and they just said you know what again think one fin right we're just reviewing some of the TCP stuff doesn't mean that you know it doesn't mean there's a problem it just means hey let's go back over sequence numbers how those work and how we can make sure we understand what they're doing because in session number two that's what I'm gonna start getting away from the fundamentals and start going into case studies then you'll need to do some of this in your brain so let's go back over sequence numbers and how those are counted okay let's go to this next trace okay same trace oh no no no oh same is the first one okay so sequence numbers we've already talked about in the handshake how one ghost byte of data is exchanged it's not true data it's just a ghost byte it's just to increment the sequence numbers on both sides that is exchanged in the handshake all right so if I come over here to my if you look at my sequence number sequence AK sequence AK so my sequence went over my starting sequence number is always zero in that well where shark clears it to zero I send mice in the receiver acts one he sends his zero and then I act one to him so we basically sync up and start with sequence number one AK number one if I come down here to the first actual packet with stuff in it going from the client to the server you see four fifteen fourteen that length of packet all right this one actually has stuff in it length fourteen sixty okay let's take a look what happens to the sequence numbers here now just so you know if if sequence numbers and acknowledgment numbers if they're confusing to you this is absolutely something that I definitely suggest you walk through walk through again and again back up do it slow start from the sin my poor wife when I'm doing sick with sequence and acknowledgement number analysis it looks like this all these numbers everywhere on my and I'm like just writing these numbers down and especially tracking down a sack thing and she comes in looks at my desk and she's like okay not asleep I'm well I know what not to throw away okay um all right so first real packet of data this is where the session is starting to be actively used we start on sequence number one client is saying this is a client request that's going to the server he's starting on sequence number one I'm ascending how much data exactly fourteen sixty that's how much I am sending to that server guy over there okay my next sequence number is sequence number I'm starting at plus amount of data sent this is my next sequence number since I'm not yet on that packet I'm still at one so as soon that Wireshark sees that 1460 I'm sending and it suggests to me the next packet in this direction will begin at 1461 unless there's a problem all right I don't yet have an ACK it's just the next packet now you can put all this stuff in columns and stuff I don't do that to you on purpose just because if you unless you're used to my columns and you built it out yourself it can be more confusing than helpful sequence number notice the one that we're starting off 1461 okay so we're 1460 bytes into this packet stream now I'm starting at 1461 I am sending how much data see my length right there 61 that's how much data I am actually sending or TCP segment length okay so my next starting sequence number will be 15 22 that's what that's where I'm up to when I begin a new packets and any data that way that will be my sequence number that I start from however in this case the clients sent those two packets one full one in a little bit of residual and that was the get that was the request that it was sending to the server client pauses doesn't send anything else don't need to I sent my get server comes back let's take a look at him he's starting on sequence number one we established that in the handshake sequence one next sequence number is also one why it's an empty AK look at my TCP segment length zero server's not sending anything wait a second server I ask you a question you tell me you're not sending anything we'll get there sequence number one next sequence number one empty ACK I'm not sending anything however look at my act number acknowledgement number 15 22 I'm acting what you sent this packet is a very important packet when it comes to application analysis why this packet tells me that server got my request it got there he acts 15 22 if I don't get this ACK what am I gonna do as a client I'm gonna wait some amount of time and I'm gonna retransmit what he didn't hear me this AK I I just personally call it a hush packet all right the server got my stuff my stuff is there my mic get is being processed he's going and doing whatever he's got to do on his side and I'll just hang out now in this trace file how long do I wait I wait 20 seconds that's the next thing I hear coming back from that server is that the network network client server what do you think the server now again I'm capturing client-side can't be a thousand percent sure won't bet my firstborn if I had one put on that side over there if I had that capture that's a number I'm gonna look for okay so that first packet that act okay good I got your stuff TCP at that point packet number six TCP can't do anything until mr. layer seven up there drops some data boom got it here you go server-side TCP is like I got you hanging Oh boom right so server-side TCP he did all he could do this is application stuff let me show you this was actually a fast response from this server I'm gonna show you do I have time oh I got two minutes I got two whole minutes woo check this one out this is how bad I got in this specific case what do you see people okay syn synack ACK what's my round-trip time 97 actually taken from the same two station so I won't show you the IP t TLS syn syn ack ack we got our request that first big packet come from client packet for big request full packet residual stuff coming in packet 5 server gets it packet 6 106 milliseconds later server acts back with 1496 if i looked at the sequence numbers that's probably exactly what that client sent right we see the 1496 in the act from the server that number right there act 1496 ooh then what happens okay we wait packet 745 seconds client says we still there this is from the client right 190 168 go into ten tens of servers 192 168 clients to me that's just how I work seven okay so this is an empty packet it's called a TCP keepalive that's a timer basically the clients saying look I gotta go I got stuff to do and resources to free up so are you still even there as I sent that request to you you said you got it I've been waiting 45 seconds over here and you're not saying nothing client says fit 99 milliseconds later which is pretty close to our network around trip time server rather says keep alive ACK yep I'm still here no data but I'm still here TCP is still up we're still talking let's keep this thing alive client waits another 45 seconds checks back in hey server what's up I sent that request to you you said you got it are we still talking or can I just kill this thing and where it will just be done with it server comes back 99 milliseconds later and 95 in this case yep I'm still here so the two TCP layer four they're good they're chatting they're talking now Wireshark in this case this one was sent to me because the poor clients like looking at these black lines and red letters and going oh my gosh oh the network's exploding wait a second here TCP keep the lives there just keeping this connection alive that's not a problem the problem is that on the server side that TCP is just waiting like how long does it wait well we see 18 seconds after that second keep alive let me just uh I'm just gonna come up here the last packet of a request I'm gonna start our timer over set on set time reference I can actually pull this out a different way in my HTTP profile but basically you see there your little arrows that Wireshark helps you out with you see the arrow that went to the server and this is the response from the server in packet eleven we ate it we waited a hundred and eight seconds on that server to respond and these guys actually waited that long they went to this tab hit the tab and then go get coffee come back they got their stuff and some people sat there and just spinning wheeled for a long time and of course who is getting blamed then at work let's go from one to ten gig to the desktop wouldn't have done a whole lot right so next step from here and I'm gonna go ahead and leave us on a break right now after staying this at this point from the client side all I knew is I had a hundred and eight seconds of application delay to figure out what was going on I would have had to move to the server and figure out okay this request comes in what does that server do next does it legitimately wait and do nothing for 108 seconds or does it go and talk to some other server and get some other tertiary data okay guys this was TCP fundamentals coming back in 15 minutes we're gonna pick up again with a bit more on congestion window a bit more on receive window and then some Case Files so come on back if you would like you [Applause]

Channel: Chris Greer

Views: 133,996

Keywords: Wireshark, TCP/IP, TCP, slow network, Protocol, Packet, TCP Handshake, wireshark training, wireshark tutorial, packet analysis, packet capture, tcp analysis, tcp connections, wireshark tutorial 2020, packet pioneer, chris greer, transport control protocol, wireshark analysis, tcp fundamentals, wireshark course, intro to wireshark, how to use wireshark, sharkfest, how tcp works, free wireshark training, free wireshark tutorial, network troubleshooting, wireshark for beginners

Id: xdQ9sgpkrX8

Length: 77min 24sec (4644 seconds)

Published: Mon Jul 23 2018