How to Tell if it's the Application, Server, or the Network - PacketBomb Live Stream Case Study

[Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] you [Music] hello hello welcome to the first I mean second but first real packet bomb livestream experience this is not my quarantine hair if you don't know this this is from a video I made one time with my daughter when I was a clown and I tried to force-feed her food for your entertainment to demonstrate TCP concepts because I'm that committed how we doing with the audio is it good my test earlier this week was a little low I tried to make it better I think we're about eight or nine seconds delayed so I'll keep John I'm good that's good to hear I am sporting a little quarantine self haircut mm-hmm so no curls afro at the moment it'll have to grow back out so welcome I'm there was a time when you know I would enjoy a nice bottle of Lagavulin sixteen or I don't know some other nice scotch or bourbon these days I just get bullet by the gallon so Cheers it's 6:00 p.m. here in San Francisco hope hopefully it's happy hour or whatever wherever you are this is just bullet and soda maybe next time we'll be fancier I do have some the rock tequila tera mana a big fan of the rock you know it seems like a good guy it's a pretty good tequila but tonight's more of a bourbon thing Oh someone's drinking Lagavulin right now beautiful it is good the best place to get it here is Costco it's like maybe 50 plus $60 but I have a BevMo around the corner which is I'll just run and get stuff there and they want like a hundred bucks I'm just sorry not I can't do it so all right what are we here for besides booze so eighties meat I said there would be eighties music well you heard it already that was i Adam I'm not gonna play like real eighties music like you would know because I don't know how rights streaming rights work I'm assuming I can't just play a radio station on YouTube so that was called Blade Runner 20 49 by synth wave goose you can find him on Sound Cloud SoundCloud throw him a few bucks I mean you can listen to it for free but I paid him a few bucks you know so I could download it I think it's a pretty cool tune so what are we here to talk about i have spent this time I've been a Mac guy for since the 15 plus years let me keep the chat up here so I can see what you guys are saying and so it's in the background back there now my daughter uses it for schoolwork and I built a PC I haven't built a PC since 2002 I think the last graphics card I bought was a 3d effects voodoo 2 and you know I had that in SLI to play quake ii rocket arena which is also probably one of the last games i really played on pc i tried to do x-box and PS stuff several years ago I'm not good with my thumb's I can't I can't I get I get killed so easy so anyway I've been playing a little bit I just finished Jedi fallen order great game really good right now I'm working my way through doom eternal it's pretty cool and yeah I'm kind of getting used to windows again I haven't used Windows I briefly used it for one year when I first joined riverbed in 2008-9 and then I switched back to Mac because I was Linux before that and then a Mac and then one year of Windows and I've been a Mac ever since so um wait action quake 2 mod was awesome I do remember that one yeah I hear that they you can play it in a web browser now so cool honestly Windows 10 is not bad just plenty of things that don't know how to do I just google it and I find the answer and I do it I mean the look and the feel and where you go to do stuff is all over the place I mean I know that but the fact that I can quickly find answers and go go do what I need to do was fine and the interfaces on some of these applications are terrible but fine so I did let me actually let's switch so I can show you something on my screen I'm gonna mash a button so cool this is my this is my website which by the way that if you haven't seen it this tcp/ip with toddlers is what that clown thing is from I don't know if you can hear this and I apologize watch your ears oh man that reaction gets me every time so that was you can watch that you can find on my youtube channel this is the the guy synth with synth wave goose he's um I don't know this was a cool a cool thing so I took a picture of my setup in it's in progress no don't get me wrong it is in progress this picture I'm about to show you I've even I've already changed the desk I moved the the PC up to the desk which was a mistake I posted this on reddit battlestations now it has a cable management tray underneath at the top in the back but I could not use the the monitor arm clamp it was the the tray was in the way so I had to partially disassembled the desk drop the tray to the floor and shove all the stuff in there temporarily and I gave people clear instructions don't look here this is not what your these aren't the droids you're looking for etc oh I got roasted I knew I would and we had some good fun making jokes at each other Windows terminal yeah I just downloaded the Windows terminal it's pretty good I was using I term on Mac you know I got about a week in and I was frustrated the only thing I was really really missing was bash so I could run BC as a calculator that's just how I do my calculating I don't like the graphical ones so I installed the enabled the Linux subsystem you know what you go click and enable it and then you go to the App Store and download Ubuntu and then boom I can type in bash and I have a bash shell it's great so anyway this this desk is gone I had I wasn't sure what I wanted to do for a monitor I had a 4k you know really nice LG side-by-side with this dell ultra wide which is 1440p and I had the 4k first and it was so beautiful and so crisp for text I was just like drooling but like I wanted that ultra wide immersive and it was 60 Hertz refresh rate and and I wanted the ultra wide which has a hundred and twenty Hertz refresh rate and native g-sync because I have a you know a 20 70 super graphics card and when I got the ultra wide and put it next to it I was so disappointed in how it looked it just looked like trash I thought compared to the 4k but then when I started gaming it was no contest and I didn't plan to be like gaming it to be a big factor but man playing ultra-wide it a higher refresh rate as if you know what if I just don't look at the 4k anymore I'll get used to how text looks on here and so I send it back and here we are okay so what else we're gonna talk about so yeah this is an opportunity that I don't have to do a bunch of planning too much planning ahead of time at a bunch of editing right that takes up a lot of time and clearly I haven't done much of that in a long time now wouldn't know how to do it on Windows anyways but I have a number of case studies some that I've done at shark fest but never post it on YouTube the one we're gonna do today Fidel sent me he's actually in the chat I've got a few kicking around that we're gonna do and he doesn't have to just be case studies I like case case studies I think they're fun I think they really sort of demonstrate the power of what we're trying to learn how to do here with Wireshark and packet analysis but I mean we can talk about anything I love to do this maybe weekly we'll see so real quick why don't you guys tell me like what is it you guys do are you network guys are you Lynn I mean uh security people voice people sis admins just stumbled on this and you're really confused about what we're doing and talking about case studies not made by mark I am the marketing department I'm the I do the marketing I do the content development the product development mean you know I've been fired a couple times mmm jack-of-all-trades see that's that's a tough spot to be in because everything's your fault network system man well who do you blame you look in the mirror and then like point to yourself and say this is your fault fix your server know it's your fault fix the server the network yeah so pretty standard stuff if you are security I'm sorry I don't have a lot for you my very first video was click Beatty click bait Tilly titled how to hack a Cisco router but if you get past that part it's actually a pretty cool video about how to manipulate packets on the fly what do i do at riverbeds I started out you know in tech support answering the phone solving cases then I was working back line support in interfacing with engineering for a few years I did that I don't know five six years and that's where I learned I mean before that I was a sysadmin network admin and I knew enough to like why isn't this multicast application working I took a capture oh the TTL is one it can't get outside the local subnet let me troubleshoot DHCP why isn't it working look at the options you know basic stuff but we all the time got complaints about performance in some building on campus or some application and it's like let me go look at the metrics on the switch the cpu the throughput you know these things are taken every five minutes I mean good lord that's an eternity right in network time so I didn't it wasn't until I was working on the vendor side and customers are like this isn't working or it's not working as it should and log files can only take you so far sometimes and over time I found that being able to look at the packets and understand what's happening and explain it to someone was the quickest way to solve the problem especially when fingers are pointing blame is flying when you come in with the packets you walk it walk them through it explain the thing and most of the time they're like oh okay I guess I'll go call Cisco or whatever now I'm in management so you know I'm dumb this is on the dark side I haven't kept up with a lot of the technology I started this site obviously you know like five years ago whenever it was just to dump this out into the world before it all drained away you know and I was just drooling at my desk and here we are so can we follow along yeah that's not a bad idea sorry maybe next time can we follow along live with pcap files you're using that's a great idea I'll I'll write that down for next time I just happened to have some paper here pcap I don't know what that means do I miss tech support I mean I get I missed the that little high that you get from solving a problem I mean the story that I will never forget is one where the problem was I think in a pack and it's an you know optimization something is breaking and it's a complex web app inventory management databases and we the team had been banging away at it all day they passed it around through amia and he said whoever can pick it up please progress this overnight and you know you're like I hope when you're in that situation you're like I hope I come in the morning and my colleagues have figured this out and pulling that apart I wish I still had the pcap so I could maybe recreate it because it's one of my proudest troubleshooting and that packet analysis but the high you get from that you know I still get a little taste every now and then with this but I do honestly think someone asked about how do I like it on the management side it's a different set of problems it's a different skill set and I do enjoy that it's it's certainly not for everyone and it took I took sort of a path there I kind of did a mid step where I did escalation management so I'm not like managing people but I'm managing situations and I'm not super technical and then I became the manager of the backline support team who are all like top-notch super sharp guys so it's not like I'm like why are you know wagging my finger at people for goofing off or not you know we don't do call centers metric type II stuff at least in my realm and in my experience I'm sure certain managers maybe be like that it just depends but for me it was always about solving the problem and helping now now I'm trying to help my team be able to do that better yeah micromanaging I I've been very lucky not to have to be in that situation very lucky I think certainly there are time places jobs companies where it would be a nightmare to be in management or to be managed by someone like that and I've been I've been super lucky I forgot what I where I was going I was asking you guys about what you do we're talking about that what I used to do and here we are talking about packets it'll one of my guys just went through it all i tol it'll I don't know training you know just to bring you know some structure to escalation management I hope what's my favorite packet the one that the light shines on and the heavens break open and you figure it out okay so is there something else I was going on anyway I'm open for ideas for content stuff you want to see if you're like I mean if it's something I can do I'll do my best if I can learn something or do whatever that's fine topics we can try to do something no promises but I'll do my best so what I have here is a and I think we're getting ready to actually do something productive Fidel sent me this let me just give you the quick highlights deploying servers across the company to separate vendors on air content one server sends data to the other server that plays that content out on the air it's called mos moss ba bla bla you know it's it's an it's an application protocol that sends requests or actions and expects a response back pretty standard stuff it's a TCP based protocol was we're sending a command and we don't seem to be getting response back it's not working the server's disconnect from each other what ended up happening now I'm sure this has never happened to you that the vendors kept blaming each other and then moved on to the common enemy of the network and that's when I was brought into this fed up and they're saying you know what the problem here is your servers are on different switches and VLANs and that's you know obviously that's broken how I mean come on application developers what do you expect so okay I you know I'm blah and then I figured it out for them basically is what the rest of this says so he figured this out he asked me if I think he said you want to take a look this and then I asked him if I could I said great job you nailed it and can I use this as a case study now if you want me to look at your packets and maybe do a case study or whatever it's a good idea to anonymize the the data and to me the easiest way to do that is a little utility called trace Wrangler now I've not been able to use this much because you know I was a Mac guy and I'm not gonna spin up a VM you know for like a little utility or whatever when I've done when I've been at shark fest presenting and I've needed like I finally got to the point where I'm about to present like oh wait I need to anonymize these peak apps I'll be like hey Jasper who is a longtime presenter at shark fest can you do these for me on your tool so Jasper wrote this tool because it filled a need for him trés Wrangler ooh you know what it needs a theme song like a cover like a old country-western TV show theme song like the riflemen I don't know anyway you can download it it does a lot of cool stuff which we're not going to get into because frankly I probably don't know how to do it all but he has presented at shark fest if you go to a shark fest why shark shark fest and you go to the retrospective and you can go to all these years of shark fest content and there are they've gotten better about it in the last two or three years where they have a video literally of every presentation it used to be maybe just the main room they'd have a video otherwise you just get the slides which isn't always useful this guy here you go tackling the haystack he talks about using tres Wrangler in this talk but I think if you search you'll probably find another talk where he just focuses on tracer angler it's a handy tool so what we're going to do to use it is to anonymize these peak apps we're gonna add files we're gonna take these two peak apps that we have and we're gonna add them to the top and you can put as many as you want if you've got one if you've got a hundred you just put them all in here and then you tell it what do you want to do on these files extract edit merge we want to anonymize so I don't really know what all these features do I'm gonna be honest but I know the basics of what I'm trying to accomplish and you know if I play with a little bit more I'll get more familiar with it so for the payload we know this is TCP we were done told that rawhide that's like that's good I need to do a rawhide parody song for Jasper we do not because this is an application we don't want to remove that data right you know if I knew we were just looking at IP and TCP anything like layer four and that's it and nothing beyond layer four I don't care about it then we could remove it but I don't want to do that I want to keep it in because we may have an application layer problem so you can replace the the the P cap in G header information if you want that's fine I don't care or you can pass it through it'll it will sanitize the interface information you know you can sanitize the comments if there's comments and I know yeah so that could be handy we're not doing this wireless Ethernet fine we will replace the maket layer two MAC addresses now it will keep the vendor part the first three bytes it will keep that intact so if it's HP or juniper it'll still say that which is useful and it will replace scramble the last the last three bytes so cool VLAN I don't feel and in this traffic he said it was VLAN but I don't know if it's well no it wouldn't be this wasn't on trunk this was on access port whatever replace it okay IP so if you want to replace specific IPS with another specific IP you can add that configuration here I don't care auto mode and here's the cool thing now there are two P caps and obviously we want the IP replacement to be consistent across all the P caps well this tool it will do that for you and there's other things here that are probably cool and honestly I think that's all we're gonna do I'm gonna fine it'll replace the IP ports that's fine I don't really care so much but potentially you may want to leave the ports alone just so there's some consistency there it depends on your situation man you know if you need that information you then move pass it through ok that's it okay and then you just hit run and it goes zipping zipping yeah I've got a I got a rise in 730 700 X so you know crank through those four megabit files pretty fast and that's it it's done so if we go to here now it's appended to I didn't talk about the output but there's you can give it a different prefix you can put it into a different directory you can do all kind of cool stuff so if we recall I believe it's the en PS server sending information to the bit central server so let's have a look at this no I already have this one set up exactly how I want it so let's this one anon now I'm sure you can read that great let's make it bigger said okay I'm looking at I got my laptop up so I can see what's happening so now I have a super-clean this is a brand new install because I uninstalled it and then reinstalled it so everything is default Wireshark we're gonna make some changes because when I would walk around in the office and I would see people looking at peak apps with all default settings you know I would sigh heavily and then go help them so first off just the layout it doesn't work from you guys and also I was really I was struggling to figure out where things were one I don't do this that much yeah I don't mess with my configuration once I have it set you know small tweaks but it went from scratch I haven't done that in ages and on Windows practically never so we're going to go to preferences and yeah what yeah that's all fine columns we're gonna talk about columns because this will not do fonts and colors do they have Comic Sans I don't I guess I'll leave it alone layout first thing I'm going to do this is my preferred layout and I will talk about why once we get to back to the main screen we're not gonna worry about capture right now honestly I don't usually do that much name resolution I do like to resolve MAC addresses so I can see the vendor especially when I'm a vendor and I'm troubleshooting I want to know of course pretty quickly you learn to identify your preferred vendors you know identifier but I like to see am i talking with a Cisco box a Juniper box you know whatever it is I like to have that transport names i I want this off I don't want to see HTTP DNS I want to see the number I want to see 80 I want to see 53 and I don't really want to see network names I want to see IP addresses sometimes it's useful and I will turn it on but for this we're not going to and so we're going to ignore the rest of this and there are some TCP stuff we might take a look at and that sure if it's gonna come up in this case study but there are definitely TCP preferences that I will change or toggle on and off depending so we're gonna hit okay now here we are the reason why I do it like this is because 99% of time I'm not looking at the packet bytes so I just move them out of the way sometimes I want to quickly look at them so instead of having to go toggle it back on it's just over here and then that gives me the flexibility to get more real estate for my packet details or for the packet list so that is my setup now second thing that is annoying and this one is under view colouring rules can you see that more or less these are all terrible I think I'm gonna leave bad TCP so if there's TCP analysis flags and it's not a window update a window update is a good thing this actually I was I think I was sitting in the room when Laura Chapel complained about this to the developers at shark fest and they changed it because the window update is a good thing right that's the receiver going hey I can take more data now so let's not let's not call it bad TCP and anyway I'm gonna leave it there I don't care about any of this stuff if you do that's fine you can leave it or maybe you want to change the color I don't I don't care TTL lower expected I mean I can't I haven't seen that too many times check some airs fine these really no I don't want protocol coloring rules we'll come back to that one trial is off now for this one I don't really care about fins so much but sins I care about a lot I want I don't mind maybe calling out a syn packet so I'm gonna say TCP v z-- equals two which is where sin which is the sin flag well that's just the scent that's not a syn ACK that's fine I don't care in my other my Mac where I previously did Wireshark I had custom coloring rules we're not worried about that for now so there we go that looks so much better interesting that it didn't update the little doohickey on the side you still see a bunch of blue that's not there okay so now let's talk about some of these columns mMmmm reword goldfinger you know I do have they I do have a good video how to set up wireshark you can by all means go check that one out on my channel Wireshark 2 vs. legacy is this Wireshark - what version is this um that was three where do I look help 3.2 I haven't used the old Wireshark and so long I couldn't even tell you I think early on this is had a bunch of bugs in it and some features that I really cared about but for the last couple years at least there's no reason in my opinion well I'm not a these days I'm not a power user I don't use it everyday and even then there's tons of stuff in Wireshark I never look at for me this one's good oh and by the way I just want you you know not to but you know I am in this list there's a feature I put in here we won't talk about it this time but maybe next time so um okay first thing if you're going to add one thing to Wireshark this is what you need if you look under the frame you want a delta column time Delta from previous displayed frames so that you can at the packets that you're looking at whether you have a filter or not you're looking at the time difference between every packet so you can spot delays you can spot patterns so we're gonna add that and basically anything in the packet list just about I think you can right-click and add it as a column and we will edit that to just be Delta and I'm gonna move it over here by time now time by default is time since beginning of capture and that's where I generally I leave that as my default but sometimes if someone says hey the problem occurred at 1:30 p.m. I go ok well where did you capture it what time zone because the time the times displayed to you in your Wireshark is relative to the time zone settings on your computer so if they captured it in central time zone and I'm in Pacific time zone I got to do the adjustment if they say 1:30 I know it was 11:30 you can change this under a time display format some so you can go to time of day if you want I'm just gonna leave it time since beginning of contra ok now for me I don't really care about the protocol sometimes it might be useful I don't care so we're going to remove this column now the length I do recommend having a length column and it is your preference if you want to have the full packet length like this right are the frame length or something like TC peelings me personally I spend a lot of time at layers 4 and above and I care about the the payload length tcp length because that's a direct reflection on application behavior and MTU MSS those those are things that just by looking at the packet length I can draw conclusions from and I mean you can as well from the frame length to some degree my personal preference is TCP link so I'm gonna add that and if you look you see some things have brackets around them if Wireshark has brackets that is just information Wireshark is telling you is not in the packet let's pull out the old packet drawer byte drawer for example stream index I'm clicking on this it's well now I just ruined it if I click on source port it highlights that in the bytes there it is that is the source port there's a destination port there's no stream index there's that and there's no tcp length right but it is based off information in the packet so it did highlight but my point is if you have a next sequence number that's not in the packet that's a calculation so we're gonna add segment length apply as column and I'm just gonna call it TCP length and I say wrong things all the time either because I just misspoke or I just have the wrong information and if you if I do that I know there's people in this chat who are pros feel free to correct a correct me for everyone's benefit so let's I'm gonna remove this column there are other things I personally like I like to have sequence number next sequence number act number because those are critical when you're looking at TCP performance application issues but you don't necessarily need it all the time and if you're new this is already overwhelming you open Wireshark you're like oh what there's so much information what do I even what do I even start so you know what this is this is a good start to to start looking at an issue so Fidel sent these pcaps we're looking at the I just don't don't do that I'm I'm displaying only you know a 1080 window off of this larger monitor so if I move things around it screws it up so but what what do we have here we know there's two hosts talking and there's a problem but let's have a look one thing I like to do especially Oh one thing I wanted to point out and kudos to Fidel on his what did he say he kept blaming each other so they're saying oh it's you know it's on different switches and VLANs clearly that won't work he knew that wasn't a problem but he elite he simplified it and to make them happy so we could get past that and move on so he span port on the switch he captured both ends while and in creating a new file after five megabytes so you have like a rolling capture and then you can walk away and say hey let me know when the problem happens but let me know quick because we don't want to lose the data and then when they say the problem happened you go in to stop the capture on both ends you get the time that happened and you can go in and hopefully find the issue so this is this is a good approach that he did here but for me stranger don't know what am I looking at so capture file properties are useful so like when I was doing this day in and day out and someone comes me says hey this problem we can't figure it out here's this data can you look at it and they captured it both ends when did the problem happen to happen at this time and you open this window up and you see that the time span the start the first packet and last packet don't even cover the time of the report and problem I don't even go further I close it and say do better please get the data next time so especially if you're looking at both ends I would want to compare these two pcaps and make sure that they overlap that they share the same time right and if they do so it additionally what else is in here I don't know it could be a ton of stuff well to Ethernet so layer to only two I through net addresses just two IP addresses so I'm assuming these are the and of course he would you know when you're working on a problem your your ideally going to have the IP addresses that you're looking for but there are a bunch of TCP connections between these two hosts so which one has the problem huh boy how do we find that out so there's a few ways you could go about this if you know the time of the problem you set your time display to you know time of day if you need to do time zone adjustments in your head or whatever do that go to the time I mean they're not obviously telling you to the millisecond they're probably telling you to the minute ish but I think this P cap was what 20 minutes 30 minutes right so that least narrows it down a lot maybe you find a reset maybe you find an obvious problem that's one way you could do it however what I tell people before you even get to this point with looking at packets you've got a lot of work to do the more work you put upfront playing detective and pushing for more information getting as much information as you can understand the problem top to bottom as best you can before you even gather data and look at data you're gonna come off way better and one thing I didn't mean to talk about at the top is like about Wireshark in general and I am I on my page at the beginning I have a short little video about when you use Wireshark and it is not for everything Wireshark is when you it's it's a tool it to me it's a scalpel it has surgical precision you're looking at every single packet so if you're trying to get like trends and top talkers across your enterprise network or over long periods of time you've got gigabytes of data whereas you can go from that down to the problem and fix it you don't just like start with Wireshark right you have work to do before you get to this point and Fidel did that right he set up the capture proper he asked the right questions got the right information and then he's got just the traffic that he needs sometimes if you go too far you miss if you capture I don't recommend using filters when you capture because you could miss something important what if it was DNS that was a problem what if it was a database connection that was a problem filter after the fact if you can okay so you could go look at a time but he had additional information he had it the application layer he had a did I lose my scroll I did I lost a fancy little thing on the right nope where's the reload button there we go windows right and it's probably not Windows so we have some information at the application layer now Wireshark doesn't by default understand this moss or MOS media object server protocol and you could go like okay well maybe it just because remember we anonymized the the the port numbers to so we may have to tell Wireshark what this is and if we go down to like M there's no MOS and I could I think I did google it a point is I don't believe there's a dye sector in Wireshark for this protocol but we're in Locke one is plain text too it's a really simple protocol so what you can do is we're going to take the we're going to take that ID information message ID copy we're gonna try to find it in this pcap and over the years there have been different ways that I've accomplished this and because this is an active project that is literally taking input from people and changing all the time there was a discussion on the Wireshark mailing list just a few weeks ago about changing some very default behavior around sequence numbers because people decide it works better a different way so just because you did something one way in the past doesn't mean it will always do that so here's how you do it now control F or command F if you're on a Mac for find and I'm going to put the the message ID in here right now it's not a display filter it's a string and I don't want to search the packet list I don't want to search the packet details it's not in there it's in the bytes I want to search the bytes and I'm gonna say find and it highlighted this packet so ideally this packet in its payload has that string now that's just what seven digit numeric string I mean we've got 53 hundred packets could it be in more than one well let's find out uh well it is it's in more than one right I mean does that mean the message ID is being reused I don't know we'd have to look but let's start at the first one let's go back start at the first one and what we can do is right click follow TCP stream and we get the the payload all concatenated together right with these little periods in between each of the bytes and you can just kind of glance right so the red is what's being sent and further down blue or their responses so we can kind of see it's HTML e right with these the brackets and and you you close it off with a slash so you know this is mos Maus an ID a message ID okay so we know we're looking for a message ID a command of some sort an ID story story body you know pretty pretty basic application and so here's and at the end of the met at the end of the message it closes the whole thing off right because everything has to match and closed off and then here's a response with a message ID with an AK right pretty straightforward so we want to find that message ID in here and hopefully it is the message ID not some other random string or else we'd you have to keep looking now what's about this one you can't just search for that it won't find it because look at what how it appears on the page it has a period between each character so if we do find did we find it where's the message-id 106 75 9 n um Wow oh wait did the space matter it did huh it is a message ID one of six seven six zero eight message ID it is an ro replace that's just an application protocol command but that was what was said in the information that we had when we were talking to the vendors and whoever else about this problem so this may be this is it we scroll down we can see payload various information about it looking for a blue response so I don't know about this out this protocol application I've never looked at it before I get the gist of it just by looking at its behavior but if I need to go deeper I'm gonna have to Google I'm gonna have to read I hope we don't have to but that's that's what you have to do when you get stuck trying to understand how an application protocol should behave because you need to know the expected behavior and many times you can infer it but sometimes you can so scrolling down I never saw any blue and here's the end of the request right now a cool thing about this if I click on some of these bytes it'll go to that packet so I'm gonna close this this is the end of the message well and right away I'm looking at this and I'm going okay so clearly we're sending data you know the game we're now we're just thinking about TCP remember TCP doesn't care if it's web if it's DNS if it's moss if it's something else it's job is just to send data and make sure it gets there in order and read trends you know make sure it gets there right you know about TCP you've heard about it so that's all we're looking at right now and looking at just what I see I see it's sending data right and I'm looking at the TCP link 1460 that's a very standard them it's a full size frame right I click on that 1514 byte to the frame that's that's a full size full size frame full full payload and then there's one that's not with a push flag to me that means that's the end of the data applications done sending data for now the the buffer is been emptied where you have a smaller size a bunch of full size frames and then a smaller one with a push flag signifies at the application layer I'm done from the receiver side it means oh that's all the data let me push it up to stack to the application and then we have some acts come in from the receiver and a fin ACK and if you look there's a 10-second delay from the ax to the fin ACK so that's the end of the connection right then we do a graceful close we're done but when we looked at the date so we never get any data back you know again so this is where sometimes you need to know the application protocol is it multiplexed is it replying on a different TCP connection or something it's not but things you might need to know so the question is was there like some data loss you know what happened here is the network at fault what we're trying to understand is if the network did its job or not and this is where I would normally put up my DCP columns but we'll just look down here so the last packet that was sent had this sequence number 92 9 with a next sequence number of 995 1 and so next sequence number is just the current sequence number plus the amount of data you sent remember sequence numbers are just counters there that's how ECP works that's how it keeps track of the bytes it's sending so it knows where we are in the byte stream right and then we get an AK and that's teach me saying hey I got some data and the acknowledgement number is three three eight nine which means I have received all the data up to and including seven four three three eight eight and the next sequence number I expect to receive from you is three three eight nine that's what that means and the next one the acknowledgement number increases so it received more data which is you know it's delay a little bit so it's acknowledging these packets up here and then we get the last act and it's acknowledgement number is nine nine five one which means I've received all the data up to and including seven four nine nine five zero and the next sequence number expected is nine nine five one so if we go back to the last one that was sent we expect its next acknowledgment next sequence number to be nine nine five one and it is what that means is there is no outstanding data on the wire it's been sent it's been received and acknowledged there's nothing outstanding the network did its job then there's a 10-second delay right and the receiver which is the bit central server because you know what we're done now maybe I would like clear the the filter and see if what else is happening in this ten seconds there's nothing happening right so now my question is okay this was the sending side we didn't receive anything no the the data was clearly received because we received the acknowledgments but did the cert to the other end try to send something I never got it well he captured from both ends good for him let's go look real quick we're almost done well no so this is the bit central server there's no guarantee it's the same TCP stream which was five so why don't we do the real quick search find this right and follow TCP stream oh wait let's let's make sure we find it get rid of that pesky space and find okay there's a message ID it's a replace we scroll down no response and this is from captured on the side that's supposed to respond and we can verify that that is the same packet by looking at the IP IDs and both captures it's the same so armed with this information you are able to go back and say this is not a network problem the network is doing a job there's no loss here the bit central server never responded and after 10 seconds it closed the connection there's the problem so he went back to the vendor gave them this information they said oh well you know what there is a 10-second timeout and it will close a connection after 10 seconds XYZ doesn't happen you saw that it was a you know a fairly lengthy ro replaced request with lots of information the server was taking longer than 10 seconds to process that information before sending a reply but the application had a timeout and enclosed the connection so what could you do you could say oh you need to spend several thousand dollars and upgrade your server such that it can process the request faster or find the other bottleneck in the application server we need a patch or you could just increase the timeout to 30 seconds and go on with your life and that's what they did so job well done Fidel good job being the hero making sure that it wasn't the you know showing that it wasn't the network [Music] that's our first case study thank you for tuning in like I said feel free if you there's a bunch of old stuff bunch of stuff on my website if you go on the website if you want to get emails about new stuff that I'll be doing on here obviously you can subscribe to YouTube and I guess you get you know like notifications if I'm gonna livestream but sometimes I send out I think I've sent out two emails in the last year anyway you can join the email list if you do you will get Auto fed like a mini course so if you don't care about that ignore it but that's the only way to get you know was to get on the list is to join that but you can email me at Kerry at packet bomb.com I'm on Twitter but I don't really do Twitter that much and if you wanna if you have a case study you will look at I've got a few more percolating the background I'll try to have one ready for next week a Content you want to see and I'm you know I'm happy to maybe go outside of Wireshark it's just that's what I know it'd be you know tres Wrangler was another tool I brought in you know we can look at other stuff I mean you know why not so there is I don't I just learned about discord you guys know about discord some child set up a discord server but you know kudos to him the Wireshark discord server I went ahead and reserved a packet bomb one but I don't know that I'll do anything with it so you don't know how you even get to it but I reserved the name but there is a Wireshark one hop on and I hope you guys are having a drink I think I saw Dan on earlier I hope you're not drowning in chicken eggs I met Dan through this and he lives in in the area we met up for lunch and he gave me twelve organic pasture-raised chicken eggs that were delicious but I think he may be drowning in eggs with the thing that's everyone's stuck at home so everyone he's drowning Hey yeah it's all your egg needs hit up Dan I think that's I think I've said everything I'm gonna say I'm gonna order some dinner and thank you for joining I hope you found this useful I'm always happy to get feedback and we will see you in the next one [Music] [Music] [Music] [Music] [Music] you

Channel: PacketBomb

Views: 1,468

Keywords: wireshark, packet analysis, tcp, networking

Id: 7gsgwZdHx1c

Length: 66min 22sec (3982 seconds)

Published: Fri May 29 2020