Case Study: All Web Pages Load Slowly

thousands of people turn to the internet for help because they have a burning computer and network problem that they cannot solve if you have a problem if no one else can help and if you can find them maybe you can hire the packet a team they either carry with Pekka bomb.com so I had Brandon on my email list you know we're chatting about packets as we do and he mentions he's got this site where users are having trouble loading web pages is extremely slow you type in the URL takes forever load it sounds like it could be you know DNS or something maybe but you do a dig or nslookup it's fine so I said send me a packet capture let me look at it so we did I looked at at that night send my feed back the next day email emails me problem solved so sometimes you know I kind of troll around sites where people are having problems and try to help them out but they disappear on you a lot of times you know there's all the Stack Exchange sites and whatnot and read it too sometimes so I'm more likely to put in some effort someone on my email list because I figured they actually probably care anyway so let's let's jump in so we sent me this capture and all I know is websites are really slow it's not a great place to start with data you really want to have it a little more well-defined a very specific use case so but you know what I'll give it a shot so what I want to know really is what am i dealing with there's what 63,000 packets if I go to say conversations and I look at TCP there's over 2,000 let's sort these by port oh boy there's a lot of port 80 a lot where do I start let's look at let's the HTTP and let's look at I don't know packet counter is that one okay I don't know great stat we've got oh we've got over 2000 requests and over 2,000 responses I so where do you start I mean I don't know there's a lot of stuff in here potentially so I poked around a little bit I saw maybe some stuff I don't know I said look you know what size did you go to and he said I I loaded NYX calm I guess like you know basketball Knicks so all right let's start with that so if I want to try to find a request for Knicks calm I'm gonna search for HTTP host contains Knicks okay that's probably it we can look at HTTP and its host Knicks at Tom Knicks calm get great there it is so let's do a follow TCP stream okay looks like it's a 302 moved temporarily to nba.com slash Knicks okay do you know what look if you saw in the previous videos you know I added this Delta column that shows you the delay between packets as they are displayed you know there's there's a lot of packets in it here but I filtered on just this stream so this Delta column is showing me the packets between these two packets so this is an eleven and a half seconds to get back so we we have the three-way handshake we have a get request this all happened instantly the now here's something I just noticed and I did not notice the first time you see this ACK actually you see the syn ACK he came back instantly that should tell you that whatever responded with a cynic is local I mean it did it in a hundred and what is this microseconds 187 microseconds and I'm sure Knicks comm is wherever Knicks comm is across the internet I didn't notice that the first time look at that so that raises makes me raise my eyebrows but we noticed that the actual response this is the first one back from the server destiny a source port 80 took 11 and a half seconds that's an eternity now you'll notice that this first one back from the server says TCP segment of reassembled PDU and then the second one is the move temporarily so what's happening here is you can see here to reassemble TCP segments it means the actual application data the TCP payload was split across two segments two packets and Wireshark is just putting them together for me now usually when I'm troubleshooting HTTP I want to turn a particular feature off because this HTTP 1.0 302 moved it's not in the second packet it's in the first packet I want to see it there so I'm going to right click on TCP go to protocol preferences and I'm going to disable allow sub detectors to reassemble TCP streams now that packet is the first shows up as the first one okay so that's is this the problem it might be eleven half seconds to get a response that's just a 302 that is questionable to me the second issue is why is a 302 split across two packets I mean it's very little data that seems strange so let's go find just go back host contains in BA if we can find the here we go get slashed NIC so here's after we did the the reader you know after we got three or two let's do follow TCP stream and okay we got a 200 in this one so there's the get the ACK now look at this crap fourteen over fourteen seconds to get the first packet back from the server so that's we got a huge delay for the 302 and now a huge delay for the for the this first packet that's that's horrible something going on and I don't even know if it was the same in luck the same destination IP I mean because it could be maybe there's something going on with that particular server but this is definitely a problem and I'm looking this again and we're seeing this weird this HTTP 200 ok it has 17 bytes in this packet that is if we click on the I mean that's just look that's just this string so it's just the response code is is in its own TCP segment and then the rest of the page that's weird I would not say that's normal to have just the response code in its own segment so now what let me so this column you know is Delta actually so let's look at and so now I'm curious about this this is the second time we've seen this the response split up with the very first one having just a little bit of data so let's let's get all the responses HTTP response ok and look these are all 17 bytes they're just including the response code same thing for this 25 bytes I bet it's eight bytes more than this one eight characters more so what about the delay for these if you look at this this column is showing me the the time between each of these packets which I don't care about they're all different TCP connections there are their own TCP streams this isn't that useful to me to know if this was delayed really bad like it has in the first two we've seen so what we want to do is right-click on TCP protocol preferences and we want to calculate conversation timestamps so what that does is it adds something to the TCP header we can expand it here's timestamps and this is essentially the Delta but it's just for the relative to a single stream so I'm going to do time since previous frame in this TCP stream right-click applies column so now I let's sort know let's resize this or edit let's just call it TCP Delta okay so now let's sort it oh goodness gracious look at all these eighteen seventeen fourteen thirteen all these seconds this is how long it took for this response to come back that's really bad I mean that's look at this all these double-digit responses holy crap terrible I would say this is and these are across all kinds of different source IPS source IPS over here and there are all kinds of different source IPS so it's not like one server that's slow there's something going on so again all these are truncated responses with just the response code in the first packet so you know at this point I'm thinking there's some kind of proxy or HTTP filter some kind of security device at the local site because that's that's just not normal and maybe it's having some problem and so it's introducing these delays now as I already mentioned let's look at one of these quality stream and let's resort by packet order now as that I mentioned I didn't actually see this the first time and if I had it would have nailed it for me but the local response time is extremely fast so there is something there's a local proxy of some sort so I mean the Senate comes back and you know less than a millisecond and then we have the big delay and then we have just a header in its own packet so those two things tell me there's something definitely going on so now I want to try to figure out anything else I can about this device so if I look so I wanna I'm thinking maybe I'll get lucky and this thing is layer two adjacent so if I look at traffic that the client sending it's sending it from Adele so this must be a Dell machine to a Cisco MAC address so probably it's default gateway I look at the syn ACK the source address is not coming from a Cisco it's from incoming from a cameo comm MAC address I don't know what that is never heard of it but there's some kind of asymmetric traffic flow which in and of itself is fine but that's a tip that's a clue so I want to look at this I'm going to copy the value and I'm gonna go let's go look it up cameo communications incorporated okay cameo networking built to order okay so clearly and it's some kind of networking company yeah I don't know if this is one of their products or just has their you know one of their NICs in it or whatever so I'm gonna go back and look at the other responses there was one that caught my eye earlier it was the 400 bad request I mean you know 200 okay that's cool 302s moved as fine 404s sure but a 400 and what's that about so let's check one of those out follow TCP stream and look at this and the title is sim filter I don't that is but that's a flag to me that sounds like some kind of filter some kind of HTTP device let's copy that let's go google it sim filter battle of the bids the none of that looks particularly interesting to be honest what about Dan's guardian that also looks promising ahaha true web content filtering for all last updated in 2012 so at this point I give the feedback to brandon saying look you've got you've got your HTTP responses coming back and I pack it by themselves that's weird you've got these long delays you've got some this traffic being sourced from a cameo comm MAC address and there appears to be a Dan's Guardian software in the mix here so he took the MAC address found the IP address found a filtering device that was forgotten about in a closet somewhere and recently at that site they had think upgraded or replaced a server that was also the local DNS server and everyone forgot about this web proxy so it was pointing at the old decommissioned server for its DNS server so it's it's DNS was actually timing out and once they updated the DNS server on it everything was fine so pack team back again check the captured let's begin whoop there it is yeah so that was cool problem solved guys could you figure this out another way yeah probably but if this was like 15 minutes of looking at a capture I don't have to like go point fingers and blame the network guys or blame someone else that was it so this is what I'm calling the the packet the the packet a team it's gonna be on the packet 18 - an email list we can talk about packets maybe if you have some time that you solved a problem with packet analysis I'd love to feature it on the website or if you want to work on a problem together and look at some stuff let's do that I'm all about it so until next time bump bump bump bump bump

Channel: PacketBomb

Views: 8,053

Keywords: packet analysis, case study, HTTP, wireshark

Id: KtIsj2KHBVw

Length: 16min 52sec (1012 seconds)

Published: Sun Aug 03 2014