- This was an example where
they were gonna get ready to forklift out a ton of
equipment on their network and throw a ton of money at it. And I'm like, "Guys wait, before we do that, let's find root cause. Let's make sure that we're directing our funds in the right direction and at least isolate the
box that it's coming from." So and then this, we were able to do with these examples. (upbeat music) - Hey, everyone it's David Bombal, back with Chris. A lot of you have given
a lot of great feedback about the TCP/IP deep dive video, and you're asking,
"Where's the second video?" So here we go, Chris, welcome. - Hey, it's great to be back David, looking forward to digging
back into some more TCP. Let's go deep. - So what are we gonna look at this time, is there something specific? - Yeah, so we were talking and looking at some of the comments, and I think a good place to chat would be around the maximum segment size. That is an option in the TCP header. One reason to focus on this is because it's just often misunderstood as far as how it relates to the MTU. So the network, I mean, I know I'm
preaching to the choir here, but the network has an
MTU and ethernet MTU, it could even have an IP MTU. What does that have to do
with MSS and how does TCP, how can we help TCP with
overcoming fragmentation or hitting its head on an interface along the
way, along the path? - So off camera we were
discussing something and you were saying that
Maximum Transmission Unit, or MTU's often misunderstood? Did you wanna talk about that a bit or should we just jump
straight into Wireshark? - Yeah you know, actually I think there's a video that kind of illustrates
what I'm talking about. So why don't I show that real quick? I'm just gonna bring it in. So this is a video that I
like to use in my trainings to talk about this whole subject. All right, so when we're learning how this whole process works, it's important to first think about how we can't just take any size packet and kick it out on the wire. And this is a video that sometimes I use
in my training classes, this is actually a bridge
that's in the United States and they call it the "11foot8 bridge" because this bridge is exactly 11'8" tall. And here's the problem, is that trucks will try to go under it and it's a can opener, Bam! It just takes them out. So this is actually a whole video that you can watch on
YouTube if you'd like to. And this happens enough to where... Wow! This is a whole thing, right? So what does this have to
do with network traffic and our subject? Well, really we can't take
a packet that's too big and just send it anywhere that we want. That is regulated, right? So first let's talk about
MTU and what that means. And like I said, David, I know you have a
lot of experience in this, so I'm actually interested in hearing how you can contribute to the
conversation for sure, so. - I'm the dumb guy. So it's all on you Chris, come on. - Stop, stop, yeah, whatever. Okay, Mr. CCI. All right, so basically, ethernet, at the ethernet layer we have, and let me just draw this out for you. All right, so if I take ethernet, right? So I have my destination
MAC, source MAC, EtherType, those are the fields of an ethernet frame. And then I have this payload
and that's just my data, right? Oops, sorry so wiggly. And then down here I have my FCS. This is four bytes FCS. This is data, and then we have, dest MAC, source MAC, and then EtherType. Or this is actually link type. So another conversation. But anyway, so this is
where a station's gonna put its destination MAC address
or its source MAC is here where it's going is destination. But this data field, this is
where this is variable, right? It depends on the amount of
payload that I'm sending. Keep in mind, ethernet can work with a lot of different protocols. It doesn't have to be IP. It could be, IPv6, it used to be... There are some other competing protocols at layer three that were
really, really common. Data, this is a variable field. So basically, in the ethernet standard, it was decided, why
don't we take that data and limit it to 1500 bytes. Now this is actually something
that made ethernet take off in the early years when
there was a lot of competing Layer Two technologies is because ethernet, it was efficient and that it could carry 1500 bytes and it could do so with just
an 18 byte overhead, okay? So four bytes here, if we add all this up. So I've got six bytes of destination MAC, six bytes source, and then
this EtherType is two bytes. So that's a total of 18 bytes of overhead to carry 1500 bytes. That's not so bad. So this is why a lot of times ethernet, you'll see on networking interfaces and routers along the path, you're often gonna see that 1500 and that's a good thing, right? That's a good amount of
data to be able to carry across the link. Now keep in mind, this
is transmission unit. How much can I send? Well, in the days of
ethernet where you had, I got ethernet and inside this data, I've got IP, let's just keep this simple. In fact, let's just draw this again because I don't like that. All right, so I just cleared that out just to be able to take a look
at the data field now, let's just imagine that this
is just that payload, okay? Now let's take a look
at the first 20 bytes. Let's just imagine this is its IP, okay? So now this is 1500, from side to side. All right, that's 1500 and IP is going to add 20 bytes as long as there's no options, and then the next part-
- We have to say IPv4, right? - Oh, good call. Thank you, IPv4. If it's v6, then you got 40 here. So let's just work with v4 for a moment 'cause let's go back
in time when you know, we didn't have so many
different types of fields and header values that
we need to consider. Our IPv4 and also the TCP
header without options is also 20 bytes. That means TCP has a
maximum amount of data that it can carry for a payload and still fit within the MTU is going to be 1460. That's why a lot of times, if you're looking at your TCP handshakes and you see a SYN, and you
go down to that MSS field, we've done this on previous videos. You're gonna see 1460 a
lot of times advertised is an initial MSS, okay? What that's saying is, Hey, David, I'm gonna connect to you. I'm a client, you're the server. I can receive 1460, okay? So you can send me that. That's another reason why MTU and MSS are a little difficult sometimes because MTU is Maximum Transmission Unit, MSS is Hey, David, don't
send me a payload any larger. It's kind of like a maximum
receive unit if you will, but that's MSS. So this is advertised in the SYN. It's only advertised once. This is just a beautiful
blissful, perfect world where there's no other protocols.
- No VPNs. - No VPNs, no GRE tunnels, nothing else, no PPE over E or any other type, or like you just mentioned
one, no other IPv6, no other anything else
that needs header space. So this is where protocols
start to go, okay, let's roll back our sleeves, and now we're gonna start
fighting for that 1500 while still having a
payload that makes sense. This is where TCP can become involved. And this is where we can take this MSS that gets advertised in the handshake. And we can actually bring this number down so that we don't bump our heads, like the 11foot8 bridge. We're not going to smack
it ourselves into an MTU that is lower than the
actual full packet payload. How are we doing so far? - I think we just need to emphasize MTU is per hop, isn't it? So if you go across like a
10 gig, 100 gig interface, the MTU could be a lot higher, but if you go across, like we said, some other kind of interface, the MTU could be lower
and that's also a problem. And I don't wanna jump the gun, Chris. I'm hoping that that's where
you're gonna explain MSS and you know why this is a nightmare. - I think that's a great call. In fact why package tracer,
why aren't you coming up? So, yeah, actually, why don't I talk about that? - Chris Greer actually uses packet tracer. - Yeah, he use packet tracer. I mean, but here's my beef with it though. Can't capture and export PCAPs out of it. So points for GNS3, points for GNS3. All right, where are we? So let's go ahead and go into our MTU. So we were actually just talking David, so this interface, this is just a basic
network, we got two routers. We got a router here,
we got a router here. Now this is where, and don't worry about my
sniffer in the middle, okay? So this is where we could
have an ethernet interface on one side, we could
have a 1500 over here. And in the middle we could
have a different MTU. And that's where, like I said previously, if we just had a flat ethernet network with no other considerations,
it's 1500, 1500, 1500, 1500, this conversation isn't as complex. The complexity comes when we start needing to have different MTUs along the path. I'm just gonna say 1436, or we have to start accommodating for other technologies, such as IP tunnels and GRE, and IPSec. And if I have take another type, like a Point-to-Point
Protocol over Ethernet, that's gonna borrow eight
bytes from the payload. Those technologies, depending
on what I have in the middle, that's going to reduce
the MTU on the path. - Yeah, I mean, it becomes a nightmare. Like I was just thinking, and we don't want to take it too extreme, but like VXLAN, IPSec, GRE, like you said, PPP over ethernet. I mean, there's so many of these, and then you get the crazy
stuff where it's IPSec over GRE, and then over MPLS. You know, how many times
do you wanna like do this? So you got all these
different technologies adding and stealing if you like from
that MTU so bit of a problem. So,
- Yeah, absolutely. - I don't wanna jump gun again, Chris, because I remember, I'll
just tell a war story while we're here, I remember...
- Oh, war stories from David Bombal everyone, war stories, let's go, let's hear it. - I remember doing like
VPNs years and years ago, and it was a nightmare because I mean, GRE, IPSec,
VPNs across the internet for many, many sites around the world. And it's like, okay,
you've probably experienced this as well, where a
ping works, ping's great. But people complain that
the network is slow. So my pings are great, network's fine. But nothing goes through and it's exactly this problem, isn't it? Because large packets are being dropped. Sorry, go on. - No, you're fine. I mean, I would never wanna
jump in on a war story from David Bombal, that's exactly right. But what you just said is
exactly why a lot of times, for me, on my side, being
coming from the packet world, that's when I would get called,
pings work, the path works, things are still slow and why? And a lot of times that can happen even if somebody doesn't realize it, if they migrate to a
new technology is easy and add one thing, but it breaks another. And that's why I'm super
passionate about this topic because you gotta understand
what's happening end to end. See David and I, if you
and I are communicating, especially over TCP, we have zero knowledge of
the network underneath it. - I mean, you're in
California, I'm in the UK. There's I mean, how many hops is there? There's a lot of hops,
different technologies, we have no visibility of that. I remember this, I remember reading a
document on Cisco's website, talking about MSS and MTU took
me a while to understand it. So Chris, hopefully you're
gonna make it simple for everyone watching, no pressure. - Yeah, yeah. No problem. Let's do it, I'm up for the challenge. So first of all, let's just... How is it that I've
gotten so far into a video with you so far, David? And I haven't had Wireshark open. - Yeah, I'm a bit
worried about you, Chris. - Yeah, I'm falling off. I'm falling off my game. All right, everybody. So it took me way too long to
get to this PCAP, I'm sorry. But here we are. Okay, back to the fun and
let's go ahead and do this. You can kind of see what
I've done with this profile. I'm in my TCP playing profile
and I'm just gonna take a look at this first packet. This is a SYN and I'm
going to come down into the handshake and take a look like we did on previous videos. You guys already know
this, this is old hat now. We come down to options, maximum
segment size is 1460 bytes. Okay, in a perfect world
with maximum capability, this station can receive 1460. 1460 plus 20 byte TCP header,
plus 20 byte IP header, no options, that gives me 1500 in a beautiful
blissful, perfect world. - And the reason for that is just make sure everyone understands, this PC has an ethernet
network card in it. And that's why we get to the 1460, yeah? - So 1460 will be the payload, but TCP is trying to stay under the 1500. Which is the card, yeah. The cards like TCP goes,
Oh, we're ethernet, cool. - I'll just say this, Chris. I dunno if you can bring up your Mac, sorry to take you on a tangent. So if you look at your
Mac ethernet settings, like network preferences, on your Macs, on your WiFi connection. - Oh yeah, sure. - So if you go to network preferences and you go advanced, hardware, so we can see the MTU 1500, yeah? So if you go manually. - Good call. - I think you can set that to manual, if you go to the MTU
there, you could set that. Like if you had a 10 gig card in your Mac, you're gonna want to up that because the MTU on like 100 gig or 10 gig, or 25 gig interfaces, you're gonna wanna make
it a lot bigger than that to get the throughput,
otherwise it's too slow. - Yeah, and you have to have a full... You gotta have a network that understands everything in the middle. Yeah, that's a good point. - Yes, I mean, I need to post that video. I'll link it below once I've posted it, showing you how I connected
a 10 gig interface to a Mac, and then the throughput is
low through the switches until you up that. And I mean, there's lots of
videos like that on YouTube. - So cool, honestly, that's new to me. I didn't know you could change that. Wow.
- Yeah, you have to because otherwise you'll never get your
like 10 gig full migrate. - And does TCP change
its own MSS after that? - So Chris, just remembering
what you spoke about like 1460 and 1500 because TCP recognizes the ethernet MTU. If I increase the ethernet
MTU, then the MSS will adjust. So if I make the ethernet MTU 10,000, then the TCP will be bigger
based on the headers. - Yes, and that actually gets us into jumbo frame environments. Where we have larger
connections or we have a system, end to end that understands
we can use larger MTUs. So that's where we wanna be careful just because I send out,
if I had the capability, just because I can send
out a 8,000 byte frame, what if the next switch
next to me can't accept it or the next router can't
transmit it, right? So at end to end, we have to have that understanding
that jumbo frames work. So that's just, but typically in most
environments end to end, you're gonna see 1500 as an MTU. - Yeah, I mean, Chris,
just for the real world, I mean, you are troubleshooting
client networks, not always in a data center. So I mean, those large,
large frames are typically in a data center, aren't they? Even though it might change now because people are
getting gig to their homes and even high speeds these days. But I mean today in the real world, it's typically 1500, yeah? - Yeah, absolutely. And that's gonna be a limiter for a time. But one day they'll go, you know what ethernet
was written in the '60s so let's go ahead and move forward. (David and Chris laughing) We'll find out. - So Chris, I just wanna make
sure that everyone gets this. What is the difference between MTU, Maximum Transmission Unit
and MSS Maximum Segment Size? Can you just explain like very quickly, what those two and the
differences between them so that everyone gets that piece? - For sure. It depends on basically the
layer we're talking about. So let's go ethernet Layer Two. How much data can it send out on the wire? What is its Maximum Transmission Unit? So a lot of times you're
thinking switches, you're thinking router interfaces. What is the Maximum Transmission
Unit for ethernet MTU? You can also have an IP MTU, but most of the time we're
talking about ethernet MTU. Maximum Segment Size, see, keep in mind that ethernet deals in frames, IP deals in packets,
TCP deals in segments. - [David] That's correct, yeah. - So when I get a huge drop of data down from an application, it
just goes thunk down from, oh, here, TCP deal with
this, and it's 10K. TCP looks at that data, it's a stream based protocol so it goes, how much can I put in each segment? How much of this 10K can I chop up? It doesn't really care,
but it's gotta know, I don't wanna make
anybody upset down below with the ethernet interface. So it's gonna work in tandem to figure out what do we gotta work with? No? Okay, 15? Cool. All right, so that tells
me the maximum segment size that I can put as a payload
within the actual packet itself. Now, just to make sure
everybody's clear on that. I'm just gonna draw out a whole frame. So this is the ethernet component. Ether, keep in mind there's headers. So ethernet and let's just
deal just for simplicity, IPv4, we'll tackle IPv6 on another video. So I got ethernet, IP, TCP, and then I have my actual payload. Now TCP does not care
what this payload is. All right? It doesn't care, any application. This is MSS. The maximum amount of data that I can carry within a payload. Does not at all include the TCP header, the IP header or the ethernet frame. The M- so right here. So this is, sorry. Let me go here to here, that is MSS and that as a maximum, it can be 1460 that's because the minimum
TCP is gonna be 20, IP 20. The ethernet frame is 18, but that's not even
accounted for in the MTU. The MTU is from here to here. So ethernet goes, my payload is 1500. That does not account
for the ethernet frame. Any VLAN tags, any other Layer Two stuff that might be appended after the payload. So the ethernet payload is the MTU. The TCP payload is the MSS. And that's why these two concepts can be a little confusing sometimes. Okay, so since we're talking about one gig or less interface and that's what I'm dealing with here. So I'm talking about standard ethernet. I gotta fit it underneath
1500 bytes, that's my budget. So TCPs going in a perfect world, IP is 20, TCP header is 20, that gives me 1460 for a max payload. And again, this is an advertisement of, Hey David, you can send me 1460. Okay, so let's take a look
at the opposite direction. I'm not gonna look at any
of other options for now. Let's really focus on MSS. I even added this as a column here. So 1460, the server in the
opposite direction says, cool. I can also do 1460. Now, keep in mind, this is
not a negotiated number. The server could say 1432, 1400, 1300. And that just means I
will send 1460 to you, but you cannot send 1460 to me. You gotta send 1300 of what
or whatever I advertise, not negotiated. This is an advertisement of the MSS. This has never repeated
again, this isn't a handshake. So this is again, why it's so important
to capture a handshake? So let's take a walk down this PCAP. This was a PCAP that was given to me by a friend named Vladimir. He's got a really cool blog, maybe we can link that down
below about this exact problem. And the reason why I like this example is because of what I'm about to show you. And that's where two endpoints can actually do something
called Path MTU Discovery. Let's learn how that works. Okay, so we're basically
transferring a file. I'm not gonna worry too much about the NetBIOS Session Service
and the application up above, we're just moving a simple file and it's not until, let's keep
our eye on this segment size. If we come down here,
we're gonna notice... All right, ignore that for now. Just trust me. Okay, boom! Big packet, big packet,
big packet, big packet. So right here, the server
is trying to send across, I know it's the server
'cause its source port's 445. He's trying to send a
1460 over to the client. 1460, big packet, big packet, big packet, then less than big packet. So this is just a block of data. Right away, we get this
destination unreachable, fragmentation needed. Whoa, okay, wait a second, hang on. Let's go back up to this first packet. This first big one. And if I come down into the IP header, if I take a look there, it
says, "Don't fragment is set." What that means is, Hey
router, don't break, or any router, or any
device along the path, do not fragment. Don't break this up
into different segments. IP can fragment. That means if David, if
you send me a huge frame, I can break it up into smaller ones, add a new IP header to each one and then send it along its way, it goes all the way to the receiver, the receiver then pieces
it all back together and then ingest all that data. I have an IP fragmentation
video on my channel, if anybody wants to get a
bit more into how that works. Okay, so DF was set. This means don't break it up, okay? And in fact, in a TCP thread, most of the time, all
packets have the DF bit set 'cause TCPs like, I
don't like fragmentation. I don't wanna fragment. Fragmentation does a few things. First of all, I gotta bother a router with splitting things up
and adding a new header. It's not as efficient. Now my end point, David, you now have to bother with
reassembling everything. So if I'm dealing with a
fragmentation situation, my throughput goes down because now we have to add another process like fragmentation. So TCP wants to be
efficient, don't fragment me. But we hit this router here, this is 1.1. Let's just check this
out, let's inspect this. Okay, so IP header and
let's take a look at the ICMP message. Destination reachable,
fragmentation needed. Uh-oh, we hit our heads just
like that bridge, right? We hit an 11'8" bridge. But this is what's interesting, in this packet, this is where
the router can advertise the MTU of the next hop. So this is called Path MTU Discovery, where the router is saying, Hey, 1300 is what I have next. And you just sent me a 1460
without the extra big headers, you actually sent me a 1500 byte packet. I gotta break this thing up. I gotta take this payload, snap it up and I gotta add some new headers, send this along its way. So this packet goes back to the sender. Let's take a look at the next packet. So now the sender goes, Oh, I ran into, remember, let's remember that 1350, next packet. Okay, so the TCP segment
length here is 1310. So now we take 1310, we can
add a 20 byte TCP header and a 20 byte IP header,
now we're at 1350. So the sender got the hint. He reduces it, he sends it. That one gets through or we don't see that retransmitted down here. And if I take a look at the next packet, now this has a TCP signal length of 150. Well, if I add those two things together, I'm back up to my 1460 bytes. So now I just took those
1460 bytes, split it off, 1310 in one, and 150 in the other, and then that data can move along. And I noticed that the
sender did the same thing with those other large packets. So that is Path MTU Discovery. - You gotta remember the... It's been a while since
I've done that years ago that I had that huge issue with the VPNs, but Cisco devices allow
you on a tunnel interface as an example to type IP
path-mtu-discovery or something. I'll put the actual command below because if you don't do that, the routers just drop it, don't they? Then they say, sorry, tough luck. And I dunno if this is what you found on your like troubleshooting
war stories, Chris way. If someone doesn't do that on a tunnel, so let's say they put it across
a VPN tunnel or something, that's where the pings work. But all the traffic, well,
you can't send traffic through because like if you op- the typical issue, I've seen, pings work, but you can't open up a website
or a website is really slow or something like that. Wherever you're sending
larger packets, it's slow, but the pings are fine. And that's because the routers
are not communicating back to the clients that Path
MTU Discovery is available and the routers are not
communicating that back. Is that some stuff that you've seen, yeah? - Oh yeah, what you
just said is exactly it. First of all, the router has
to be capable of doing this, advertising the next MTU. Second, that needs to be enabled, so it needs to generate the
thing in the first place. And then that ICMP message needs to make it back to the endpoint. So if a firewall or some other device in
the middle could drop it and then the endpoint
needs to react to it. - So our endpoints, currently, Chris, you're doing this every day. Windows is that the default is
MTU path discovery is on now, is that right? - It should, yeah. And the reason why I say should
is because I haven't seen it actively in the wild, in a little bit, it's been a while since
I've seen a PCAP like this, so that's why I borrowed
this one with Vladimir 'cause I think it teaches
the point very well, but as far as in the wild, I just haven't seen do TCP stacks on these different operating systems, still react to Path MTU Discovery, okay? So yes, in theory, have I actually seen
it at the packet level? It's it's been a while since
I've seen the packet level, but back to what you were saying, one thing that can trip people up, especially when talking
about MTU versus MSS, let's go back to the ping example. All right, so you mentioned, pings work, but anything else doesn't. Okay, so why is that? Well, okay, let's just do this. Let's just ping David Bombal. - [David] It's a bad website. - (chuckles) All right, we'll look at it from
my perspective, right? You know, David Bombal, he's a whopping... Actually, I'm gonna stop this. What can we learn here? So first of all, if I bring this in. All right, boy latency, but that's not your fault. Also TTL, okay. We learned in previous
videos that gives me a guess, as far as the number of
routers I'm going through, looks like I'm going
through 10 routers, right? I'm guessing you started at 64 and here's my 64 bytes. So 64 bytes was put
into a TCP or I'm sorry, an ICMP packet. Basically, it's a very
small amount of data. That's a lot of times why a
ping will work in this scenario. We're not really stressing things. So something that I'll do and we'll just come in,
and we'll start to tinker with that length field. So let me just clear and
you can do this on Windows or Mac, or Linux, any of them. Let me just do a ping, but this time I'm going to change, this is basically the packet
size, and let's do this. Let's make this thing 1500 and I'm going to ping davidbombal. And you know me, I can't not capture this
while we're doing it. So just 'cause it's fun
to learn side by side. And this is what I
encourage everybody to do. I mean, do these tests on your pings and these types of
things that we show you, but capture it at the same time. 'Cause that's when you
can really understand the flows and what's really happening. So I'm gonna come in here
and I'm going to go ahead and do capture. I see some traffic coming
in and what I'm gonna do is I'm just gonna say, ping -s 1500. So that sets the packet size
or weight does it, to 1500. This is what's tricky. I get a lot of people
saying, "Oh goodness, okay. Let's just see if stuff works." Now this does work, but let's actually see what happened. Let me come in here, I'm
gonna stop, gonna do ICMP. And if I come over here, I can see my total length
is 1500 bytes, wonderful. Okay, expanding IP. And if I come in here, let me see where my total
length is 1500, so wonderful. But 1508, so let me actually just remove, I'm just gonna remove this and I'm gonna take your address, David. And I'm gonna drop this
up as the destination and actually flip the
direction, just call it addr. So now I've got both
directions of traffic, but check this out. I got one packet that's 1500, the total length is 1500, but I had eight bytes of ICMP, right? So this still was a big packet. In fact, the other data that I sent. So this is actually 1500. Now keep in mind, that's
the maximum packet size, take out 20 bytes for the IP header and then I need eight bytes for ICMP, so in that first packet,
I'm only able to send 1472. And then the next packet, I'm able to send the rest of
the data, the other 28 bytes. And that gets me up
to, this is why I said, this is not an advertisement of the actual overall packet size, this is the amount of payload that I wanna send to davidbombal. So my system fragmented it because it had to, because of the NIC limitation of ethernet, that's what I'm going
over from this system. Let's do this. I'm gonna add, what I can do... So I can do -D. Now, if you're in a Windows system, you use -L to set the
length and you use dash, I think it's f, like lowercase f to do, do not fragment. So what I'm doing is I'm
telling ping, send a ping, but don't break it up. And what does my system do? It goes, whoa, wait a second. Right, so you're trying
to send 1500 bytes, Chris, like let's back up. 1500 bytes with an ICMP
header, which is eight and then you wanna send another 20. So buddy, your 28 bytes over budget here. We can't even send that
out the front door. - Yeah, I mean your local
computer's doing that because the network card is, as we looked at through
the preference is 1500 and you're trying to squeeze too much in, so immediately it's kicking
it back to you, yeah? - Correct. And I don't even see a
single packet out here because my system's going, nope. Now let's try this. See, this is where I'll talk to my clients and they'll go, "Oh, we have an MTU problem because we can't send 1500s." Remember when you're pinging with a 1500, that does not take into
account the IP header itself. That's why up here, 1508, that does take into account. It's advertising. It's saying, Hey, I added eight
bytes for the ICMP header, but this gets put into an IP packet that then needed to be fragmented. As soon as we added the
do not fragment is set, that's where we got in trouble. So let's do this. - If you said it just below is I dunno if sorry, I'm
jumping the gun again, Chris. Like whatever it was, then
it should go through, yeah? - Well, let's try it. Let's try 1472, do not fragment is set. And I'm gonna come over here. Let me just start my capture again. I'll leave that filter
just in case I actually, or your IP doesn't change
on your site depending on where it's at. There we go. Okay, so we're seeing
some traffic go through and let me go ahead and stop the ping. We're gonna come back here. Let's go ahead and take a
look at our first packet. So this is don't fragment. The total length is 1500, but if I come down here, the data itself, so 1464. So if I add my eight bytes,
that's gonna bring me to 1472, which I said I wanted to send. So I basically threw 1480
out there on the wire minus the IP header. If I go one byte more, 1473, then message too long. Sorry buddy, you're over budget. You're one byte over budget. Okay, so you wanna see a cool trick with a Mac system? All right, so if you're not sure, kinda ran into this one the other day. You can just do a range
of MTUs on a Mac system. Honestly, I haven't tried
it on a Windows system, so I don't know if you have David, but. All right, so basically I'm saying start at this range and then stop at this range. So that's the low g, big G and you can look at the help menus and it's all in there for you. Let's do 1480 just for grins. I'm gonna say, do not fragment. Actually I need to say,
what's my interval. How much do you wanna
go up for every attempt? And then I can do -D and then I can do, that's do not fragment, davidbombal.com. Boom! So check us out. 1476, 77, 78, 79 80. All right, we hit ours ceiling. - That's nice. - So that tells me now, but let's remember something here. This does not take into
account our IP header. So we're still good at 1500. On the wire, our MTU was still 1500. The reason why I wanted
to show this to everybody is because I've worked
with a lot of customers where they'll come in and they
will basically ratchet down their MTUs locally because they think that
the network in the middle can't carry 1500 because their pings stop at 1480. And I go, whoa, hang on 1480
doesn't include the IP header. So you're still good at 1500, no reason to drop it. Now, this is where we're
going to get into the topic of TCP MSS clamping. - To the client thinks they
need to set it to 1480, is that what you said? - Yeah, so the reason
that this is relevant is sometimes I'll work with clients and I'll see them actually
changing on systems, local TCP MSS values. - On their client PC, so they go to like a thousand PCs and they try, and change it, yeah? - Thank you or they do it on the router. - Okay, yeah. - They'll tell the router to. So I'm gonna get into an
example in just a minute, that's getting into our war story. So the router can actually
reach up when it sees a TCP SYN and it can change an MSS. So that way, I don't have to
go to all these other systems and change the MSS. If I have a GRE tunnel, if I'm doing any kind
of point-to-point over E or anything that's going to
borrow header space upstream and I need to reduce the
payload instead of going to every system and saying, Hey, TCP, MSS, let's reduce this. I can do it on a router level that way every SYN that passes by, I just reach up and change the MSS. I'm sure that's a value
you've seen in the while too, or have set yourself. But as far what to set it to, sometimes I'll be working with
clients that have set 1480 or 1472 on their routers and say, "Well, we have to change this, we have to clamp TCP because we have an MTU problem." And I say, "Well, what gave you? Why'd you think that?" And they go, "Well, look at my pings." It stops at 1480, that means I have a path along where- I have a router along the way somewhere that I'm hitting my head under. I'm hitting my head with 1500s, so I'm bringing it down to 1480. This is where this is misunderstood though because when you send a large ping and this is the takeaway point, if so far, if you're swimming
in detail a little bit, don't worry, capture some pings and check it out in Wireshark. But just take this away, the ping output will only
show you the payload. You always have to mentally add 20 bytes to these numbers you're sending. That's IP...
- That's the way it should be 1460, that's
what you're saying, yeah? - Yeah, so for example,
getting back in the Wireshark, let me flip it back now, let's flip back over to that PCAP again. So now if we get back into our PCAP, we understand a little
better why it's very common to see TCP SYNs with a
maximum segment size of 1460. We need some space, TCP and IP. Now that's in a perfect world. What about when TCP has
some options in a packet? So for example, time stamping or if there's a packet that's lost and I need to do
selective acknowledgement, those are topics we're
absolutely gonna be hitting in the TCP deep dive series,
don't worry everybody. But those are things where TCP header might increase in size. It could be, I have a
TCP header of 32 bytes because I have some options
that I have to make space for, that will affect my payload. The system accommodates for that. TCPS gonna go, Ooh, I
have a larger TCP header, I can't put 1460 in this thing. Let's go ahead and reduce that guy down to 1420 or whatever it is, that way I have some more
room for this header. And I'm still staying under my bridge of 1500 bytes for the ethernet MTU. - Yes, so that's on the local device. And are you gonna talk about MSS, like configuring that
on a router or something to advertise that back to the client or is that a separate conversation? - That's getting into my war story. - Okay, great. I keep going to the war story. - All right? I've kept
you hanging long enough. All right, so let's do this. Okay, so let me set it up here, guys. Okay, so let's go ahead
and talk about the problem that was experienced. So first of all, let me
just set up the complaint. It was exactly what you just said, David. Ping's work fine and we
try to open up a file, and it takes forever, or we try to pull a file
down from a shared drive. We try to open up and retrieve emails, anything we do initially, we just see this like
spiny wheel for a time. And then after that it's
laggy, but it eventually works. And those are the tricky ones. Not broken, if things are flat out broken and there's a broken path, that's easier to resolve, then it's slow. And especially, Hey, pings work, so the path is okay, so to speak, but moving data isn't. All right, so let me just
build this scenario for you. This was the client's side of a capture that we took and I went
ahead, and I scrubbed off. There's no data encapsulated, I changed the port numbers
and changed the IPs. So literally, basically the timing
is all that's preserved and some of these other values. Okay, I'm actually going
to hide this for right now. Let's go ahead and come up to that SYN. So the SYN, if I come down here, I'm gonna take a look at my options. I got maximum segment size
1460, window scale eight. So I have a large TCP
window received buffer that I can work with,
and we're using SACK. Awesome. Okay, like muscle memory, as we're start working through Wireshark, I want everybody to start to do this. It should become, start
to become automatic. I'm sitting client side because here's my latency is
between my SYN and SYN, ACK. SYN, ACK comes back 86 milliseconds later. That's my network round trip time in case, so I was able to hit my endpoint, port 80. And he comes back, let's
check out that SYN, all right. Maximum segment size is 1432. Okay, that just means,
I have to send 1432, but he can send 1460. Widow scale is zero, this is
just a trick on server side, especially if a server is sending
data and not receiving it. It's just a way of activating this option without it actually needing
to multiply to a huge window. This is something you'll see on servers kind of another topic, but basically it just means yes, you can use window scaling,
David, that's awesome. SACK permitted and then yes, we can do selective acknowledgement. Cool, so there's our handshake. Let's come down. Client sends 350 bytes to server. The next thing that happens just around my network round trip time, I see 93 milliseconds later. I see data come in, but previous
segment is not captured. What that means is I had
a jump in sequence number. So we learned about
this on the last video, we were talking about sequence numbers and how they increment. Basically, look at our jump. So this sequence number started at 1461 and we're sending 563 bytes. So the next sequence number will be 2024, but this one is 1461. I send this to you,
David, you ACK back 2024 so that I know you got my 563. That's what the server's
trying to do here. But wait a second, time,
time, time, time, time, 1461. Where was the 1460? This is the first packet we
see come in from the server. So we go as a client, we go cool. ACK one, because I didn't get that 1460. If I come down into my SACK field, 1461 to 2024, what that means is I got your 563 bytes,
but I'm missing 1460. You gotta fill in the sequence numbers between one and 1461,
where was that big packet? Right away, that was an indicator to me that we were running into an MTU problem. - So someone along the path dropped it. - Correct. Somebody along the path dropped it. Well, this is where things get fun. 21 full seconds later, this is sequence number
one, 21 full seconds. We sat around spiny wheels, spiny wheels, spiny wheels, spiny wheel and
then all of a sudden, boom. And then things start going. Looking at the packet level, hold up, 536, 536, 536,
this is from the server. I'm capturing client side. We wait 21 full seconds
to start to get this data. What on earth happen in 21 seconds? It's a long time. I don't like waiting. So let's go ahead, well, you don't like it. Wait, you don't like waiting either, we are impatient folks these days. So let me come into, this is where I said, okay. And this is getting into the point, everybody of multi-point captures. From a client perspective, I was able to get some clues off the wire that we were looking at an MTU problem. Something maybe even with MSS clamping, this continued to work, but now I realized, hang on a second, we only have part of the
picture from the client side. Keep in mind the server side, it's experiencing something else. It's doing something in 21 seconds. Let's go find out what. And this is where we come
into the dual side capture. So what I'm gonna do, I'm just gonna move this guy over. I'm on a Mac system, which makes it a pain to start up two applications
at the same time. So I'm just gonna do open -n
/applications/wireshark.app, which I should turn into a little... I'm lazy that's why I haven't done that. Anyway, let's go ahead and
drop in my server side PCAP, let's do this. Now, okay. Let me just resize, clients on left. Server's on right. This is as real world as it gets. If people show me that 21 seconds, I say, "I gotta get a client side trace. I gotta see what's going
on the other side." For right now, time doesn't really matter. I'm gonna just pull this and pull this in. Actually, you know what? Let's take a full look at
this before we go too far. Now the first thing I thought
was just fun and weird is that big red line there. That's helpful. So SYN IP total length is 64. Well, the complaint, if I expand this out, IP says total length
exceeds packet length, which is 52 bytes. The actual, that's 52 minus
the ethernet frame itself. So the true frame size was 70
that was on the wire itself. 'Cause we can see up here, 66 bytes on the wire,
Wireshark doesn't account for the FCS, that's the
four bytes on the very end. It just says, okay, there was 66 bytes that's including the ethernet frame, but Hey IP, you're saying that
you have 64 all on your own. So as soon as I had 18, this is broken. Here's what's weird. If I take a look at the original SYN, look at the IP header, total length is 52. When it was received on the server, the total length was 64. Client lets it go at 52,
server receives it at 64. And this is the same IP ID, 4081, 4081. Someone's changing
something in the middle. We get that sense? Yeah, like what's that about? Okay, so there's a clue. Let's just keep going. It still worked. So our SYN still got there
and the SYN, ACK was returned. So that's why another tip for
the listeners or the viewers, if you see some weird looking things, some flag and things keep
going and it was quick, that's probably not where
you're troubleshooting, right? Like it's a clue that you store away, but don't get too hung
up on every little flag that you see in Wireshark, otherwise you're never gonna
get past these black lines and red letters. Sometimes they're involved,
sometimes they're not. The thing that I look for is the complaint was slow. So I gotta go find where slow is. And down here, if I look at my Delta time, whoa, that's full seconds. This is what they called me for, okay? So SYN, SYN, ACK. And if I come down to my... Let's take a look at our options. So we have our MSS came in. All right, so MSS can came in. MSS came in, 1460 bytes and I
return it and check this out. 1460 bytes, hang on. Server let's go of 1460, but I receive it and 1432 noted, filed away. So let's keep going. So SYN, SYN, ACK, this is that 350 that
comes in from the client. Then right away, the server sends a 1460. It almost immediately. This 1460 starts at sequence number one, this 1460 never makes it
over to the other side. It hammers two packets
out there, 1460, 563. The 563 made it. That's the 563 over here, the 1460 didn't. I can see that side by side. And ACK comes in from the
client, the server goes, it waits, basically this is a
full retransmission timeout. This was three seconds. Now TCP has evolved since a little bit where it's not gonna wait quite as long, but at this stage of the game, we're still building our, it's
called a congestion window, and also our retransmission timers. The server still trying to figure out, well, I don't wanna retransmit too quick 'cause what if this
packet's still in flight somewhere in the world? So it waits three seconds
and then it sends a 1460, it retransmits. Well, now it goes into
its doubling mechanism where Hey, after three seconds I tried, let's go ahead and try again, but I'm gonna double the time. Six seconds, huh? Still nothing. This is called basically an MSS black hole or an MTU black hole. We didn't get any ICMP
data back from the network. That nice ICMP message that
was fragmentation needed, wasn't coming back. The Path MTU Discovery wasn't coming back. That was either disabled
on a route somewhere or a firewall was blocking it. Probably disable on a router somewhere. So the server had no recourse. It's like, okay, 1460, 1460. Now this TCP stack did
something interesting. This is where the server
goes, you know what? Let me just try 536. Now 536 is an interesting
number that it tried. This is basically the minimum, I'm gonna try to say this well, David, this is the minimum maximum
that you can receive as an IP station. If you're an IP station I expect because if go look it up on the standard, you should be able to
receive a packet of 576. At at least. And that's in the IP standard. - 536, yeah? - 576. - Okay, 576. - If I take off the IP header, 556. If I take off the TCP header, 536. - I'm glad you said that, yeah. - So this is TCP going, this is basically the last ditch effort of trying to get data across to you when you're not accepting a large packet. The station was like, here's the minimum max packet
that David can receive, let's just try it. So he sends a 536 and it worked. Whoa, he got that packet. Okay, let me try this one. Let me try this one. He got those. Whoa, okay, here's the rest of my data. And now for the rest of this, see 1432's are coming across from you. But now for the rest of
this entire throughput, for the rest of this connection, I'm scared to try
anything bigger than 536. - So you basically slowed
the whole thing down, is that right? - It slowed the entire throughput after we overcame this
retransmission time. So now going back to client, this 21 seconds was burned because we were waiting
on three retransmissions from the server with a big packet. Finally, the server was like, 536. Whoa, that worked. Okay, you have a very, very... David, you have a very, very low... It's either an MTU or
you as an end station, but we're hitting our heads here. So let me just drop my, you know I'm tall, so this happens sometimes
I go through a door. It's like, Oh, let me duck under that. Problem is though, is that once that happened, the station never tried anything larger. Now this is where TCP implementations and flavors can vary. On another station, maybe
it would've tried again at a larger value. I'm gonna be honest, what I would've expected after a few retransmissions, some stations would even send a reset and say, you know what? I'm sending these retransmissions across. I try three times, if
you're not there, I'm out. I'm gonna go ahead and
drop the connection. So that's where it just
depends on implementation, the type of server that it was,
the stack that it was using. And those little, I call 'em nerd knobs that are inside that TCP stack. - I mean, it's a nightmare, Chris. So, I mean, okay, so you saw the client, you got the capture from the client. You saw that there was a problem, you kind of thought it was MTU, that was like sort of a hint. And then you had to get a
capture from the server, but what happens if you
couldn't get the capture from the server? - Well, see, now experience would tell me, I gotta go look for an MTU
problem and the reason, is because data that is
coming across jumped 1460, that's experience, right? So that tells me, and also if I take a look
at that MTU coming in, I would go, maybe the server knows something I don't know, 1432, right? So here was everybody
ready for the punchline? Who know what it was? - Yes, I was gonna ask you what he- 'cause I was hoping, okay, so what was the solution? 'Cause that's what everyone wants to know. - I know. Well, I'm gonna go to packet tracer and I'm gonna show you
just to illustrate this. Okay, if I bring this back in. So basically, so the WAN in the middle, I had no knowledge of
and I still to this day, let's just draw cloud, a puffy cloud. All right, a big black, well, I don't know what was in there, but this is what we found. So this was a client talking
to some server over here. Okay, so keep in mind, I let go of a SYN and
it had 1460 in the MSS. That arrived over here at 1460. Then in the return path, he let go of 1460 and it
arrived at 1432, all right? What we did is we found
that there was a router here that was doing MSS clamping. What that means is
there's a command in there and David, again, I'm
preaching the choir here. Basically when you see a SYN, take out the 1460 and
replace it with 1432, the reason, I don't wanna do this on every single end station, I'd rather just have the router do it because I got stuff I gotta
make room for in the middle, GRE tunnel, any number of
things that are happening. Basically, Hey payload,
get a little smaller 'cause we need some headroom here for an WAN technology or
a tunnel in the middle. Here's the problem. This router was upgraded, this was upgraded at the local office, but that command wasn't restored, the MTU or the MSS clamping. So what that meant is this happened in the outbound direction, see the server or the router when it received a SYN,
it would change the MSS. So it receives it at 1460
and it would let it go at 1432 and it would just go across, and arrive at the endpoint. But the 1460 in the opposite direction would be maintained by this router here. So what we had to do is go
in and we had to fix that. We had to say, "Okay, let's
just clamp that down to 1432." Every time you see a SYN,
let's clamp it, 1432. That'll give us some room up above. Now, 1432 is kind of an
interesting number to me. And that's where I, and
basically that's the punchline. That's what happened. As soon as this was
restored, things got working and let me tell you why. Things started working
because now, look at that. Let me get rid of that. Things started working because now the ingress MSS, if I took a look at that SYN coming in, now the server knew don't send
anything larger than 1432. Don't send that. The problem was that the
clients were telling the server, Hey, I'm good with 1460, you can go ahead and fill all
of these packets to the brim, problem is that we were
hitting our head on that bridge and the server was trying
to compensate for it. But once this SYN came across
and gave us 1432 to work with, then the server could go, okay
and it reduced the payload. And then now, we were able
to make it under whatever in the middle needed
that extra head space, which I never found out
exactly what it was, but I knew that it was something that took some extra headroom like a tunnel. - Was it a Cisco device or
was it another type of device, those routers? - It was a Cisco. - Can you remember the
command that we're typing? - Yeah, you know what? I'm glad you asked. MSS adjustment. So this is from the Cisco site, the actual configuration. See everybody, audience real quick. Can you imagine preparing for an interview with David Bombal? - (chuckles) Can you
zoom there, Chris, sorry. - Yeah, so what we did
is we found that one, this is MTU clamping, right? So the router's getting involved here. It sees a SYN and it adjusts the MSS down. And again, why? 'Cause that prevents you
from needing to do it at every endpoint. For me, I was like, okay, this is great, but let's learn more
about what's in the middle and let's even maximize it. Like what if 1432 is... Let me tell you why. Let me tell you why that
number caught my attention. Okay, so let me go back
to my little drawing here. - Sorry, Chris, I didn't
wanna put you on the spot, it's just I'm tr- there's another command that I'm thinking about called IP MSS. That...
- Yeah, if you... This was one I- - I'm gonna link a document below because I've actually got a Cisco document where they've got that exact command. I think, tcp adjust-mss, but there's another one that I'm pretty sure that exists, that. - There is. There's more than that one. Let me see, is it in this document? - See, you're better prepared than I am. - See, I knew you're
gonna ask that question. Yeah, this is a big world to get into because once you start
like really looking at MSS and how to tweak 'em and tune them. All right, well...
- Yeah, tunnel that's the command I'm looking for. Tunnel part MTU discovery because what you can do, you can advertise the... And just for everyone watching, I mean, I'll confirm it
in the description perhaps 'cause this is like old memory
for, I think Chris and I. - (chuckles) Yeah, but it's important to know how it works. - 'Cause what you're supposed to do if you like create a
GRE tunnel or something is you've gotta advertise the MSS. Sorry, you've gotta advertise
the MTU to the clients. And then the clients, Chris, I think it's true today that
most clients will support MSS. So you advertise the MSS to the clients and then they automatically adjust the MSS based on that setting. So you don't just hard code it, you advertise that this is
the maximum segment size for this link and the clients will, that'll allow them to work
out the maximum segment size along the path. But I like what you've done because you've shown
like a real world problem where something went wrong. - Yeah, it did. And even to the case you
were just talking about, it's back to what are the
myths that people believe and what are some things that get adjusted maybe without understanding it fully. For example, in this case, they were like, well, we were
told to disable ICMP replies, had you not done that then path
MTU would've told the server what the actual MSS was, or we would've been able
to get in and find out what the true one was. 'Cause basically in the middle, we were dealing with a tunnel
that was taking up space where that previous command
would've been helpful, that I showed you on how
to ramp up your packet size until you hit your head. But basically, this is
what I did with 1432. I was like, I think this is what happened as far as how we ended up on that number 'cause if you take 1432, this is what my brain did with that. If I have 1432, okay? As a MSS clamping on the router, what that means is that if that's my max, then I've got my TCP 20 bytes here. All right, that's TCP. And that brings me to 1452. And if I come up here with my IP and that's assuming
we're not running IPv6, if we're running IPv6, then
just take an extra 20 bytes off of all of these numbers. So IP 20 bytes. So now we're sitting at
a packet size of 1472. And I was thinking, how
do we end up on 1472 as a full packet size? And they're like, well,
that's the maximum ping size that we could get through. And I said, "Oh, that's where we're at" because if we say 1472, that's not again, my max packet size. We gotta think about this also is going to add the ICMP header and the IP header on top of this payload. So I didn't actually get to work with 'em much further on this. I mean, here's the problem sometimes that I run into David, is
sometimes when things get working, then it's like, okay, go away, you know? And I go, "Wait no,
let's do more testing." And they say, "Yeah,
but it works, go away." So anyway, this number often tells me, it's like, okay, we gotta
ping of 1472 through. We're clamping our TCP MSS
based on that successful ping, but the ping isn't taking into account the ICMP and IP header addition, the 28 bytes that could also go through. So that's where I was like, I mean, I don't even know if
we need to clamp this network because if you're getting
1472 through successfully as a ping, that means
you're really getting 1500s, your total packet length,
once you add again, ICMP. Let me draw this better, sorry. If you're getting 1472 byte pings through, well, eight bytes of
that is your ICMP header or you can add ICMP eight bytes to it. And then also there's a 20 byte IP header, you are at 1500. If the biggest ping that
you can get through is 1472. So this number was interesting to me because I think they were
assuming 1472 was their MTU, so now you have to drop it down 40 bytes to get to the TCP MSS of 1432, right? So I don't know, maybe there was something
in the middle that was, we did have to take to account for, but this changed the again, root cause. Let me back up here, root cause was over here on the receiver, I was getting 1432 from the server. The server was advertising 1460, but in the opposite direction that clamping didn't happen. The server thought 1460 was still legit that was getting dropped by a router. So yes, we did have an MTU
clamp in the middle somewhere, but the receiver never knew. It never knew that it
had to reduce its MSS because the outbound router
usually took care of that. So anyway, that's my war story.
- So this Cisco command again? Sorry. - The Cisco command...
- Sorry, the problem having me on the call is I'm gonna wanna see Cisco again. That's Cisco, you had that PDF, I think. - Let me this one. - That's another one. - There's another one. It's the same command.
- Is that the one you had? - It's the same command. - Okay, so the problem
was someone typed that on a Cisco router on one side and then they disabled
ICMP, is that right? Replies or something, what you say?
- Correct. So this was enabled on the outbound router from the server and then
routers in the middle were not responding to ICMP or they were not advert- the ICMP messaging was disabled. And then that would've told
us which router along the way had the lower MTU and then we could've made a
decision from there, right? - So, and then the solution
was to do this on both sides. - We needed to do this
on both sides, yeah. - I think what I'll have to do, Chris is I'll put some links below. There's a bunch of Cisco
documents like we've grabbed some, but you know what's nice is like often on the Cisco documents,
you read the theory about it, but you don't see a
practical example like this of like a war story nightmare. And so it's great to, that's what I love about like you and I, I'm not sure if I'm
allowed to say this, Chris, but you know I've often discussed, like Cisco people like me may know like the Cisco
commands and stuff, but when it comes to this, sometimes they get lost
or they misunderstand and you come from this from
the Wireshark point of view. So it's great to see this, man. So thanks so much for sharing. - Sure, and you know, just as easily, you'd start
working through packet tracer and looking at commands,
and looking at buffers, and looking at things that I'm lost. So packets aren't the
Imperial truth of everything and that's just the only truth there is. I mean, there's a lot of other important data sets out there, but for problems like these,
especially the weird ones, that's usually where I think, okay, Wireshark, let's
actually see what's happening on the packet level to
then be able to make a decision going forward. This was an example where they were gonna get ready to forklift out a ton of equipment on their network and throw a ton of money at it. And I'm like, "Guys, wait, is that before we do that,
let's find root cause, let's make sure that we're directing our funds in the right direction and at least isolate the
box that it's coming from." So, and then this, we were
able to do with these examples. - My takeaway from every time we talk and every time I talk to someone is it's you never stop learning, do you? It doesn't matter how long
you've been doing this because things change. And I remember doing,
like how many years ago. I'm doing like 15 years ago or something. I was dealing with this on
VPNs and stuff 20 years ago. I dunno, a long time ago. You know, it's still the same
problems in a lot of networks, but stacks change and commands
change and stuff like that. But we still often deal
with the same old nonsense. I like to say, you keep forgetting things, but it comes back again. So Chris, again, the
video's getting a bit long. I wanna thank you so much for sharing, I don't wanna kill it. Is there anything else you wanted to say before you wrap it up? - Yes, actually, if you don't mind, I know we we're limited on time that just reminds me, there
was one last little thing, a feature I wanted to
make sure everybody knew with Wireshark. So this is just a quick little PCAP, maybe I can share this one, but if you come down to, there's a few TCP threads here, but a feature called, if you right click and do follow TCP stream, that'll pull the stream date out and then you get a thumbs up.
- I like it. - So that means you can like, so, Hey look, look, look everybody, the package's telling you to like, okay, it's not me.
- Like the video, I like it. - It's right there, it's right there. And then I think there's
another feature in this one too. I didn't show you this yet. If you just go to, I think this file. So if you ever have HTTP, you can go to export objects, HTTP, and there's a really cool
thing, if you ever have JPEGs, be careful, this is where malware... We're gonna have a malware
conversation again. But if you have a JPEG, you can also preview
it, which is kinda fun. And then, Oh, there's a subscribe button. So there was a like, and
a subscribe in the PCAP. I mean, come on. - (chuckles) That's brilliant. (Chris chuckles) Well then, so you gotta like,
and subscribe to the video. Is that kind of what you like hinting at? - I mean, the PCAPs are saying that. I mean, that's it? No, actually this is
a fun one to generate. I'll share it with
everybody if they'd like. - That's brilliant, Chris. Thanks so much, man. - Great to chat with you,
David, have a good one. - I'll just say this to
everyone who's watching, you know, Chris and I are like, it feels like we're opening
up a can of worms here. It's just like more and more stuff, but let us know, do you
want us to go even deep into this stuff? I'll put a bunch of
links to Cisco's website, which I found useful and stuff
like Chris has found as well. But if you want us to go even
more into the crazy depths of the stuff, let us know
in the comments below. Thanks Chris. - Bye, everybody. (upbeat music)
