Lets Tech: NetMon, Don't Be Afraid Of Packet Captures!

[Music] hey folks Chris with Microsoft here aka Big Daddy nines back with another let's tack I diluted that we're going to do some network capture troubleshooting when we were doing the performance series the network piece and I never really got around to it in fact things got really crazy and I haven't got around to much anything so back again here today to kind of cover some of the the network troubleshooting now the reason that the reason that I wanted to do this is because there's there's a lot of folks out there that I run into that really they don't know how to do a packet capture they know how to ping and let's be honest ping is not exactly a super-duper really good utility for doing any it means a good utility let me let me retract that say so it's a perfectly fine UDP broadcast utility but you know there are a lot more powerful tools in your tool bag you just need to know how to use them so you already know how to use Wireshark or you know any of the packet capture utilities out there then you know this is going to be all hat to you so this this isn't really for you but we just show you what what we're going to do here I've got a really simple little setup we'll start up here on this this top right area how so we've got a domain called fabric cam which is behind a little pfSense firewall that I put together that keeps the 10.0 1.0 24-bit subnet separated from the 10000 where I have another little setup we've got a little contoso DC sitting over there so those are 24-bit subnets meaning that these are segmented the PF sense is acting as a firewall and a router right now it's kind of open both ways just for us to get kind of kicked off here then everything's attached to a switch which has an external fire which gets a firewall which gets us out to the Internet so this is the contoso BC that as you can see is in the Tendo dojo dot Network and it's over here on the right side we've got the fabric and calm so fabric ham for break them however that's pronounced I'm not I'm not really even sure it is on I'm going to go ahead and flip off the firewalls here because we've got another firewall we're going to be using for testing and I don't want to be having to configure two firewalls at once you see we've now turned that off you can see this is ten dot one dot ten and and just really quick I can I can hop in the ten dot into the firewall real quick and get logged into this and kind of show you we don't have much configured in the firewall right now it's fairly straightforward at the moment so let's just go to kind of the the rules on the firewall we've got two interfaces we've got this land interface and the land so when is the same network that mr. contoso over here is and when is or landis is the the network that the fabric am DC is on alright so a pretty simple setup they all have access to the internet and you'll also see really quick just to show you how I've got that is let's do a route print as you can see right here mark that you can see I've just plugged in a static route for the 24-bit subnet so that any traffic going 10010 is going to go through the 1000 201 which is the public interface for this little firewall over here that we're looking at and you can see that on the LAN interface I currently said anything on the 10.0.0.0 network is going to be able to get to anything on the 10010 24-bit network so everything's open it's all wide open in other words I can ping the other one from here so 10 dot ODOT 1.10 I think it was yep that's it and theoretically he should be able to go the other way too so let's go and pull this up a command prompt hang ten dot odo dot what is your IP address mister sir it was like 202 or something like that it is 202 2t yep so you can see we've got a connectivity both ways so I'm using pink so this is all good stuff but there are other utilities that are built-in that I discussed in the in the the performance series go look at the network section we would cover things like paths paying and and I think I even had some of the sysinternals utilities which we're going to loop back to that one of these days I promise but you've got built-in utilities that are even better than me that are in that we're not using those today because this obviously doesn't tell you anything about the network itself so what do we do when we have a network communication failure between point a and point B or point B and point a just using ping doesn't necessarily tell you the whole picture in fact even using something like pork query doesn't always tell you the whole picture yeah the port might be open but it may not be responding properly as a matter of fact I've probably solved more Active Directory issues just using net Mon then even getting in the Event Viewer because sometimes what you get in the error message that pops up on the screen or what you're seeing in the event logs don't really give you the whole picture but when I look into net Mon and I look at the packet I might see an in event from Kerberos saying this is not supported because of this weird ops obfuscated code and then I look up that up the skated code on being and there it goes it says oh well you know you've got SMB signing turned on or something weird like that you you see the the error message that didn't really give me tuned in there so you can actually use net Mon for more than just troubleshooting network of utilities and you might be wondering why aren't we using message analyzer well frankly because we're not you we're not using network analyzer a message analyzer because most people have already got net mon installed on most of their systems and if they don't it's you know not a problem I'm not saying you should do this or the other the concepts you can use here would work for Wireshark it's just going to be a little different using the command so I'm going to show you how to set up packet filters and you know going beyond just the standard capture so you can see we're just going to do a quick net Mon search on Google or Bing and then click download and it'll take you to this page obviously for most of your servers you're going to need the x64 version of it and we'll go ahead and pull that down and get it installed I've already taken the liberty of installing it here on fabric ham and it's already up but I wanted to kind of bring it down because I wanted to show you that there are in fact two things happening during the install neck that in the past has confused some folks they'll get the first part of it installed and not the second so what we're doing here is we're going to go ahead and launch the Installer you'll see that it's going to install the network monitor capture and the parser engines you need both and it actually kicks off two separate installs it used to be two separate files and you may still have that on your network this is 3.4 but there was another 3.4 release that had both of the installers separate you need to have the parsers and that's really in my opinion where NAT Mon excels especially when you're an active directory person like myself if you do a lot of AD work and you need to be able to see what's going on the built-in Network parsers are great for being able to interpret what's going on a lot of people though they get scared about using a network capture utility of any time because they know a lot of stuff goes on on the network right oh by the way you saw me click run as administrator that's another thing that doesn't automatically happen so always click run as administrator otherwise your captures will be empty it's another reason people don't use it much they launched it they tried to capture and they got no data so there's a problem but they know that there's a lot of things happening on the network and if you're capturing all of that how are you going to read it and you know even if it weren't for the fact that you're making a photocopy of a lot of the stuff going across your network and you know that that's going to become a you know a big ugly thing in a hurry a lot of people they don't know what to look for well I want to show you just a few simple things you know just like when you're analyzing a blue screen of death right 80 to 95 percent of the the problems that you run into can be solved with some easy tricks you know any of the deep core analysis beyond that is obviously a little bit different but you know for most of the stuff just be able to see what's going on that's what we're going to be covering here today alright it finished the first part you'll notice when you hit finish it's automatically just going to start installing some more stuff that's the parsers right so it's a two-piece deal so look at well that installs we're gonna kind of look at the interface here you'll see that it's got several different little adapters listed you may have more of these if you have a multiple if you have multiple adapters on your system you're going to see one of those for each you also see that might see the isatap that's ipv6 stuff since those the Teredo we're going to worry about those at the moment because everything in this is just going to be ipv4 stuff so we're going to click new capture that's kind of step one in getting all this going I'm going to ahead and slide this to the left right now because we're going to be setting up a capture with kind of some specific things note that there are two very important things on this screen right here what you see up there is a display filter but you'll also see capture settings and you'll notice that there's a capture filter alright so let me explain the difference between those two when you filter on the capture you can filter only down to things that you want captured whereas a display filter is inside of what you've captured what do you actually want to see going on right so the the built-in filters have little hints so you've got a bunch of standard filters if you can't remember what showing you here today then you can come back in there but make sure that you're not trying to apply a display filter when what you're really wanting to do is apply a capture filter right to keep your capture from getting enormous and capturing everything that's going across the network what you want to do is actually just if you're having problems from point A to point B then on point B you need to filter to point a and on point a you need to filter to point B that way you can see both sides of the equation and none of the other crap flying around on the network so that's lesson one that's really the biggest lesson that you're going to that you're going to need to to be familiar with so do that in your capture and then if you've got extra stuff you want to filter on later like you only want to see DNS traffic or only Kerberos traffic then put that in your display filter those things can be put in the capture filter as well if you know up front what you're doing let me just give you an example of a capture filter first off in here I can go ahead and turn off the adapters that I know that I don't care about but here's a really simple filter so we already knew from looking at this guy that his IP address is this 1000 202 so I can do an ipv4 dot address see it's already populating and I can enter to complete or tab to complete and then net Mon likes two equal signs when you're going to do that we're not going to get into why until later but it's an actual equation you can do not equal to and things like that and that's why there's two equals but just remember ipv4 dot address equals equals this is a most common thing you're going to use right there take a look at it right and then not surprisingly you put in the address 1002 to 202 there we go just like that and it's just as easy if you're looking at multiple things say we got three DC's on on this contoso side over here and you've got this one fabric am DC over here that's not functioning properly probably just put my hands on the wrong side of the screen for what's actually above me here but you can just say or ipv4 dot address this is a perfectly valid caPSURE filter Oh to all 203 there you go that's it now here's the thing everybody forgets that a button right there apply you must apply your filter before you close it or it will not be filtering anything right and if you want to change it on the fly you can do that by changing it and hitting apply again while the capture is actually running it'll change it right in line when I click close and I click start all that is going to happen now is that network adapter is going to be filtered and look for only packets that come from this other person now earlier I had mentioned point A to point B and point B to point a it always matters it is always urgent and important that you want you know in except in situations where there's just no way to do it at all times one must always capture from both sides now you can tell that the capture just started you saw it just popped up two lines that's what you're waiting for sometimes on slower systems net1 takes a few seconds to initialize once you see your filter in place that means you're good to go it is now looking online and you'll notice that it has actually you know sitting there monitoring stuff on online and you know I could on this system go too low very slowly very slowly go places wow that's spectacularly slow come on little guy hit me with another tab I probably set all these up with a single processor core and that now have denial of service myself so what is going to show you was if I go someplace like Google or Bing dot-com obviously I am now generating network traffic and you would expect to see a lot of that coming in a network capture but if I minimize this you'll see there's nothing there's nothing because we filtered to it right but on the other hand if I ping this guy 10.1 dot ten I think it was you'll see ICMP traffic just hit so now we have a whole bunch of pending packets kind of fluctuating up it's dropping those out because we put the capture filter on it right now this is where a display filter might be important you can actually do those right in line I could say add the source to the display filter and apply it now all I'm going to see is stuff coming from the source fabric Han right so now I'm only looking at those four packets and then it's just as easy to remove those so they're still in the capture that we had okay back to my point before the tab freaked out you should always capture both sides the reason I say that is because I don't know how many times I have been looking at Network captured okay in a network capture and seeing the weirdest of behavior only to finally remember oh yes net want an instructor from 15 years ago told me always capture both sides and I quote capture the other side and lo and behold the other side is not seeing any traffic I'm seeing strange stuff happening on this one server the other side is not seeing it what does that tell me I'm not actually talking to that other side and that actually happens you know you get firewalls little intercept port 80 traffic and you don't even know what's happening but it looks like you're talking on port 80 to something out there but what you're not doing is talking to where you thought you were and then you don't see any traffic coming from the other side all right so again whenever you going to launch net want montt you need to run it as administrator and you don't need to keep checking for updates since net want Mon has been kind of deprecated do new capture and so on this side obviously for anything we're going to try we're going to do the ipv4 address equals that other guy ten dot dot one dot ten there we go and I don't need any of the ipv6 on there so now I've got that one on and I hit close okay so pretty straightforward anything we do between these two servers right now is going to go into the capture another little trick right here let's fill that up a little bit this little button right here this one that says auto scroll that's a good thing to have right so now you can see it as it's going so I know it should be obvious but for I don't know a year I always used to say gosh I wish that NAT Mon had an auto scroll it was up there the whole time I just never noticed a big blue button so anyway hey look use the auto scroll it's cool that'll that'll keep the latest packet going and when you're monitoring for things going on things hopefully more interesting than ICMP traffic the you know you can see actually the motion of it going all right so let's kill that for a second now what we want to do is we want to kind of go through some ideas of why we would expect to see something useful this just looks like a whole bunch of just nonsense to most people and you know well frankly it kind of does look like that but you know there's some obviously recognizable bits that are going on in here so before we start digging into what you might expect to see during a failure which we're going to set up a failure and we're going to look at what that might appear to be by changing some of the firewall rules on that on that PF sense that we have running here so we're gonna get in here mess with that a little bit but let's talk about the different components of what we're seeing here first of all this this tab over here it says all traffic my traffic unknown if you're looking on everything that is going on that box and you just want to filter it out and just look at certain pieces and parts of traffic this can be useful but for just troubleshooting a point to point I usually just turn that tab give myself some real estate here in the capture window obviously the buttons up here for starting and stopping the capture and the capture settings we need to keep those going this is the start page we were out these are a bunch of parsers that are built into net Mon that's your actual capture and if you're running multiple these are the tabs use their display filters below that and then below that is the actual frames alright so inside of these columns these are just incremental frame numbers you know they go up every time it catches a new frame this is the time the time offset since the last one process name that will appear sometimes and sometimes it won't like if you're in the DNS console and you're doing some tests it might actually say MMC dot exe or DNS MGMT depending on how you launched it it might be DSA dot with Active Directory services so whatever it's running from a process sometimes that can be useful but we move over then we've got source and destination now source and destination it will attempt to actually look up the DNS address of what you're getting so it'll get you a pretty friendly name that is not always terribly helpful if you are looking at multiple machines but if you're just looking back and forth between these two if you want you can actually change the source and the destination up here in columns the other thing that I find useful a lot of times is it tries to tell you protocol name right here and it tries to tell you kind of what port you're talking on over in this description area which is actually an actual breakdown of what's in the header of that packet which we won't go into a lot of details here because that's getting outside of that 80 to 95 percent bit we'll cover a little bit maybe but not everything in the college you can actually add stuff so one of the most important ones that I find myself is like destination port right here I will a lot of times put the destination port right next to the destination because gosh that comes in really handy when you have multiple ports going on so you can also get in here and to add in the source port I'll usually take the source port and move it right next to source like so there and so now we can see we have not updated the screen for whatever reason oh it's spinning blue done it trying to add that in there we go see now I've got more stuff in my capture window yay more things to look at all right so there are no ports right now because everything we've been doing is UDP because ping does not actually produce a port it's just a broadcast right but if I was hitting this guy on port 80 or something then we would see some ephemeral port range stuff on one side and 80 on the other over here on the far right is the conversation ID this can be helpful because typically what you're going to see and we'll cover this in a second is a sin a sin AK and an AK look at what those look like but that's high and the other guy's saying I hear you saying high and he says oh I hear you saying high that you said high and so forth it's a handshake three-way handshake and during that they kind of decide on a port and some other things like that anyway it can get convoluted lots of different pieces flying around during different conversations it's nice to know what conversation ID you're talking to the ID of the conversation if it's mixed up with a whole bunch of other stuff happening at the same time this can help to kind of clear that up you can even sort by it but not well the captures actually running okay and then again down below this is you know what was in that traffic in this case very simple echo reply message boom okay and you can see abcdefg and e f g h i right so it actually inside the ICMP bits it threw in some letters and stuff right so you see that's what went across the network inside the packet when you know when it goes out and it says hey the bytes equals 32 well that's how it got 32 bytes it started throwing alphabet characters at so there you are okay so now we've got captures running between these two servers right and mostly happy traffic at the moment so we might do some stuff to generate something other than ICMP traffic and then let's break some of that stuff all right so we're already capturing we're already good we already know that the networks up we have two different forests so I guess let's create a trust between them so let's go to domains and trusts we'll do that on this contoso side and so go to properties and this should generate a whole bunch of ad traffic here so we're going to go ahead and do I don't know if he actually can I bet he can't resolve we may need to do a quick forward look up zone on both sides actually so let's do NL tests des get DC contoso wait am I on c'mon contoso I'm going to get these mixed up every time yeah he can find contoso just fine fabric am calm hey Dad no such domain and you would notice no I did an NL tes which should do a quick little bind ish thing to the other D see if it knew where it was no traffic happened here he never attempted to look that up alright so so I should have already had this set up but let's just fix this problem real fast okay we'll do this real fast by building a conditional forwarder we'll call it fabric and calm and we'll pop in the address of the other DC Godot 1.10 was it ten yeah so okay we saw the traffic look we saw a port for 453 now you'll see the source port here you'll notice that it said DNS because it's actually the DNS console on the other system talking to DNS on this side so those are process is talking so now we actually have something interesting to look at right on the source port five eight nine seven nine so that's an ephemeral port right so those ephemeral literally means something that's there for a little while and then gone the ephemeral port range is 49,000 up through 65,000 535 they're just quick used ports windows we use but we know then at DNS uses port 53 and so you'll notice that the destination port the port that it was going to go talk to on the other side was 53 here on this side and then we replied back coming from 53 going to that ephemeral port right now it said it was unable to resolve that that port but I betcha it actually can and it's just being a Butthead about it we're also going to store this an ID in case later we decided to add some more so we'll say everybody in the forest gets a copy of that forwarder and then when I go into fabric ham and I look at that I should be able to do this now gasket TC still no love for you and s lookup fabric am calm okay so it found it you saw here we had some more DNS traffic you see that be the source again we were looking for a DC so we did a DNS query so now we've gone from boring old UDP icmp traffic to something a little more interesting on DNS so we've got some stuff flying back and forth on port 53 looking for something and if we go down to here query ID response success so you can see in the far right over here that's what it looked for so the way this conversation started and again we're looking at the conversation ID is now 16 so we see that this is new conversation we have come from that other dress over here to us a ephemeral port to port 53 what did I say I said query for fabrikam fabric am calm and that's what it looked like over here in the machine language ish type of stuff so we had a reply and we had a success right if we dug down in a little bit further you can see here was the fabric ham ipv6 address and other information you can see we had a nice pretty little conversation without even having to have a three-way handshake so that was just capturing a quick DNS query that's as basic as it gets on on capturing this stuff now why it won't actually do a desk at DC for it I don't know it's never making it over here to the other side so I'm not really sure why he's doing that it could be that he'd cache that he couldn't find it before but without that working we probably won't be able to create the trust we can give it a shot but since it also might just mean that we're timing out something funky that happened we know pretty much for sure that we're going to need to go back the other direction so we can go really quick and set that up going this way so it's a DNS on him - and where did it go to that let's go to DNS this is definitely a server that I didn't set up with enough power so we're going to fix that next time we have to reboot this guy give him perhaps a little more power which I can't do on the fly sadly this is my twenty sixteen bucks oh and by the way didn't mention before that the server on your left there's obviously a twenty twelve box so the other one is the new 2016 yep one on virtual processor yeah we need to give him more powers because that's going to stink it's going to make everything really super slow all right so we got fabric ham dc1 we're going to go in conditional forward we're going to build us a new one and we're going to point it back to contoso comm and you are going to be ten dot zero dot one twenty something or other - - oh I have no idea why I can't keep that straight my head is beyond me but that's why we have G sheets 1000 - OH - ten Oh - OH - OH - all right we go should actually be saving some decent network monitor traffic going back there it's already happened it's just not showing up in the deal alright so we're going to store in the forest we're going to click OK and ok there we go so we've got DNS going both directions now and we saw the conversation hit so there's another little boring conversation inside a net mom in this case but yeah auto scroll see hey he knows what both names are why would that have changed all of a sudden - fabric Raymond contoso well see we have name resolution we know that's working now alrighty still no such domain I wonder if we can get the other direction because I am really pretty sure PS get DC come - so calm works just fine going this way albeit slowly oh my god it looks like a teletype let's get this all this server needs another core badly alright so there so we got some interesting stuff doing we even had some LDAP message stuff so it's a UDP version of LDAP happening there so obviously you know port 389 on your LDAP PSA's right well this is 389 but it's a as you can see LDAP message which was a UDP broadcast of that so sometimes you'll see UDP versus T DCP in the protocol name but since the since the parsers on it actually knows what kind that was until it puts that in there it's a little bit of geek trivia there for ya all right so we've got to go in this way I've probably done something stupid on the side that's keeping it from being able to do that maybe our I kam this a B maybe I have it displayed our i kam SABR i kam looks right to me why you know want to go and resolve the names my little guys say be Rik him I didn't mean to ping you silly guy mmm all right let's try it again but this time we're going to rip down Mon and bind to fabric Hamden come all right yes we can bind but we have a DNS problem with it preventing it from actually doing that so here we just byte binding we just created some really interesting traffic will fix the DNS thing in a little bit but we're going to turn off auto scroll and let's look at what we just had happen here since we went out of some of this boring world and got into a little bit more of a typical kind of a conversation that you see right over here where it says s that means sin and a s means syn/ack that's the high and the other side saying oh I hear you saying hi and this is it coming back again and say I hear you here saying me that we said hi something like that let's look at it on this side because it's more well I was going to say it's great because it's got the DNS resolution so no it doesn't stick to this side since we're looking at it and then we send back and forth some LDAP messages so this is a typical kind of healthy conversation between two servers right we could go and look and see what they were saying but most of that is not going to be terribly usable to us and I have got yes it's a tell that message is not going to be anything really usable but what you're looking for is are they talking do we see anything in here that would indicate that one of the servers does not like talking to the other server or is doing what we will call a retransmit on one of the ports we're going to get to that in a little bit and it's the biggest thing you're looking for is out here is do we have this server talking to that server but that server not ever answering back well we do we do see them talking back and forth when this server talks to that server and doesn't get anything back that's when you'll start seeing repetitive conversations and out here in the description you'll start seeing something called a retransmit that means I I was able to talk to you at one time but you're gone where did where did you go hello hello hello and then fail usually at least three retransmit sometimes more some applications never give up and notice tree transmit for ever but there's one right there for one reason or another not sure what it might have been this server from conversation 187 when he sent his ACK he didn't get anything back back back didn't get my act back okay god I should not be trying to be funny so he tried it again right so he retransmitted the ACK and that happened right here not right there 187 sorry isn't that a code for like homicide oh so he resent that so when you get a bunch of retransmits in a row especially when they are on a specific port in this case this was 389 if we see a whole bunch of those you know you might have a problem and here's another one right here it's just another little port push on on that so that could be that I'm overwhelming my firewall or this poor server which was hung up in la-la-land there for a bit much more likely with it's one little processor core it's literally less powerful than my phone at this point got hung up it had 100% processor utilization and didn't respond back and so the other guy retransmitted so sense an ack ack now these are very interesting to go track when you have a when you have a latency problem you'll get retransmits these aren't indicative of a big problem but these these do happen but when you get a bunch of them that's when you know you're onto something all right so I'll show you an example of that in a moment but here is another time we would use the display filter right so I don't want to lose this whole capture and I've already captured it so it doesn't matter anyway but I want to see anytime we saw some O's retransmits and where they were coming from and what port were they happening so so if I can remember this is a property if we spell it right it works even better property dot TCP re transmitted like that there's another one that you'll use sometimes called a TCP syn retransmit it depends on if it happened at the beginning of the conversation and especially if it's something like a portal ad type deal so if I apply that display filter now what it's going to do is take its one little bitty processor core and go through that that list and go find out all the times during that capture that we had this you can see it happened at three different times during the 187 conversation during the 190 conversation and during the 192 in all three cases it was dot 0 dot 202 the other guy trying to talk to us and in all three times it was him retransmitting those packets so if I go over here and I do a property about TCP retransmit not reap are you serious right now ret really you don't want to do anymore do you re T RA NS M it's because it's a TCP retransmit there we go all right so when I apply that to this side of the conversation what we'll see is kind of the opposite right the source is now the contoso side and in fact we we're talking to him on port where is it I don't have the port deal so if you if you haven't turned on your destination your source port there right here if I'm hoping that the mouse is showing up right but you can see the source port you can see that it was 389 twice and then endpoint resolution 135 twice and then a port 80 actually so he saw what this guy saw so it's pretty much lined up just means that he tried to talk to him but he wasn't answering probably again because processor processors are bound up right at that time and so it it just continued to see a non responsive behavior within the time that it was expecting to see it come back right so anyway you can see it with or without the port and the reason I have it in here and I have it in as a column there is for me when you get into these deep thick captures sometimes it really helps Plus you can actually do copy source port to the display filter and then you would only be looking at let's say destination port being port 389 so that's a good reason to have that as well the other one is the TCP syn it's on and on auto complete summary transmitted so this would be a good time to have both like that so now we have a filter with two different things that we're looking for and apply it now of course we didn't have any sin were transmitted so that's not actually going to show up with anything usable right now but yeah so there's two hugely powerful tools you can use for those types of issues all right so let's let's clear off our filters the display filters let's leave the captures running let's run another test here and it was just being a pain it was just taking a little while it didn't want to work right then all right so we've got a name resolution on both sides of these so they both know how to find each other and they both know how to do things like look each other up so we should be able to get through a trust wizard at this point so we are on contoso and there we go lots and lots of sense in acts the other piece you see is an R that's a reset send sent ack ack they talk and then they reset when the port is done what that is it is an indication to the other server that you can release that ephemeral port now I'm done with it and it goes back into the time wait and it stays there for a little bit and then it becomes an available port again usually what you'll see before the reset which I didn't see any of them here is offend so send syn ack ack payload data in this case you can see we had SMB to stuff we had just general SMB I mean TCP / 445 which ad does use that it doesn't just use 445 for file transfers it does actually use it for authentication stuff sometimes so anyway when he was doing a whole lot of little queries against him some of the things were like okay what is your bias name would a very you know other bits and pieces of information that you do that LSA RPC for instance it needs to be able to query that so that I can actually see hey are you something I can trust and that it answers back as long as you don't have a Stig on that server it will allow it to answer at least a basic minimum piece of questions anonymously asking you are you something I can trust and it comes back and says I am something you can trust in my name is this and it's like good okay well that's what I was hoping your name was alright so we're going to switch this over to force trust so we see more Kerberos than we do ntlm type stuff yeah no we'll just do it two way trust why not that's fine and since I know the other side we'll go ahead and create this on both sides so the other side is contoso and it's just got administrator right at the moment and password in and then we'll wait for it to do things we actually should see quite a bit of stuff happening over here as these guys negotiate oh I forgot my autoscroll there were lots of things happening we just missed them all because reasons all right so you'll see that now we have some LDAP sa-l we've got the let me turn off the auto scroll for a second we've got MSRP see we've got some more SMD stuff we've got some buffer stuff but a whole lot of communicating on port 389 port that was mostly port 389 and 445 I was expecting to see possibly some 88 in there since I did authenticate I bet if I went up high enough I probably do get to see some of that I'm not spotting it so what do we do maybe what we do is we go TCP port equals 88 and apply that and see if we had any Kerberos traffic anywhere I didn't say source port and I didn't say destination port I just said any Kerberos port evidently we didn't need that even though I did give it a user name and password just to show you that that display filter will work we'll go ahead and show it on 4:45 so you can see now all of the 4:45 communication there so this is kind of lesson number three TCP port you can do it on source port you can do it on destination port you can even combine these so we could even say or property TCP retransmit and apply that and it'll say hey either stuff on for 45 or retransmit so you can see and mix and match these pretty good so lot of cool stuff you can do them all right so let's we've got that removed we need the auto scroll on let's go ahead and continue through this forest wide authentication that just means that we're not doing said filtering we won't want to go turn that on for right now because we're not we're not doing a trust video we're just doing a port type stuff okay so there was a reset there's the fin so it's going to say before and then I got distracted CIN CIN ack ack pay loads of data and then when it's done you'll usually see a fin the other one will acknowledge the Fen and then you'll see the reset why are we doing keep Alive's are you not I thought no I was expecting that I had clicked next and we were now spinning donut see this was an indication to me that it was waiting on me for something so I thought that this actually was stuck so we're going to force slide not selectively off we're going to say next next and finish and I wouldn't be too shocked to find that we do finally get some Kerberos up in there since we had a password but I guess that's probably going to be nice or I just died oh okay there it is okay do I want to confirm it yes we do all right let's confirm the incoming trust and we are seeing lots of good stuff happening and it finished all right so we reopen the dialog books and go check it out so first and foremost we need to go and look at the name suffix route and make sure that it knows it does so we're good that's always something you want to check inside your trusts is do we have a name suffix route to know how to get to the other domain and we do so we're good okay there we are all right so we've got a trust we can now do off scent acacia nikai know stuff between these two forests and I did that rather than doing a full on just domain across two different subnets because I want to be able to restore constrict the ports and then occasionally only a handful of different ports compared to the entire 80 range so now we're going to do is we're going to start squishing things so the first thing we're going to squish is we're going to try to get you know what the first thing is we're going to do is I'm going to pause recording this video and I'm going to shut this DC down and I'm going to give it more processing power and then I'm going to bring it back up alright it's back up and running again a little while it didn't want to shut down proper and so now that it's back up a little more more juice on it alright so what we're going to do is we're going to take the firewall rule here which is allow anything from the Tendo network to the 10.0 1.0 Network we're going to constrain that a bit so I'm going to get in here and I'm going to remove this rule and I'm going to put some replacement rules in and actually what we'll do is before we put the replacement rule in we'll go ahead and disable this just to generate kind of some some standard retransmit type activity over here so let's let's just do some really simple stuff Rep bad and and bind that is going to work because I didn't apply the changes on the firewall and this should if I got it in time it should cause it to fail and if it ever actually makes any attempt whatsoever then we'll actually start seeing some some retransmits happen but at the moment I don't know okay server down all right what did we see we have nothing pending we didn't see a single retransmit and I don't know why we should have seen actually several several attempts am i running here and is that currently oh that's why I still had that display filter applied all right or maybe not because keep alive reset let's try this little doohickey again let's try to bind fabrikam comm and see if we get any traffic going I can guarantee that on this side we won't have any traffic past the last couple seconds just because well we shut off the firewall rule that allows it so I would I would think that we would be seeing at least oh you know why because it's trying to relook up the server it's doing a DNS query and the DNS query is failing so it doesn't even attempt to do anything else so let's let's be fun here let's cheat 1.10 alright so now we've got an LDAP and a retransmits or ssangsun and now we see the other kind that I told you about the sin retransmit so we never actually had a good conversation so we're trying to resend the sin and he's saying hello and hello and hello and server down so this is pretty much a right out of the bat sin retransmits because we don't have anything but that's not how these usually go right normally somebody screwed some rule up somewhere on some firewall and some things work and some things don't this is why sometimes pings work and you still can't get across just Y port query can be very handy but if you don't know specifically what's going on on the wire this is a good thing all right we never saw anything on the other side and we wouldn't have expected to so I'm going to load up some rules here that should be kind of the bare minimums to be able to have stuff work across the trust and I'll pause the video so you don't have to sit here and watch me adding rules into a firewall because I'm pretty sure you've all done that before I'll pause the video and be right back all right I've got them loaded up okay so we've got several different ports you can see in here so let me cover a few of these now this isn't the typical what you would see as like requirements for ad in the big article right it lists a bunch of Swiss cheese holes in the firewall not all of those are necessary for maintaining a trust but you know some some of the biggies obviously like Kerberos and you've got LDAP and you've got the endpoint mapper which we'll be covering in a second and the ephemeral range which also becomes important here so anyway everything is still working the way it was before just now we've kind of paired that down to a few different rules but let's say that the endpoint mapper was working for our PC communications which is basically going to come down to a CB radio communications is one way I guess everybody sits on channel 18 I know this is really really out of date stuff but if two people wanted to talk they would say you know go to 43 or something like that right and then they'd chat on 43 and then they'd say clear 218 and they go back basically you talk on one open channel to go and find what channel you're going to go to the way to computers when they're going to do our PC communications is they're going to start on port 135 they're going to answer on the other side with an RPC request in the ephemeral range and as you can see here in the capture it's 49 1 52 through 65 535 their ephemeral that disappeared quickly the entire purpose of those ports is to to do an exchange to do a bit of communication and then when it's finished those ports get closed people very frequently will go close these because there's so many of them and it's scary they're like well why do we have to have all of those ports open so let's let's do what many network administrators sometimes do and shut that down and say I don't think we really need it what are all those ports for so now we're going to we're going to do in a little bind exercise here so binding to fabric on fabric am should now be broken and we'll go ahead and kick up a capture to kind of look at that a little bit and we'll wait for it to fail here and when it does we'll kind of analyze what's going on and I don't know if I got the capture started in enough time to really make that anything usable so let's do this again we're going to we're going to clear off this capture and start it again and then we're going to wait for that capture to start and for some reason even though I gave it more processors that one's still running a little slower than the other but as soon as we get the little indicator that that is running and we'll go ahead with it ready go big guy come on show me show me you know how to network all right we'll pause the video since this thing's going to take 45 minutes to start it's Network cap okay it's more like two minutes but alright so that's going so we're going to hit this again and you'll see that there's a lot of communication going here so this is a condition where we did actually get some ports open but not others and now we need to find out which ones those are so now we're going to look at the actual traffic we can already tell that it's failed we can also see some transmits the the transmits have now stopped and it's gone ahead and sent a port reset so it's going to talk about one of the big things that we're going on here may turn off auto scroll and stop that capture for just a second okay so when it comes to endpoint mapper so you can see that we're once again just to kind of reset we are can't oh so and the contoso DC is going to attempt to talk to the fabric AMD see on our pc you can see that's ms RPC communication so what we'll do is we'll we'll hit it from one of our ephemeral ports on port 135 that's the breaker 1-9 channel that's the hey what what port are we going to talk on it it's going to give you a bind ack and then you're going to send it a endpoint mapper request and then you'll get back a response and all of those things worked everything worked just fine inside of the endpoint member response it'll tell this computer what port to talk to it on next so you can see it reaches out and it tries to talk in this case on 49 7s or 670 and it sends a sin and then as part of a different communication it actually acknowledges the that it got that port but then on this you can see that it is attempting again to retransmit it on that port and again nothing opened up so it eventually gives up and it resets so you can see we had Kerberos stuff we had several different things that were work just fine here and then all of a sudden we kind of hit a brick wall the reason for that is because the the the femoral range is closed so that's that's a good indication to you when you see it in that that range the once again the forty nine thousand one fifty two up through the sixty five thousand 535 it was closed in this case and different ports do different weird things depending on what it is that is the task that you're attempting to to accomplish so it's not always a matter of just you know endpoint mapper stuff and actually it occurs to me that we have why is that not enabling that rule you need to turn that back on there we go and turn that on there may need to change the type of trust really quickly so that we can show some additional weird behavior that you get when some ports are open or some points are closed but you know the main point if you're you know if you've gotten this far this is the type of thing you're looking for you need to know kind of what what ports are communicating and even in some cases what applications so that you can kind of pare down where is it break broken sometimes what you'll see is something's taking an incredibly long time but it still works and if you're doing a net Mon capture what you might find is that it's trying three or four times in a traditional way and it has a fallback method that it'll eventually drop out of in this case our PC is going to be just our PC but there are other times when that's not necessarily the case so anyway let me let me set a couple of other little things up and I'll show you a couple more examples I'll pause the video okay as you can see I have disabled 389 standard LDAP we've still got all the other ports including global catalog secure LDAP and all that stuff got a little group here and I have a member in that group and when I look in here I can see that that user object from the other domain so that's been created but if I try to add actually live me let me just try removing that person from the group and then let's go back and read him I just saw him he was just there but when I try to go in and try to add that person to the other domain even though I was able to query that before we find that we start shooting these LDAP messages across and they're failing so these are sort of like sending retransmits but they're not because what we're what we're looking at when we see that there's the trip retransmits right there the LDAP messages that is the same as a UDP version of LDAP I think I'd mentioned that earlier so you'll see the LDAP message a lot when a client computer is first trying to find domain controllers it'll send out kind of a broadcast and see who who responds back and a lot of times that's how it determines which domain controllers are online a little bit further down here you can see that it just kept on throwing LDAP traffic at the the other server but it wasn't able to get that through so this is one of those weird conditions where we could view users from another domain we could manage that but we can't actually do anything as far as adding new ones because that one particular port is not open so when we tried to hit the the query when the initial piece and see if I caught that or not when I opened it up I'm not sure if I was running on the capture yeah I was ok so when we first started this capture that's where it actually found the Chris object from the other domain because that was our PC communications viewing that name now I understand that that object is actually a foreign security principle but that's fine it doesn't have the identity information from this for foreign security principle the name Chris would only come up as a a bunch of goods if if we didn't have that working it would still see that it was there but the foreign security principle would come up but this right here this an RPC communication which did work actually showed us the Chris object but when we removed it and tried to add it back in it switched over to an a an LDAP transaction and we've got LDAP shutdown so that's why that that second part of this started to fail when the reason I'd switch the trust type actually was because I needed to go to an external trust because otherwise instead of using LDAP it would have been using it would have been using s LDAP instead and it also uses port 445 and some other things so I thought it'd be more interesting to shut it off like that so let's turn LDAP back on and re-add the cris object so now you should be able to go in and check the name and the LDAP transaction when you depart we have a capture running should have started kneeling more than likely not sure if we let the router have enough time to do its thing yes I know that I got that enabled still showing disabled here that needs to be enabled by the changes let the little process on the router reboot and let's try again should be working actually any second because it is retransmitting on 389 and we just turned 389 back on of course it's in the middle of a communication that failed so it doesn't know about the other side NFL so let's go ahead click it again now it works so now we've got this Chris from ferb recomm added back into the group but let's turn off the RPC communications this is so funny disabled there we go apply changes alright so now we have no RPC edge so let's stop and restart that and it's going so let's go back into our little group and now this time when we go over to the members tab you'll see that it's doing what I was mentioning before which is it's attempting even though it is of a foreign security principle and it does actually exist in this directory it does have to go out and resolve that name so you can see right here we are retransmitting trying to get back a hold of that other domain controller and it takes forever I've actually seen this work believe it or not sometimes it'll try several times on on an RPC communication and actually fail back to another protocol not just in looking up users but other functions that Active Directory does Active Directory will sometimes piggyback communications on other ports you'll see it sometimes try and LDAP transaction and then move to SSL that you'll sometimes see things that should have been on probably other ports wind up on port 445 I think I mentioned that earlier actually anyway this is about to fail when it does finally give up and the reason it's taking an incredibly long time to fail and stuff just failing instantly is because the DC knows that it's talking to the other DC it's getting that communication between it it's getting handshakes but then when it tries to go to the end point mapper it's totally confused and so now we close the communication and it you know it goes back in and it tries to move to s and B this is what I was mentioning before but anyways it's rolling through several different protocols trying to do things it's confused because it can talk to the other server it has token with the other server it's done DNS queries to look and see if maybe it needs to talk to another DC that hasn't worked it is still sitting here timing out waiting to please go find the members which you've probably seen this and now it's given up it has finally said I can't show you the friendly name of this object I know what this thing is this is actually an object in the local directory and that grid would match this but it's a foreign security principle but it couldn't match it to the name because that one court that it wanted to talk to wasn't it being answered by the other domain controller so took it forever finally failed and like I had mentioned there are some times when you will see things actually succeed you know this wasn't really a success but there's other times when they'll do that and sometimes that's when you want to get in and do a little bit of an that morn trace and see if you can figure out maybe what's going on under the hood there so yeah anyway there's a good couple of examples for how you can go about setting up your net mount races for both sides of the communications doing some port filtering some display filtering some filtering for sin retransmits and tcp retransmits and LDAP messages which are effectively the same type of thing UDP type of broadcast hey find me a domain controller and what it looks like when you need several ports open and maybe only one of them didn't get open properly by the network team anyway I hope this has been informative I hope you've enjoyed this this is part of a series called let's Tech and it this is the first one of these you've seen I have a number of other similar type of videos where we just kind of go through and talk about tools that you can use technologies that are out there in the Microsoft stack that are part of your tool belt or part of our year a lot of times these are troubleshooting related so if you want to go check out the playlist for that the link is in the description and if you know if you did find this informative I sure would appreciate a like and if you want to see more videos or know when I upload new videos hit that subscription maybe even consider hitting the bell so you get notified and otherwise thanks for watching oh you guys have a good one see you next time [Music] you [Music]

Channel: bigdaddy9z

Views: 4,981

Keywords: netmon, troubleshooting, windows, server, packet capture, network trace, microsoft

Id: w-N6MRs3OE8

Length: 69min 11sec (4151 seconds)

Published: Wed May 17 2017