How to Configure Traffic Monitoring with ntopng on pfsense

tommy here from orange systems and we're going to dive into end top ng with pf sense now ntopng is a tool that allows you to dive deeper into the packet and flow rate of data going across your pf sense it's a great tool it offers layer 7 visibility but i want to please note this exception to that that it is not doing full ssl inspection therefore some of the deep visibility you may get in this will not necessarily be 100 accurate or not just labeled as unknown traffic so take that with a grain of salt on there but it will allow you to at least see where the data is flowing from and to from different ip addresses internal and external it also will create a time series slice so you understand data over time this does not keep all the historical connections this is not a way to trace backwards to old connections something like that should be done by taking all the syslog data and piping it over to a tool such as graylog which i'll leave a link to a video down below on graylog if you're looking for something to give you more historical data not just like packet flow data and charts which this does too but before we dive into this video if you'd like to learn more about me and my company if you like to hire for a project such as network consulting there's a high-risk button right at the top if you want to support this channel in other ways there's affiliate links down below to get your deals and discounts on products and services we talk about on this channel now the first step is to always make sure you're on the latest version of pfsense if you are on an older version and you try to load newer packages you will very likely run into some type of conflicts with the packages newer than the version of psense so whether you're using community edition or you're using the pf sense plus edition that comes with the netgate hardware make sure you're on the latest version go to system go to package manager we're going to show on the installed packages that i already have it installed here if it's not installed easy enough go and install it under the available packages under end top ng then bring it over here to diagnostics and we want to go to the settings now a couple boxes to make sure our checked here is check this to enable end top kind of obvious but keep this settings this is an important box if you do some configuration and you would like all those settings to be saved and exported into the backup xml that's what that box there means then we're going to set an admin password confirm the admin password and then select all the interfaces you want it to monitor also of note it is bound to port 3000 so make sure you don't have the web admin interface for pf sense bound to port 3000 and if you have additional networks that are also connected to this system you want to make sure you put a block rule in so those networks don't access this interface unless you want them to for example for me like maybe lan2 i wouldn't want devices to access any of the pf sense interface and i'd probably put an additional block in so it also can't access this interface on port 3000 just a couple little housekeeping notes on there scrolling down the mode generally this is fine to consider all rfc 1918 networks local but you can customize that if you have some special use case for it i do not geolight2db license key this is the geoip information database you don't have to fill this in if you would like to you have to register register for a free max mind account but of note that's only if you care about any of the geoip data and you want to do the cool overlay map thing i recommend doing it it's free because it looks cool but it's ultimately up to you delete data down here if you would like to purge or you have goofed up the settings beyond what you can remember how to undo uh delete data will purge and reset this and you can then start over and start this process again once you have all this you click save down here at the bottom and then you can access and top ng now when you first access the interface it's admin and whatever password you would come up with to put in there and the first thing is going to bug you is right here in the corner contribute to the project by sending encrypted anonymous telemetry data i will do that dismiss i'm going to contribute i do not need them to contact me and i'll hit save now once that's done the next thing i'm really going to recommend and we'll just dismiss some of these other things that come up is toggling dark theme i wish it defaulted a dark theme it does not but nonetheless that's how you change it over to dark theme we're going to change this to expert view and just run through the settings real quick here are all the different settings for example if you have any specific interfaces that you wanted to ignore post that destination if you have a couple more customized use case you can go through these i do not but i do want to change the time series data down here host time series enable full host time series creation full limit to bytes score light or turn it off we're going to go ahead and go full right here and we're going to go ahead and leave this one fine so toggle creation of layer 7 application create time series per application requires more disk and io it's generally not needed but if you want to keep adding as long as your system is fast enough to do so you can actually turn a few more of these things on to get a little bit deeper into the traffic i'm going to leave that one at none but we will turn on vlans because i have some i want to make sure i separate out some of that data that it may have related to those autonomous systems toggle the creation and application for the asn system we can go ahead and do that countries i'm not raided by any internals and we're going to hit save alerts i don't use the alert system in here maybe you'll find it more useful i don't care as much about the alert data um because i'm not monitoring this actively for alerts but if you want to get more advanced which is out of scope of this video there's even ways to have these alerts that it does find if it finds what it thinks is an anomaly through the series of tools it has built in here you can be alerted of some of that data and it's got ways to get those alerts external way out of scope of this video and it's not even something i really use so i'm not likely to do a video on it but you can go to their site for more documentation on it now the next setting we can change if we want is active discovery of the network you can actually kick this off manually or you can tell it right here to say every 15 minutes go ahead and run an active network discovery up to you if you want this running on your network it will automatically find things as they flow but it's not actively looking at things that just happen to exist on the network so if they have traffic going through pfsense it discovers them and finds them this is a way to find everything right now and start looking at your local network well on the interval that you set here that's really the last setting i'll i'll go ahead and turn it on to show you that you can but honestly doing it right here is easier like right now we'll dismiss this right here we're going to switch to lan make sure we're on the right interface we'll go to dashboard and then just go network discovery and you just hit refresh and it will actually go ahead and start discovering the local devices on this network of which there's only a couple so it found the wind10 lab in the debian lab which there's not going to be a lot of data because we just turned this on so it sees the different connections that this computer is making things it may have going on and even windows compared to idle certain amounts of microsoft connections are kind of to be expected like windows update and things like that what i want to do now is pivot to showing you where i have a little bit more data by logging into the one at my home which i've been running for longer so it's had time to collect more and there's things like netflix and other fun stuff that's been being used at my house so we have more data to actually see all right so we're logged into my house so i can take a look at the different traffic lows and what's going on there so we have some data to play with here this page isn't that great this is just your standard traffic dashboard page the application page is a little bit more interesting but as i mentioned earlier so many things are encrypted now that a lot of it just falls under the unknown traffic because this is not doing full ssl inspection it's not unwrapping the traffic to be able to give the details so some of it just throws up the unknown but unknown and is just a classification that it's missing it doesn't mean we don't know where it's going which can offer us other insight that's where we go down here to the flows so if we click on flows this will give us a little bit more insight to where the data is going and you can see there's several pages here so i can take and dive into any one of these devices and say where is that traffic going go page three page four page five it's gonna go on and on if i actually connected here to office i think it's like nine or ten pages long because there's just more connections and more things happening in here now for any one of these and i'll use this one as an example because this is google's dns we can click on this and it brings us to a page for this particular ip address and any data we have on this external ip address we know it belongs to google it's remote it's a dns server we can click on virustotal it'll open up a new window and let us know if this ip was ever flagged as malicious so we have quite a bit of data that we can gather on this now please note these are ephemeral and disappear so the flows that are going to it now are because they're active but after a set period of time which you can customize a bit those flows will expire and this ip address will fall off of this it only keeps longer term data for internal ips unless otherwise stated where you want to modify it not to do that which generally speaking you'll run out of space if you try to track every ip that every computer ever connected to not really the purpose of the way the end top ng program works here but at least we can look through and see the different traffic tcp udp how much what peers it's attaching to chromecast ultra google nest mini couple devices on my network that are talking directly to it we can look at the flows themselves and this is nice because now you're starting from an extra ip and looking at what internal things are talking to it so we have one two devices talking to it now let's go over actually to the hosts themselves so we look at the hosts and let's pivot a little bit differently let's take a look and find something in my host list here that is pulling a lot of data and you can pivot from that way now these and we'll just take like.90 right here i can look at this one and say all right here's the traffic here's all the flows connecting this one you know it looks a little bit different because we have this right here where we have a home and because this is a local ip it's going to retain the data including all the way over here where we can see a time slice of when this was using data i'll actually set it to one week i've only had this firewall a couple days at my house and i don't know how long this particular device has been on so we have a couple days of data so on different dates you can see that you have the different traffic let's pivot again to one that i know has a lot of traffic this particular address is the chromecast in my living room where there's a lot of tv being watched probably a few movies so if we look at this one we'll see there's 17 gigs of data since we've started tracking this so like i said it's probably only been on for roughly maybe less than a week that i've had this turned on if we look at the flow we can see what it's doing right now so probably not much no one's really watching tv so we only see a little bit there was more on here obviously when something's being watched on amazon on hulu or whatever this chromecast is connected to how you'll be able to see that data flow if we look at this this is where you can really dive in and see when we may have watched some movies so bite sent on each day each one of these time slices as you go through min max so yeah 12 28 when we started last is 1-4 that's actually today and the total traffic is 17-2 and you can dive into going all right this is when we were clearly watching maybe a movie here or maybe some streaming going on here and maybe another one here and you're going to get the idea where you're getting some of the data on here let's look at it from an application standpoint it does understand it's easier to understand dpi data for things that are application based when it comes to like watching amazon or youtube so you can see that amazon video was 17 hulu was 7.9 and then youtube right here at 17.9 it's actually not as much on netflix as i thought netflix is only kilobits probably just browsing through a few things but hulu is definitely where we watch quite a bit more and then amazon video three gigs right here then you can click on amazon video and now you're looking at it from what did amazon video do in the last week so we filtered it for a little bit more and that narrows it down to just amazon video this is what's kind of cool is you can get these pivot points back and forth to kind of understand some of the data flow a few other things you can get in here list of mac addresses host pools networks looking at the different slices of networks that are on here you can get the top hosts this is kind of a neat slow graph but it will show you where the traffic's going and who's pulling what traffic as it goes through here then you can also go to the more fun as long as you've updated that guip database what i p addresses are these going to based on the geoip database which is funny because it's not recognizing a couple of them that it should that should be right here in the detroit area where i'm located so occasionally as i said the database can be a little bit inaccurate so it doesn't even seem to see that but for some reason i'm certainly connecting to a few of them over here interesting and uh what are we doing overseas here scroll out and oh boy we have a few different ips and you can click these ips again and they're going to bring you to pivot to there that particular one has been purged and timed out you'll see that from time to time because if they're only temporary connections where a ping or something was sent you're going to get a timeout on it now of note for those of you wondering and this is one of the reasons you may do this if you were doing some type of torrenting traffic especially when you need to seed all of those different isos for different linux distributions i do recommend putting that over a vpn and i've got a video on that topic this will allow you to do a traffic dashboard on your openvpn connections as well it will then show all the hosts that you're connecting through through there and you can be able to dive into traffic matter of fact if we look back at the traffic dashboard here and we look at it from an application standpoint much of it's unknown but nine percent of it's some bit torn also if we went through goip on this specific one right here we look that majority of them appear to be over here in europe which is also because it does in my video as i just demoed pia swiss pia swiss is the private internet access tied to a swiss connection so you're going to get some more european connections out of there and apparently a few from uh australia and places like that so if you're interested in end top ng it's great to install on pfsense certainly gives you a lot more insight into where your data is coming from where your data is going and can give you a just a lot of fun or maybe a rabbit hole of oh my gosh i can't believe i paint something in russia type of information now as i said depending on how you're configured things you can start at least understanding what on your network is going where and those little links to virustotal are actually very helpful so you can look up some ip reputations but of course don't just stop there go over to graynoise.io and start looking up there and looking it up and showdown and i'm hoping this whole thing sends you just on a rabbit hole of fun of learning networking and where everything's connected to and gives you some insight into the traffic going across your pf sets links to the other videos i mentioned down below and thanks and thank you for making it all the way to the end of this video if you've enjoyed the content please give us a thumbs up if you would like to see more content from this channel hit the subscribe button and the bell icon if you'd like to hire a sure project head over to lawrences.com and click the hires button right at the top to help this channel out in other ways there's a join button here for youtube and a patreon page where your support is greatly appreciated for deals discounts and offers check out our affiliate links in the description of all of our videos including a link to our shirt store where we have a wide variety of shirts that we sell and designs come out well randomly so check back frequently and finally our forums forums.laurensystems.com is where you can have a more in-depth discussion about this video and other tech topics covered on this channel thanks again for watching and look forward to hearing from you you

Channel: Lawrence Systems

Views: 99,396

Keywords: LawrenceSystems, bandwidth, network monitor, linux, computer network (industry), gnu/linux (operating system), netflow, pfsense, firewall, ntop, network traffic measurement, pfsense ntopng, pfsense ntopng configuration, pfsense ntopng install, pfsense ntopng stops working, ntopng pfsense tutorial, pfsense router, pfsense firewall, pfsense tutorial, pfsense build

Id: P8oxTUoF2Nw

Length: 16min 17sec (977 seconds)

Published: Tue Jan 04 2022