Networking security and Wireshark. If you're
someone who suffers from chronic internet access issues at home and are looking for a simple and
effective way of monitoring your internet access quality 24x7 in order to: one, understand where
your problems are; and two, keep your ISP honest and potentially switch ISPs, Net Probe was written
for you. So, I had this exact same challenge in my own home network and professionally. There are
thousands of tools that do exactly what Net Probe does, which is basically to monitor the internet
quality or the network quality of your ISP. However, I couldn't find a suitable tool that was
one, free and open source; two, simple; and three, allowed you to collect as much data as you want
over time so you can compare over long time frames. And that's exactly what Net Probe does.
So, this video, we are going to go into how to install and set up Net Probe on a PC or an old
server in your home network, how to navigate GitHub for those of you who aren't really familiar
with Git, and really just how to read the data, how to go through each one of these metrics, and
how to use it for your own purposes at home. So let's get started. All right, so before we get
started, as always, a little bit of theory on how Net Probe works. So, to get started,
you want to set up Net Probe on a PC, Mac, or Linux machine that's directly connected to your
ISP router via an Ethernet cable. Okay, you don't want to use wireless for this. You know, an old
kind of machine you've turned into a Linux server is good for this, an old PC you're not really
using, or even your primary PC, granted that it's always online, because you want this to be running
24/7, so that it's constantly testing your ISP performance. And that's also in the GitHub readme
if you take a look through it. And what Net Probe is doing is fairly simple. From that PC, where
you install the software, on a regular basis, Net Probe is sending out probes, as you'd imagine,
through your ISP network to various destinations, either DNS servers or public websites. So, in the
case of public websites, we're using the probe, quote unquote, is ICMP, it's ping, and ping allows
you to send some traffic through the router, through the ISP network, and then over the public
internet to a series of public websites. Now, this isn't just one server obviously, we use multiple
websites that we're testing, so if you look at it that way, you're probably looking at something
like this, where you have a destination here, destination here, destination here to different
parts of the internet, which do not reside on the same actual networks. And the reason we do this
is because we want some diversity, such so that the only common element in those connections is
really the connection through your ISP network to the internet. Okay, so we do ping tests to a
variety of different well-known public websites, which I'll show you in the config file. We
record back the latency and the loss detected, and then we average those values together. And
what that allows you to do is get a sense of the performance of this leg here, because if, for
example, we only tested one public website, well, and you saw issues, those issues could be here,
or the issues could actually be here, right on the actual network that that public website is on. So,
by increasing the number of targets, we increase the diversity and the statistical probability
that the issues we detect are actually going to be within the common leg, which is my on-ramp to the
internet, and not necessarily the internet itself. Okay, so that's the first piece. With the DNS test
we do in Net Probe, we do the same thing. Right from the same software, we send out DNS queries
to various DNS servers online. Now, the DNS test is not here to test your ISP unless you use your
ISP's DNS server as one of the destinations, which I'll show you how to do. But it's there to just
measure relative DNS performance, so that you can compare, say, the DNS offered by your ISP to, say
like, the ISP itself has its own DNS that it gives you, but you also want to see how that compares
against like Google DNS and like you know, Quad9 DNS, things like that. That's what the DNS test is
there for. But that's effectively it, right? It's a tool that is sending out on a regular basis,
pings to various public websites, ICMP ping, measuring latency and loss, the latency allows it
to measure jitter because jitter is just a rate of change of latency, and we also do DNS tests. But
that's effectively it, and that's really why you want to have a physical connection here because
really, you want to make sure that if there's any problems, it's not, you minimize the possibility
of the problem is here. Okay, if you use wireless here, chances are you're going to see all sorts
of packet loss and latency and jitter, but that's not going to be because of this leg over here,
likely it's going to be because your wireless has interference, and there's microwaves in your
house and other things. So that's effectively it, we're sending these probes out on a regular
basis to both websites, DNS servers, collecting the data, and then Net Probe, the software,
is actually aggregating all that data into a database and allowing you to build nice visuals
so you can track that performance over time. So, let's switch gears now to the more practical
side and show you how to install Net Probe. Now, the repo, or repository, that has the code for
Net Probe is posted in the show description. I also posted it in some previous videos as well.
The code and the application are open source; there's an open source license I throw in
there. So, feel free to use it for personal use, for use even if you're a small business, I don't
mind you using it. If you need to fork it, you want to modify the code, go right ahead, it's all
yours. Now, if you're already familiar with GitHub and Git, I'm not going to cover how to clone the
repo; you can look up lots of different tutorials on YouTube on how to do that, but I'm assuming you
know how to do that already. Really, you just get the link, clone it, which means just copy it, or
make a local G copy to get on your machine, and then you get all the source code downloaded. If
you're not familiar, then you can just simply go and download the code to a zip file, and then once
it's in a zip file, you can extract it to a folder locally on your machine, and then you have the
files that look like this. So, that is step number one, is just getting the source code downloaded
to your machine. It does run on Python. However, because this is a containerized application in
Docker, which I'll get to next, it doesn't matter that you don't have Python installed. Now, it does
help if you want to be editing some of the config files that you have something like VS Studio
U, or even Notepad++, just to edit the code. If you don't, that's fine, it's not a big deal. You
can just deal with Noad. Okay, the second piece, after you've downloaded the code, is Docker. Now,
this may give some of you some anxiety, especially if you haven't used Docker before, or you hear
a lot of developers talking, and you know, developer jargon, speak about Docker. Docker is a
tool that allows developers, or just enthusiasts to code, to deploy containerized applications.
Containerized applications are applications which, cloud applications are, or well, cloud
applications are containerized applications, put it that way. They're really applications
that can be deployed to a variety of different environments and operating systems and things like
that, without really worrying what the underlying operating system is, and that's what the benefit
is here for Net Probe. The benefit is that, with Docker installed on the host machine, so the
PC that you're going to be setting this up on, it doesn't matter if it is Linux, Windows,
or Mac, as long as you have Docker running, you can run the software, you don't need to have
any local packages or modules on your machine, it's all self-contained. Okay, now installing
Docker is actually super simple. Again, I will post a video in the description,
showing people how to actually install Docker, cuz there's probably a hundred tutorials out
there on the internet about how to do that, but so, I'm not going to cover that right now, but
what I want to cover is that once you've installed Docker, if you're doing it on Mac or Windows,
you would get a Docker desktop GUI that looks like this. And really, what you need to understand
is whether or not it's running, and in this case, Docker is running because it is on. I don't
have any applications running in Docker yet, and we're going to show you how to do that with
the code, but the first step is just to make sure that Docker is installed and running successfully.
Believe it or not, we are literally, you know, 30 seconds away from actually being able to launch
and use the application. That's how simple it is. But before we get to it, I just want to explain
some of the configuration, just show you a little bit behind the scenes. Okay, the code is here;
I've just opened it up in VSS code. Again, you can open it in Notepad; I've done a lot of
development in Notepad because really, there's not really much you have to modify. Net Probe
was designed to just sort of be set and forget, and let it run, and then collect data, and then
you can review the data in a week or whatever you want. But the one file I want to point out is
the ENV file. Right, so if you're a developer, you know that the ENV file is usually used to
store variables, right, usually private variables, but in this case, I made it public because
ultimately it's just configuration. And there's some details in here that I wanted to go over real
quick. The unique IDs up here are not being used, at least in the front end. The next thing that's
relevant is the sites, right. So, I mentioned in the initial theory section, with the ping test, we
actually test a number of really prominent sites, so Google, Facebook, Twitter, YouTube, and Amazon.
And the reason we do that is because those sites are highly available, meaning they're likely not
to go down. They're on networks where that are very well-managed, so, and they are diverse,
meaning that some of them may use Amazon, but they're, and some of them use Google Cloud, some
of them may have their own cloud, but at least when you're accessing those sites from a single
location, you have a lot of probability that the network they live on will be actually a diverse
network. Okay, so that's actually the sites that are used in the latency and loss and jitter test.
For the DNS test, we have a number of different DNS servers defined here. The purpose of the DNS
test is to actually test your own home DNS against a variety of different public DNS servers. So
these, these first three, okay, so Google, Quad9, Cloudflare, they're kind of here as the controls,
meaning they're here; they will be testing this, or Net Probe will be testing these, and showing
you the performance of these DNS servers versus whatever you put in here as your DNS server. Okay,
so the idea here is that you put a DNS server in here that you use at home. Right, so if you need
to, you can go into your command prompt and do an IP config, and then go look for your Ethernet
adapter, oh, let's pull this down, let's look at our Ethernet adapter, and my DNS, oh, I'll
do an IP config /all, and there, there's DNS, and you can see my DNS servers defined in my local
network are 10.10.50.0 and 10.10.1.1. You may have your own DNS servers provided here by your ISP, or
you may have configured a public DNS of your own that you like to use. The idea is that you plug
that in here and save it, so that you can actually map and see over time how well your own DNS server
does versus sort of, you know, the big three DNS servers provided by the big companies. Okay, with
the idea that, hey, maybe if you see that your server is lagging a little bit, you can actually
go and switch to one of these better-performing DNS servers that are better for your network.
Okay, we also have some information about Redis. Yes, the password is "password" for Redis. This
is all internal, meaning it's inside of Docker. Right, think of Docker like a virtual machine,
like you have multiple little virtual machines spun up within Docker, which are the containers,
and they all communicate with each other. So, Redis is one of the layers that transfers data
between components. And yes, the password is "password," but it doesn't matter because it's not
exposed to anything except for Docker. And then we have some information here about health
score weights. So, if you remember the GUI for Net Probe, which I'll pull up, you see here,
there's an internet quality score at the top. Now, I mentioned this before, this metric is really
just a made-up metric, okay. We take all the data we pull from packet loss, latency, jitter, and
DNS response, we calculate those values together with various weights, we come up with a score.
This score has no meaning outside your network. This is strictly so that you can keep track of the
relative performance of your network, one day to another. Okay, so if you do want to go and tweak
those values, you can go ahead. I do not support, um, or if you want to do it, do it, try it out,
see how it goes. I can't offer you support if it breaks the application. These are the defaults
that I've set. This is just to show you what those values mean. And then the last thing is just
some more configuration for components that aren't really used. So, if you are modifying this file,
a couple of areas I would recommend modifying: number one, add your own DNS server here, under
this line here, line 14; and, uh, and number two, if you do want to modify the sites, you can, just
again, the disclaimer that if you use too many sites, if you break the line, it may break the
application. And then number three, you're free to change these three kind of sample DNS servers,
but just make sure it's three. The application is not really built to accommodate more, um, or feel
free to go and modify the code to accommodate more. Okay, so let's get to actually launching the
app now. So, let's launch the app. I'm in command prompt, in the folder with Net Probe Light,
and I'm going to do one command to actually start up Net Probe and get it running, which is
"Docker compose up." So, what this is doing, and I mentioned Docker, think of Docker like, um, a
virtual machine, uh, or virtual machine emulator, like VirtualBox, right, where, within Docker,
to run Net Probe, there are multiple different containerized applications, or basically,
mini VMs, that run applications that are all working together to produce the application
Net Probe. And that's what all this output is, is all those different containers starting
up, right, in the virtual environment. Now, uh, that is really it, right, you launch it,
if you want to go back to the Docker window, you can actually see Net Probe Light is running
in Docker. It tells you the actual, um, resource usage, which is actually very, very small, right.
Net Probe only uses about 120 megabytes of memory, which is nothing, and you, you can see here that
you have five out of five containers running. And that's what I mentioned before, is that there's
multiple containers that go into making Net Probe. All you really need to do now is from
your own local machine, uh, let me pull up a window for that, open up your local machine,
go to localhost:31, and you get Grafana. So, Grafana is the container that is the visualization
piece of this. This is the one that shows you nice fancy graphs. When you first log into Grafana,
it's "admin admin" for the username and password. Once you've set that, it'll ask you for a new
password. Right now, just for the sake of the demonstration, I'm just going to do "admin admin"
again. You should set it to something within your home network that is more secure, and then
you're into Grafana. Okay, um, from here, home dashboards, Net Probe, and there you
are, you can see it's already starting to measure your internet quality score, your
packet loss, your latency, your jitter, and your DNS response time. It's really that
simple. Now, uh, within Docker, if you actually want to stop the application from running, uh, you
can just come here and hit the stop button. Um, if you want to get it off your computer
altogether, you just hit the trash button, delete it, uh, and then delete your source code,
go to that folder that the source code is in, and delete that. Um, that will delete all the historic
data, as well as all the kind of temporary files that Docker may have created. Uh, but other than
that, that's the footprint. Now, what I want to get into last is really just usage of Net Probe.
How do I actually use the tool to monitor and troubleshoot the network? And for that, you can
see I've got my localhost that I just launched, but I, I actually want to open up a version of Net
Probe that's running on a different server in the same network, so you can see the IP address here
is 10.10.50.0. Um, and this, the same exact code, it's just I've had this running for a lot longer
and collecting data, to allow me to actually show you how Net Probe works. So, you run Net Probe,
you let it run on the machine, you leave that machine on 24/7, um, it's going to collect data.
If you restart the application or the computer, Net Probe by default is configured to restart
when Docker starts, so just make sure that Docker starts at your startup, which it should,
and then the data will be preserved. So, even if you restart, you can see here I had a little
restart of the machine. If I zoom in here, um, it, uh, the data is preserved, and it'll just continue
monitoring data after that. So, let's talk about how to actually use Net Probe to monitor your
internet traffic. The way that this works, if you've never used Grafana before, Grafana offers
dashboards which have these various metrics that you can go through. You can actually change the
time window up here, so let's say I want to do 24 hours versus 2 days versus 7 days. It'll really
let you go up to the maximum amount of time that you've got collected on disk. And really, I've run
Net Probe on other machines for upwards of a year, um, uh, in the beta version without any issues.
So, um, feel free to continue to let it run, and it'll keep collecting data. There's no issue. Um,
let's start from the bottom up, uh, to understand how to use the data. Now, number one, DNS response
time. As I mentioned in the previous section, what this is doing is doing DNS queries to
a bunch of different websites. Actually, the five different websites that I have mentioned,
uh, in the, uh, configuration file. So, this is actually going to be, uh, these five sites. Okay,
so these sites are used for the ping test but also for the DNS test. It resolves the name of each; it
averages the speed, uh, at which those resolvers get you a response. So, like, a time it takes
to actually get a DNS response from Cloudflare, from Google, from my home DNS, whatever you
set this to, and also Quad9, and then displays the data here, along with your average response
time for your own DNS server. So, this response time here is from whatever you configure in the
environment file as your DNS server. Okay, um, with the idea that this allows you to keep track
of, over time, how these servers are performing relative to one another, because DNS could be a
major component in network slowness. If you have a DNS server that is consistently slow, every single
one of the transactions you do on the internet, or any of the devices in your network, are going
to be slow by a certain percentage, right? Could be a lot, could be a little. So, this helps
you keep track of that. And what you could do is actually click on individual servers to display
their performance individually. Right, and you can do, you can get a relative sense of which one is
performing the best over time. So, right now, it looks like, if anything, Google DNS and Cloudflare
are, are the winners, or at least from a visual inspection, but I can actually go back and
forth in time, compare different time periods, to see which DNS servers are actually best
here. But that's how the DNS module works. Um, let's go in mixed order to packet loss. So, packet
loss, what packet loss is doing is sending those ICMP pings out, uh, it's actually sending quite
a number of pings, not just like five pings at a time. Net Probe is sending, uh, 50 pings every
30 seconds to each of these websites, um, and it is recording back the, one, the packet loss. So,
how many of those packets were lost in transit, going to and from those websites. Packet loss on
its own is not a huge issue, if it is not chronic, and if it's happening once in a while. Okay, um,
because there's always going to be some percentage of traffic that's lost on the public internet.
You run into problems when packet loss is chronic, like you're seeing packet loss all day, every
day, um, or if packet loss is, uh, you know, it's quiet, quiet, quiet, and all of a sudden you have
like a big period of packet loss for a day or so, and then it goes away. So, that's what this is
measuring, and this is what's showing you. Um, I set a threshold at 4% for a reason, and that's
because with TCP, specifically, so TCP-based protocols, when you hit 4% packet loss, uh, TCP
degrades to a point of almost unusability, right. Um, it has to do with TCP congestion control
and the algorithms that are built into TCP, but really, if you're a network engineer, by rule
of thumb, you should have as close to zero packet loss as you can, but once you hit past this point,
and if it's sustained past 4%, you have a serious problem where, um, you need to get that inspected,
whether it's bad cabling, whether there's something else going on in your modem, um, that's
going to hamper all connectivity in and out of your network. Now, I'll see, see if I can find an
example of that. We had some extreme packet loss in this section. With Grafana, it's quite nice,
because if you want to zoom into a little bit of data on our time window, you can actually just
click and drag, and that exposes this. Now, take a look at this. This is a short period of time
where I had extreme packet loss, up to about 7%, and you can see that had a very negative effect
on my overall internet quality score, obviously, because this is a calculation based on packet
loss, as well as the other metrics. Um, but that also had an effect on latency, right, it also had
an effect on my jitter, a little bit spiked up, and it looks like DNS response times also went
higher than they should. Okay, so this is how you kind of drill in to different time periods, but
how it makes it easy to, uh, take a look at data, zoom into some pieces of data, and take a look
at instances where you have things like packet loss. Now, in this case, I'm not as worried,
because this was a short-term event, but this kind of thing, um, if it happens on a regular
basis, like, you would certainly feel it. Okay, so let's switch back to a 24-hour view. Uh, next
is latency. So, the same ping test that we're running to measure packet loss, we are measuring
latency, um, to various different websites. Now, you may have the very valid question, well, how
does it make sense that we have a one latency figure, because Facebook could be 50 milliseconds
away, and Google could be 20 milliseconds away, and Yahoo, or the other website we're testing,
could be 100 milliseconds away. What does this metric mean? And that's a very good point. Um,
the idea here is that we measure the latency to these various sites and average them out, um,
and what we're keeping track of is whether or not overall that latency increases substantially,
because something that can happen within an ISP network is that because of things like buffering,
like over subscribing links, like they don't have enough capacity in the ISP network, you could have
latency increases that affect traffic to all these websites that we're, we're probing. And this is
what this is trying to keep track of, it's for you to understand what the baseline latency is to all
these different sites, and if you do see increases over and above your baseline, then you may have an
issue that's not due to one of those sites, it may be due to your actual ISP. So, if I look at the
7-day view, you can see that I've actually had, let's see here, there's some times where the
latency is actually quite stable, and then we have these spikes, um, and then we have this
day where I had that really poor performance, and then I actually had overall latency go up and
down over and above the average, and then come back down to a very settled place. Um, now, I'm
very lucky because my ISP is actually relatively stable, but you may see some weird statistics and
ups and downs in your graph that you may want to investigate and dig into a little bit. But that's
effectively what latency is measuring. Okay, jitter is the rate of change of latency, so
this one is particularly important if you're into real-time voice and video, or if you're into
gaming because, um, if you're, if someone says you lag, right, you're playing a game, you lag, uh, it
could be two things, right. One, it could be that you're latency to and from the server in question,
so like, say it's a video game server, or like a, uh, a server for online gaming, you could have
high latency to that server, meaning every one of your actions takes a long time to get to
and from that server. Uh, that's one thing. But it could also be that your jitter is quite high,
meaning your latency, so this value here, is going up and down a lot, it's not staying at a constant
24.2, it's actually going up and down very rapidly because what that's going to do is change how
often your, the interval between your responses to that server. So, say, say you're, you're playing
like an FPS, um, and you're, every time you click, every time you move, every time you make an
action, that is a packet to and from the server, or a message to and from the server, and if those
actions are happening at different intervals, like at one point it's 20 milliseconds, another
point it's 30 milliseconds, and 20 milliseconds, that's high jitter. And what that results in is
a very unstable performance of real-time. So, if you're into, again, if you have real-time video,
real-time audio, um, or you are into gaming, you got to keep track of jitter because that's really
important. And if you see huge spikes in jitter, or like periods where you have high jitter, um,
that could be a serious problem. So, all those metrics, as I mentioned, go into the internet
quality score, which you can keep track of over time, um, and there are different scenarios
that we can look at that, uh, for different problems that you may have on the network. But
really, I just want to leave it there because I think the main thing for this is to set it up,
let it run your machine for a period of time, and then observe, keep track of it. What I would
also ask you to do is, once you set it up and run on your ISP for a period of time, come into the
Discord channel and share your results because I'd really be interested in seeing the relative
performance of everyone's individual network because I think it's really interesting that we
expose this information for each other to see, to see what kind of experience we're having.
The relative score is going to be, you know, not really relative to everyone else, but I'd
be interested in seeing how much packet loss people are experiencing, and in an average 7-day
period, how much latency, right, how much jitter, and what the DNS response is like from their home
ISP. That's it for this video. If you enjoyed it, please like and subscribe, and if you have
any questions on the topic we've covered, please join the Discord server, where we
talk all things network. Until next time.
