If you have watched my last two or three episodes
about Proxmox and faster Internet with Multipath and so on then you have probably been waiting
for this video. If not – don’t worry – here’s what we do today. We build a network. But
not just _any_ network. In essence we build a very simplified version of the internet inside
a virtual environment. It’s got a lot of features. The main feature though is here in the middle.
We will use a web interface in order to tweak the different connections here on these shapers so
that they can behave like mobile phone internet connections for example. But without spending
hundreds of dollars for hardware or exhausting your mobile phone internet plan on the first
try. All virtual. With free software. Stay tuned. (intro) Quick reminder – if you’re in a hurry or you want
to skip parts of the video then use the chapters here on the time line or in the description.
Here’s the break down of this episode. Thank you. I can hear you ask – Marc, why on
earth would I want to build a virtual network ? First – thanks for asking. In fact,
there are a handful of use cases for this. Maybe you don’t have a network that you can
play around with, but you want one. Here’s a cheap and direct way to do so. Or maybe
you want to rent a VPS, vserver, virtual server or whatever you want to call it or you already
have one but you want to test stuff in the lab before you put it to life in the real world. Or
maybe you are looking for a way to simulate bad network connections – high latency, low bandwidth
etc. and see how things turn out – you can do all of this inside a virtual world – on the
cheap, without purchasing a lot of hardware. Cool, now we know why we want to do
this. Next let’s see how we do it. I will show you how to build this inside Proxmox.
But you could do the very same inside probably any virtualization solution, be it ESXi, VMWare
Workstation, Fusion on a Mac, Oracle Virtualbox, Xen or anything else. We just have to chose
one. And I happen to have Proxmox here. We will build this environment from left to
right. This way we will have internet access at every step throughout the deployment. We
start with the perimeter router here that links our virtual and physical environment. That
is in fact a virtual machine running OpenWrt. We will then build the three shapers here.
I built them in LXC containers but you can do the same in Virtual machines. I used containers
however because they use much much less resources. In a nutshell you can probably run five or
ten containers with the resources you need for one single VM. Once we have that
we will then add all the other stuff like a client here or a machine with
OpenMPTCPRouter and a virtual VPS Server here. Awesome, now we know exactly what we want to do.
Before we can build the nodes in that network we need to set up the networks. Setting up a network in Proxmox is
done in a couple of clicks really. You select your PVE machine here on the left,
then click on network. Here you can see all the existing networks. You should at least have
one which is your physical network. In my case I have two of them, one is vmbr0 and one is vmbr1.
They are just in different networks. Now in order to create more virtual networks you click on
create then linux bridge up here at the top left, give it a name and a netmask and that’s pretty
much it. We don’t need more. Here you can see the networks that have been created. I created
networks in the 10.7, 10.8 and 10.9.x range and one in the 192.168.56.x range. But you can
create them in any private network range. The IP addresses here do not need to be in the same
network that your virtual environment will use. Basically the address doesn’t matter. If
you do however chose the same IP network then you can access that network from the outside.
OK ? Clicking on Apply Configuration up here makes the changes permanent. What I suggest you do is
that you print out the blueprint here or make a drawing of the environment and just write down
the names of the bridges here and where you want to put them. That will make the creation of the
machines and containers easier in the next steps. Cool, we have created the networks and now we
can start defining the nodes. Let’s start with the perimeter router. We will use OpenWrt for
this. If you have watched my video on how to build a virtual home network inside Virtualbox
then you probably remember these steps and can skip this chapter. For everyone else here is
how to install OpenWrt in a virtual environment. In Proxmox we create a VM by clicking on this blue
create VM button here. Now you need to give it a name and a unique number to identify it. The name
can be changed later, the number can not. Keep in mind that this name needs to be a valid DNS
name, so no spaces and special characters. Next Proxmox wants to know if we have an installation
CD. We don’t. We will use a ready made image which we grab from the web later so we select “do not
use any media” here. We don’t care about graphics. But the next step is a bit overhead. We need to
define a virtual harddisk. But as I said we will get that later from the OpenWrt website so in fact
we will scrap that disk and replace it with the one that we get from the OpenWrt Web site. Let’s
just make it small. On the next two screens we define CPU and memory. We don’t need much here.
1 Core and 512 Megabytes of RAM are plenty. As a network device please use one of the bridges
that we defined in the first step. That will actually be the router’s LAN interface. We will
add the physical interface – which is the router’s WAN interface in a next step. I untick firewall
because I am not using the Proxmox Firewall here. As a hardware I use the paravirtualized network
interface – that’s much faster than emulating real hardware and OpenWrt deals very well with it.
Quick Summary – let’s confirm but not start it yet – we will still need to add some more stuff.
Once the machine is created, we click on it in the left column here, then select the hardware
section and click on add at the upper left here. Let’s now add the outbound network, in my case
that’s vmbr0, again I use VirtIO and untick the firewall. We could also already detach and delete
the hard disk but let’s do that in the next step. Let’s browse to the OpenWrt website. I am
using a release snapshot of Version 21.02 but you may use the recent stable version. Just
keep in mind that there is snapshots without the web interface and then there is the releases
which contain the web interface. The one I want to use is under targets/x86/generic. Make sure
you use the ext4 combined image. So the larger one. That’s the second one here in the list. Copy
the link location to the clipboard or download the image to your Proxmox machine. I connected to
the Proxmox with ssh here and just download the image with wget. Paste the URL. Next unzip the
image for example with gunzip and then you may convert it using the qemu-img convert command
or directly import it to the VM using the qm importdisk command. The commands are in the
description of the video. Don’t worry. Once the qm importdisk is finished it then shows the
name of the new unused disk here at the bottom. Back to the Proxmox interface. Here we can see the
newly imported disk. Now we first need to detach and then remove the old disk. In the next step
we select the new disk, then we click on edit, select the IDE controller and confirm by clicking
on Add. That attaches the disk to the VM. Before we can boot the VM from that disk we need to go
to Options here, then edit the boot order and actually tell the VM to boot from that harddisk.
Perfect. Once we click on start in the upper right corner, we can then check the console of that
VM and it should show GRUB coming up. Inside the console we can quickly check if the network
adapters have been attached in the right order by typing ip addr. And what we should see is that the
outbound WAN interface which is eth1 in OpenWRT has an IP address from your physical LAN and that
br-lan which is the interface pointing towards the inside of our virtual network has been given
the 192.168.1.1 address. If you want to change the router’s address then you can do this now
by typing uci show network.lan – here you can see the settings of the LAN interface. And you
may change these by using the UCI set command. Like I do here uci set network.lan.ipaddr and
then the IP address you want it to have. Uci commit and reboot. Checking with ip addr
after the reboot should show the right IP address. If everything went well, we should
also be able to ping a host in the internet. If that doesn’t work then please double check
on all previous steps before you move on. Perfect. We have successfully created our first VM
inside the network which is in fact our perimeter router to the physical world. This VM connects our
virtual network to our physical LAN and ultimately to the internet. We can now move on to the right
and create the shaper machines. Right after this CALL TO ACTION – I need you to get involved
please! I have a couple of questions that only you can answer. First question of course – please let
me know in the comments if this episode was useful for you – if you have a use case for it and if yes
– which one – second question – should we build this in Docker ? I mean, would you be interested
in building a network in Docker ? And the third and last question: I had a look at the command
line interfaces of Proxmox and we could automate the deployment of this environment, that means
you would launch a script and that would build this whole network for you – would this be
something you would be interested in ? Please do leave me a comment. Many many
thanks! Over to the shaper installation. The shaper machines will be LXC Containers.
So in Proxmox we click on create CT. Again we give the container a unique ID, a DNS name and
also we need to specify a root password here. Next we select a template to use. I select
Debian 10. Next tell Proxmox where to store the disks of the container. I put it on
local-lvm which is in fact on my SSD drive. That will be much faster. Also the size of
the disk is quite small – 8 GB is plenty. One Core and 512 MB is a lot for what we want the
machine to do. Now in the next step we define the network interface that points to the outside. In
my case that’s vmbr6. I set the protocol to DHCP because my Perimeter router will provide IP
addresses on this interface. Confirm and then Proxmox will create the container. Just need to
add a second interface under the network section here which I call eth1 and that’s one of the three
bridges to the right here on my diagram. I chose vmbr7 and give it a static IP address according
to my blueprint. In my case 10.8.0.1. I am adding a third interface here, but that’s just for the
sake of making the video because this way rather than accessing the shaper via the PVE console
which is a bit hard to read on the video screen because of the white on black, I can rather ssh
into the container and use different color and font settings. Just for the sake of not ruining
my viewers’ eyes. You don’t have to do that. Perfect. The first shaper machine is up and
running. As I have added that second Interface, I can now ssh into it and do a couple more things
here. In a nutshell, the next steps are to tell the shaper that it should act as a router.
We do this by first enabling IP forwarding then we will be NATing or masquerading
the outbound interface towards the router and also I want it to act as a DHCP Server for
all the machines that are behind it. Last but not least we will create an ssh key pair so
that we can remote execute commands on the shaper – that will be important once we tweak
the quality of the network lines. Let’s go. Turning a linux machine into a router is
just one flag in the operating system really. We can do this by setting the net.ipv4.ip_forward
flag to 1 with sysctl. The box will now forward traffic from one network to another just like a
router does. We can make that change permanent by adding or un-commenting the setting in the
/etc/sysctl.conf file. Here’s the line. Just need to remove the trailing hash. Alternatively we
can put these commands into the /etc/rc.local file which is executed each time after boot. I have
also put the iptables command in here which actually masquerades or NATs outgoing traffic so
that the packets which we send to the internet can find their way back to us. All files are on
the github plus the commands to use are of course in the description of the video. Just quickly
checking if I have internet connection. Yep, that’s good so we can now install some additional
software. Let’s first run apt update in order to get an updated list of available packages. We want
to install three packages. The first one is bmon which is just a tool to show network speeds. We
will use this to monitor if and how the network is used. Next package is the dhcp server. I use
isc-dhcp-server for this. Last but not least I also install speedometer which is another
tool to monitor network bandwidth. Cool. Now let’s set the dhcp options. I want this
shaper to be a DHCP Server. The options for this are configured in /etc/dhcp/dhcpd.conf. In this
file you can specify the networking options which the shaper should provide to the machines behind
it, such as the name server. In my case that’s my physical main router, the 139.139 address here.
Then you define subnets and options per subnet. The one that is important here is the 10.8.0.0
subnet. It tells the clients to use 10.8.0.1 as a default gateway, it tells them that the
domain they are in is called testnet.lan, that they should use my main router as a DNS
server and I also specify an IP range which they can get automatically. In my case the .100 to .120
range. Last but not least I need to tweak the dhcp server in the sense that it needs to know which
ethernet interfaces to serve and which protocol family. I will only be using IP V4 so I just
need to set the InterfacesV4 option here really and set it to eth1 at least. Now I restart the
DHCP Server by issuing the systemctl restart isc-dhcp-server command and double check if there
are any entries in the syslog by grepping for the word dhcp in the syslog to see if everything
went well. As you can see, the server stopped and restarted properly. All good. We are ready
to provide IP addresses over DHCP just like a real router does. So now – just one last
thing that we need to do before we are done. In order to tune the network interfaces we will
need to execute commands on this shaper machine from a distance. For this we will use ssh. But
I don’t want to type a password for this or to hard code a password somewhere. Therefore, I
create a keypair using the ssh-keygen command. I leave all parameters at their default
value just by hitting enter each time, that means that I don’t add a passphrase neither.
Do not do that for servers which you access in the internet. It’s just something we may do here in
our safe lab environment. The keys have been saved into .ssh/id_rsa and I now just need to tell
the ssh daemon to use them. That’s done in the /etc/ssh/sshd_config file. Just need to
add the path to the keys here. That’s it. Awesome – the first shaper machine is ready
for use. It acts as a router, it distributes IP addresses and it can be accessed using ssh public
key authentication. You may now either do those steps two more times in order to create the other
two shapers or you might just clone the container by right-clicking on it and selecting
clone in the context menu in Proxmox. Once you cloned the machine you would of course
need to change the second network adapter to serve the right bridge and adapt the
dhcp settings to serve on that bridge. If you want you can also re-generate
ssh keys in order to have different keys for each machine or just use the created
ones. Perfect, our network is growing and we can now add the right hand client machine
which connects to the bridges and the VPS. Same Procedure here, we generate a Virtual
Machine – Just this time I need to give it a bit more juice. I chose 32 GB as a disk
size and give the machines two GB of RAM. As a software I install debian 10 again. I won’t
show how to install debian 10 – that’s really just questions and answers. Just make sure that
you don’t install unnecessary software packages. The most important one is to have an ssh server
selected. The VPS needs to get the bridge here as a network interface and if you want to
install a client like we did in the MPTCP video then you need to give it the three
shaper bridges as network interfaces. That’s pretty much it. If you
want to install OpenMPTCPRouter then please refer to the OpenMPTCPRouter video. In
a nutshell, proceed like for the OpenWrt router. Just this time use the corresponding combined
ext4 image from the OpenMPTCPRouter.com web site. Assign the rightmost bridge as a first
interface. That will be the router’s LAN and the three shaper bridges as a second, third
and fourth interface. That’s pretty much it. In the OMR settings you just need to switch the
protocol of the wan inteerfaces to DHCP. On the VPS, you may run the OpenMPTCPRouter VPS script if
you want to use this for OpenMPTCPRouter. Again, please watch the OpenMPTCPRouter video on how to
do this or check the description of the video. Perfect – Depending on what you want to
do with the network you should now have one of the following layouts – either
you just have a client machine here or you have OpenMPTCPRouter or maybe you have both.
In case you only have OpenMPTCPRouter installed, we would need to add an additional client
here to the very right of the schema in order to run the web frontend which will allow
us to tweak the network speeds of the shapers. I will again use a container for this, but
you could also install a Windows Machine or alternatively add a physical network interface
and attach an existing physical machine here. But I will show how to do this in a separate
video. For this video I will use a linux container and install a graphical desktop environment into
it which we will access using the X2go Client. So again, we create a container from the debian
10 template, like we did before. 8 GB of Disk should do. One CPU is fine, just I will give it
a bit more RAM and swap. 2 GB of RAM and 1 GB of swap should be more than enough. Network-wise I
attach the machine to the outmost right bridge on our diagram which in my case is vmbr10. And also,
in order to be able to access it from the outside, I attach a second network interface. In
my case I use my IOT network for this, but you could also use the LAN. Just need
to make sure that you give it a fixed IP address in order to avoid that the machine gets
a default gateway from that interface and goes to the internet through it. We want the container
to go through the shaper machines. Alternatively you could also map a VLAN through to your
Workstation. I did this by adding a USB Ethernet interface to my workstation and mapping
it to a cable on a switch next to my desk. But don’t worry, I will make a separate video on
that as well. Cool, as soon as the container boots I will have to install three software packages on
it. But first I need to create a non-root user. It is really important that if you launch graphical
user environments in linux that generally you do this as a non-root user. In my case I create
a user called marc by typing adduser marc, give him a password and a name. The usermod
-a -G sudo marc command adds me into the sudo group in case I want to run stuff as root using
sudo later. Next let’s install the graphical environment. I want to install x2goserver
as a remote viewer, mate as a desktop and firefox-esr as a web browser. That’s all. I also
installed sudo by typing apt install sudo on top. Cool. Now we have a client here at the end
of the daisy-chain and we want of course to connect to it using the x2go client.
Let’s actually do that from a Windows PC. You can get the installer from x2go.org. Click on
the mswin link here, download and install. Work through the installer and launch x2go. The first
time you launch x2go and create a new connection, it will probably trigger Defender. Just make sure
that you allow access if you want to use Audio for example. In the session preferences window here
I need to type in all details of the connection such as the name or IP of the VM, the ssh port and
the user name which it will use for connection. I set the session type to Mate as this is the
desktop environment which I have installed on the server. There are a couple of other
parameters which you may set depending on your environment such as the screen resolution,
access to the clipboard, the speed of the line, if you want to have Audio etc. Once you finish
this you can now access the connection by clicking on the newly created tile on the right part of
the screen here. Most probably Defender will come up again and request access. Let’s grant it. The
first time you connect, x2go will also ask you if this is the right machine to connect to. Just say
yes. Awesome. I am now connected to the client in the virtual environment and can open a browser
and point it to my OpenMPTCPRouter for example. In order to do this I browse to 192.168.100.1
and I can start administering my OMR router. Our network is now complete. We can start
using it. We have remote access into the environment and traffic is routed through the
three shaper machines. The very last but my no means least thing that we want to do now
is that we want to install the web frontend for the shaper machines that allows
us to tweak the network parameters. Remember, we wanted to turn those excellent
network lines here into bad lines and hence simulate 4G or LTE characteristics such as
low bandwidth or bad ping time. Let’s go. The method that we will use in order to tweak
the network lines is that we use the Quality of Service interfaces that are built into Linux.
In essence we modify the queuing discipline of the network interfaces of the shapers in order
to add latency, reduce or increase bandwidth or add random errors to the transmission.
Don’t worry too much about the details. I will make a spin off episode on the details
like quality of service, queuing disciplines, schedulers and the like. Just so much that the
command which we will use to achieve this is called tc in Linux. The tc command allows
us to a add, modify or remove a scheduler to a given network interface. In our
use case we want to use the netem filter that allows us to do exactly what we want. We
can modify bandwidth, latency and error rate. One thing is a bit inconvenient though. In
order to modify the lines on the three shapers we would have to ssh into each of them and type
cryptic tc commands on the command line. Therefore I have written a small and simple web interface to
this using nodejs. What it does is that it takes the parameters from a graphical page, connects to
the machine using ssh and executes the tc command with the right values on that machine. You can
obtain this from my github repository. The link is in the description of this video. You may
either browse to the Repositories tab here, then to mptcp-tools where you can find the script
in the nodejs subdirectory. In a nutshell, this is a javascript file where you define the servers in
lines 19 to 24 – so you might need to adapt this to your environment. In order to get the code
go to the Code button here and download the zip file or copy that link which we will use with
git in order to install on our client machine. So let’s connect to our MATE desktop client, then
we need a shell which we can find under system tools – mate terminal and here we just quickly
need to install git. Let’s become root with sudo and then as root run apt install git. Now you can
clone the repo by typing git clone and the url which you have copied from my github site. Now cd
into the mptcp-tools directory and from there into the nodejs directory. In order to run the script
you need nodejs. Again as root we install nodejs by doing apt install nodejs. Nodejs is a nice
and quick way of running web frontends with Java script actually. I love it. Let’s adapt the
script quickly. We don’t need to use nano or vi as we have a full desktop here, so we can run
the file manager which is called caja in Mate by browsing to Applications-System-Tools-Caja.
In Caja we go to the same subdirectory and open the qdiscs.js file with pluma – the built
in editor for text files. Just make sure that you adapt the IP addresses in that file to
reflect the IP addresses of the shaper bridges. You can double check those for example in
the OMR interface if you have installed OMR. Once you have saved the file, run node
qdiscs.js in the terminal. That will start the web interface. You can now launch Firefox under
Applications-Internet and browse to localhost:8080 which will show the web interface. That shows a
very simple form with one block for each shaper and one line for each interface on that
shaper where you can set the parameters for rate – that’s the bandwidth – delay – that’s
the latency in microseconds – so 50 milliseconds would correspond to 50000 in that field –
and you can add a percentage of error rate. Below I added a big text box that will actually
show the results of the command. But before we can launch a command remotely on the shaper machines
we need to copy the ssh keys over to this machine which will then allow us to remote execute the tc
command on the shaper machines. You remember ? So let me open another terminal and ssh into one of
the shaper machines. In my case I go to 10.7.0.1. The keys which we had defined are in the .ssh
subdirectory. By default the key is called id_rsa. Now you can either show the content of that key by
typing cat id_rsa and then paste it into a new key file on our client machine using vi or pluma.
Just make sure that you give it the very exact name like you specified in the script. In my case
that’s id_shaper1. In order to make it usable, the key needs to be restricted in access rights.
In order to do this type chmod 600 id_shaper1. Perfect. Back to the interface. Let me type
in some parameters and then click on apply. That didn’t work – why ? Actually my
shaper 1 is on the 10.8.0.1 interface, so I got the wrong key. But I need to get the
other ones anyhow. So same procedure to 10.8.0.1, cat the key, copy it, exit, rename the id_shaper1
to id_shaper2, then create the shaper1 key, paste the key, write and exit, try again. Forgot
to chmod. Try again. And…. It works. I have just added 100 ms latency on both interfaces which
should sum up to 200 ms latency in total. Let me browse to my OMR and see if it took note
of it. Absolutely. The wan2 interface here jumped to 200 milliseconds. All working as expected ! Now
– getting the keys with ssh and copy paste is not everybody’s turf – so let me show you an alternate
way of copying those keys over to the client machine. We may as well use Filezilla to copy the
keys. So in a terminal we install filezilla by typing sudo apt install filezilla. We can now
launch FileZilla under Applications-Internet. In order to connect to the shaper we just type the
IP address in the Host field, root as a username, the root password and 22 as the port. Filezilla
will then connect over ssh or rather sftp. In the right window we can now see the files on
the shaper machine and can easily copy the key file over by either dragging and dropping it to
the local machine on the left or by just double clicking on it. We can also directly rename it
from here. A nice side effect of this method is that we don’t have to chmod as the file keeps
the permissions from the source machine. Right - we are all set and can now start
and make the connections really bad. OMR should pick this up any second. Here we go. Guys, that’s it for today. Join me on Sundays on
discord for video chat or screen sharing if you have questions. Contact me on Discord, Facebook
messenger or reddit if you have questions or remarks, If you liked the video, please please do
leave me a thumbs up – Last but not least, if you want more, please do subscribe and also click that
bell or notification icon – Guys, many many thanks for watching, liking, commenting and subscribing
– stay safe, stay healthy – bye for now.
