#295 Raspberry Pi Server based on Docker, with VPN, Dropbox backup, Influx, Grafana, etc: IOTstack

if you ever dreamed of a raspberry IOT server with backup to Dropbox and secure remote access from everywhere through your own VPN all based on docker containers you have to watch this video great sea youtubers here is the guy who ever Swiss accent with a new episode and fresh ideas around sensors and micro controllers remember if you subscribe you will always sit in the first row having a Raspberry Pi as a home server is an excellent addition to our IOT stuff but setting it up takes a lot of time and know-how what if we could use the most modern tools to reduce this effort and add some bells and whistles let's try it by installing docker with many containers like mosquito not red Gravano in flux TB and others increase the life expectancy of your SD card by disabling swapping and by installing lock to ram automatically backup all valuable data to the cloud or in our case to Dropbox set up a VPN to remotely and securely access our home network from anywhere in the world besides that you will learn a lot of useful things about docker and containers let's start first I want to thank Graham from South Africa he did the heavy lifting on what we will see today and he created a github repository including proper documentation for you I only had to express all my wishes and dreams my first wish was to use docker containers on my Raspberry Pi because this is a modern way of organising applications but what is stalker and how does it work in regular raspbian we install applications in the file system often they are complex to install and have many dependencies that all have to be installed too especially beginners can get nuts if something goes wrong and for sure it takes a ton of time a better concept is to use so-called containers as in logistics they hide what's inside and can easily be transported because the outside is standardized docker is the de facto standard for those containers when we install docker we get a linux add-on which can handle containers the containers themselves contain prefabricated applications like no threat or influx DB specialists configure these containers and regularly adapt them to the newest version we do not have to care too much about that and the best you can go to docker hub comm search for the containers you need and docker automatically downloads them to your Esprit fortunately the containers only contain what is required this by the way is the main difference to virtual machines which also include a full operating system and therefore are much larger copied to our raspberry you can start and stop containers as well as delete them let's continue with our wishlist and IOT server for our lab as usual it consists at least of mosquito node-red in flux TB and gravano what are the steps to get those containers running first we go to docker hub and search for mosquito we find many containers luckily the first one is an official image which is good it was updated one hour ago so it seems it is well maintained and it includes versions for many different platforms including armed perfect with one click we get additional information like where the container stores its configuration data and locks and how to start it cool unfortunately these directories are inside the container and you cannot access these files from the outside also if you delete the container these files are gone and you lost all your valuable data and configurations we will later see how we can map those directories to our standard file system to keep them outside the containers this is great because like that we can separate the data which is specific to our installation and therefore valuable from data that is online available there is no need to backup the publicly available data because we can't restore it quickly we only have to backup our own data which is a fraction of the site of the overall SD card I like this concept a lot because it is a simple and efficient protection against a defective SD card and later we will see how easy it is to more or less automatically install all standard components next we need node rate here we find no official image and also the most popular version is deprecated if we search a little we get the new version this is because node-red recently had a significant update to version 1.0 the new container also contains a version 4 arm cool and it uses the well-known port 1880 now we can go on and find the rest of the containers and start them not to complicate it but too much work for a lazy guy like me fortunately we have a more elegant way to reach our goal we can use docker compose which reduces our efforts considerably and it will even get easier than that docker compose is a framework that contains scripts to Commission and start containers automatically if we look at the docker compose file on my Raspberry Pi we first see all the different containers here called services let's look at note read we find the name we give to the container as well as the name on docker hub then we see that we want the container to restart if something terrible happens also the port mapping is here so we could map for example the internal port 18 to a different part outside the container if needed lastly we see the volumes here we map the internal data directory to a directory outside the container to save our valuable data when the container is deleted the same thing applies to all other containers if we execute this Yama file with the simple command docker compose up all containers are downloaded installed and started completely automatically but grant it more for us he wrote a menu to automate the generation of this Yama file according to our needs and he added some additional scripts to the last few weeks he worked hard to have everything ready for us all we have to do is to take a new SD card install raspbian and 2a update just the normal stuff then we follow his write up and start downloading the whole project with this command now we find all files in the directory IOT stack when we start menu dot Sh we see a selection of containers to choose from maybe this will change in the future if Graham decides to add other stuff first you install docker then you build the Yama file for the stack you can choose which containers you want to install as well as the additional nodes for node rate by the way piehole creates a hole on your network where all advertising is falling through like that it is no more shown in your browsers I strongly suggest that you install port ainur as it helps to manage your containers after this step our docker compose file is created and you can start the whole stack with docker compose up it takes a while but we can use the time for a cup of coffee the time is much faster than the execution of Peters Cargill's script because containers are already pre compiled and only have to be copied to our raspberry and if we start our stack of containers the next time docker will not download again and the start is much faster after a while we can try if the containers started up let's connect to node read on port 1880 yes it works of course it has no flows included we have to create them or add them using copy paste as shown in video number 255 you can also connect to port Ainur on port 9000 or to Gravano on port 3000 but how to connect to influx DB it has no web interface we have to connect to the terminal inside a container fortunately this is very simple this commands creates a prompt inside the container and if you type in flux you can look at your databases etc if you restart menu dot Sh you find some of the more common docker commands and you also find miscellaneous commands here are disable swap and install lock to ram execute them if you want what are they for log files are written continuously to your SD card and so reduce its lifetime if you install lock to ram log files are collected in RAM and only written every hour to the SD card much better and if you happen to have a Raspberry Pi 4 with plenty of memory you will see that even if it does not need the whole memory it starts to build a swap file with two undesired effects one the swap file often writes to the SD card and the performance of these writes and reads are very slow and two it shortens the life of the SD card so I delete the swap file completely make sure you never use more than the built-in memory otherwise your Raspberry Pi will block now our raspberry is on a similar level as it was after applying Peters Cargill's script only based on the modern container concept in Graham's description you find the commands to update the containers from time to time this is the charm of docker you always get the newest version when you install them and you easily can update but of course we want more we still have two items on the list automatic backup and remote access what happens if your sd card goes sour you lose everything of course you regularly can create a backup to a second SD card using RPI clone for example not very elegant my proposal is different with our containers concept we strictly separate valuable data from data that quickly can be restored from the internet because the valuable data is minimal compared to the size of the whole SD card we easily can back it up to the cloud and because Graham put all the valuable data into a directory called volumes we only need to upload this directory to Dropbox you can do that as often as you want you can of course also use Google or another service for that purpose if you know how Dropbox so far does not support the raspberries but fortunately andreyeva pretty wrote a Dropbox uploaded a link to the write-up on how to install it and customized the uploader is in the description the important thing is that you need an API key to access your Dropbox which has to be entered during installation of the uploader a script that exports the influx database and saves the volumes directory into your Dropbox is also provided in the scripts folder you can start it by typing docker underscore backup dot SH but of course you want that it runs automatically this is done in chrome tab we started with the command Krone tap - e and at this line this starts your backup always at 11:00 o'clock p.m. of course you can also write this line which starts the backup every hour now we can save the changes with control X&Y and you can check in your Dropbox if the backup arrives regularly now we are safe in case of an SD crash we only create a new one using all we learned before this will take us less than an hour and then we stopped all containers and copy back the volumes folder from Dropbox if we are happy also in flux DB works if not we have to delete all databases and restore the backup as shown in Gramps right up the last step is the installation of pi VPN it is straightforward you find the link to the setup in Gramsci top but how does it work we need two things a connection from our PC or smartphone to our raspberry and encryption of our traffic like that only our devices can connect to our home network and nobody can read what you do and yes you heard right we not only can't connect to the one rasberry we can connect to all other devices on our home network and even to the whole internet like that you always have a secure connection also if you are connected to a public hotspot and the internet thinks you sit at home like that you can watch all the geo blocked content of your country even if you are in a hotel in Timbuktu or in another remote place but first we have to overcome a few obstacles let's start with a first how can we connect to our raspberry which is safely sitting behind our firewall hopefully nobody can overcome this protection and anyway we even do not know the IP address of our home if we do not have a fixed one which is the typical paste for most of us to find out our IP address we could enter my IP address on Google it works as expected Google's anyway knows everything but unfortunately it changes from time to time because our internet provider does not have enough IP addresses for all of us so we use a trick we use a free service like Tok DNS here we create a subdomain with a fancy name this domain will always be the same if we regularly call doc DNS from our raspberry dark DNS gets our IP address and stores it till the next call we get this regular call by adding this line to crontab - e it updates our IP address every five minutes if we now connect to our fancy doc DNS domain our traffic is rerouted to our real IP address now we have a connection to our home the first problem nearly is solved but because we are outside the firewall nobody lets us in unless we open one port and forward its traffic to our raspberry this would be very insecure but because we installed PI VPN on our raspberry only our encrypted traffic is accepted on this port now the first problem is completely solved and we can go on to find out how we encrypt our traffic during installation pi VPN generates a key that has to be transported to the smartphone or pc of your choice the best is to use a USB stick the worst but the most comfortable as usual is to send it per email now you install this key into your remote device and because it also contains the information about our fancy duct dns domain you only can switch VPN on and you are safely connected to your home network like that I can have a look at the data of my weather station from wherever I am or watch the soccer game of my favorite team which is only available in Switzerland that's all for today no summary this time in the past I saw that most of you anyway stop whenever I start with a summary so it's up for discussion if I should include it also in the future I hope this video was useful or at least interesting for you if true please consider supporting the channel to secure its future existence you find the links in the description thank you bye

Channel: Andreas Spiess

Views: 850,833

Keywords: arduino, arduino project, beginners, diy, do-it-yourself, eevblog, electronics, esp32, esp32 datasheet, esp32 project, esp32 tutorial, esp32 weather station, esp8266, esp8266 datasheet, esp8266 project, greatscott, guide, hack, hobby, how to, iot, lorawan, nodemcu, project, simple, smart home, ttgo, wemos, wifi, tutorial, grafana, node-red, influx, influxdb, mosquitto, docker, container, vpn, pivpn, dropbox, Pi dropbox, raspberry dropbox, IOTstack

Id: a6mjt8tWUws

Length: 18min 18sec (1098 seconds)

Published: Sun Oct 27 2019