Grafana Weather Dashboard on a Raspberry Pi using InfluxDB and an ESP32 - In-Depth Tutorial

following my previous video on running a weather station dashboard on the rear terminal there were quite a few requests for me to do a more in-depth tutorial on how to get it all set up and running so today i'm going to be running through each part of the process step by step so you can get a similar setup running on your raspberry pi in this example we're going to use an esp32 as our data collection node and this will collect temperature humidity and pressure readings from some attached sensors it'll then post these readings to an influx db database and we'll then use grafana to visualize the data don't worry if you don't understand what these are how they work together just yet i'll explain them each in more detail as we work through them let's start with setting up the esp32 microcontroller we're going to connect two sensor modules to esp32 the dht11 sensor on the left connected to pin 4 for measuring the temperature and humidity and then a bmp 280 sensor on the right connected to the itc interface to measure atmospheric pressure you've probably seen people use a raspberry pi for this part as well but i prefer to use a microcontroller like the esp32 or an arduino for a number of reasons the first and probably the most important is that once you've programmed your microcontroller you can switch it on and off as often as you like without having to go through a shutdown or boot up process you don't have to worry about it corrupting its memory card or locking up it's entirely dedicated to the task you've programmed it to do and it'll do it very reliably even when you've got power interruptions the second is its power consumption which really comes into play for building a battery based project you'll struggle to get even a raspberry pi 0w to run it under 260 milliamps or a little over 1 watt while i've managed to get an esp32 based device like this fire beetle board used in my weather station to run as low as 0.01 milliamps or at an average of 1 milliamp when transmitting data over wi-fi so this is over 250 times more power efficient meaning that it can either run 250 times longer than the pi or you can use a battery 250 times smaller than the pi both of which can be a huge advantage i'm going to connect two sensors to my esp32 on a breadboard using some jumpers for this example the sensors i'm using are from seed studios they call them grove sensors and they're particularly useful for people who are new to electronics because they include all of the components required to drive the sensor on the pcb so you don't need to worry about adding resistors or making multiple power connections to drive them we can then connect our esp32 to our computer and use the arduino ide to program it so we want our esp32 to take readings from the two sensors and post these readings to an influx db database over wi-fi influx db is a time series database package which you can think of as a sort of large excel spreadsheet where each data point that you send to it will be stamped with the dates and time it was received so if you send it a temperature reading it'll stamp it with the dates and time it was received and then stored in the table this makes it really useful for any information that changes over time so it's perfect for something like a weather station where all your metrics are time based this database can exist in the cloud on influx db servers or it can be run on our local network on a raspberry pi i'm going to use influx db servers in this example because this option works well if you've got a number of microcontrollers collecting information that are not all on the same network it also allows you to reboot and make modifications to your power software without interrupting the collection of data either way the setup process is largely the same for both options so let's have a look at our code and how to get the information into our database i've put together this example sketch which i'm going to load onto it you'll need to have installed the esp32 board using the ides board manager i'll leave a link to the tutorial that explains how to do this in the relevant section of my blog post which will be linked in the video description next make sure that you've got the board selected as esp32 dev module and then check that using the correct com port in the code we start by importing our required libraries these are basically packages of code that have been written to make it easier to interface with certain devices or objects so for example the dht.h library contains all of the code required to communicate with our dht sensor and we can then just use a single line in our code to ask this library to send us the temperature reading we've then got a section to define our wifi and influx db connection settings don't worry too much about these yet they'll make sense when we get to the influx db setup portion of the video we then create two sensor objects using our libraries create three variables to store each of the fields we want to measure and then set up our influx db database connection we then have our setup function which is a set of code that runs just once when our esp32 is first switched on this function starts communication with our computer for debugging then sets up our sensors using the libraries we called up earlier then connects to our wi-fi network and then connects to our influx db database next we have our loop function this is a set of code that is run over and over as long as the esp32 continues to be powered so in our case we add another line of readings to our table in our database each time we loop through this set of code first we use our dht11 and bmp280 libraries to get the current sensor readings from the two sensors we then tell our database library which column of our table each of these values should be stored under we then check our wifi connection and reconnect to the network and database if we need to and we then write our measured values to our database as a single line of data so the three measured values will all have the same timestamp this last portion of the code just displays the measured values on the serial monitor this makes it easier to debug because we can now check that the information that is showing up in our database is the information that our esp32 is measuring before we upload the code to our esp32 we need to add those network and database parameters at the beginning of the code so let's do that now the first two are wi-fi network name and password remember that the network name is also case sensitive and needs to include any spaces or underscores as well the next five parameters are all required for influx db connection so let's open up influx db in our browser and get those you'll then be given an option to log into your existing account or sign up for a new account i'm going to register a new account for this tutorial once you've confirmed your email address you'll arrive at a page like this you can leave the default amazon web services provider just select your correct region then enter your name or company name and check the box if you agree to their service agreements you'll then be asked to select a plan unless you're wanting to capture large amounts of data or store the data for long periods of time the free plan will be fine you should then arrive at your home page influx db have already done some work in interfacing with arduino-based devices so we can just take advantage of that to make the integration easier so let's click on load your data and then find the arduino client library now you'll see two boxes with a generic apr token and bucket and below that are some pre-populated snippets of code we can then just copy these fields across to our arduino sketch my sketch is based on this example code so you can also just copy the whole snippet and replace the section in my sketch if you'd prefer it's perfectly fine to use this generic token in buckets but if you're going to be using multiple devices to capture data and if you want to set an expiry on your data so that it's automatically deleted after a period of time then you'll want to create your own first place that allows an external attitude to read write and manage the information in our database we can set this up with customizable options to limit what the device with the key can do but for simplicity let's just create an all-access api token this will enable our device to read write and modify our database give the token a name which is commonly the device name and then click save we can then copy this token and replace the one in arduino sketch next we want to create a bucket which is essentially a fresh database in influxdb each database needs to have a retention policy the retention policy is how long the data is stored in the database before being automatically deleted this prevents the database from becoming bulky and slow to query so it should be chosen based on how often you're writing data to it and how long you expect to need to use the historic trends in our case we're only going to be writing to our database every few minutes so let's go with 30 days this will allow us to visualize monthly trends as well as highs and lows on our dashboard make sure that the new bucket is selected and we can then go down to the code to copy the field across to our sketch the last field is the time zone this should be set correctly so that your database entries are stamped with your local time which you'll need to establish for yourself they give you some examples for common locations and a web resource for naming other locations mine is set up for sydney australia with our influx db and network settings added to our code we can now upload the code and make sure that our esp32 is writing the information to our influx db database i'm also going to quickly change the delay between readings to 5 minutes as the indoor temperature and humidity don't change that quickly open up the serial monitor and check that you're getting the expected feedback from the esp32 this also gives you the latest readings that are being taken from the sensors any errors in connecting to the wifi network or influx db should also come up here now we've got our information being sent to and stored in our database but it's pretty intimidating to look at a table full of numbers so we need a way to gather the information we're interested in and display the information in a format that's easier to visualize and that's where grafina comes in influx db does have its own visualization tools but i find that grafana's a lot more powerful and user-friendly as with influx db grafana can be run in the cloud on their own servers or on a raspberry pi on our local network for this example i'm going to be installing and running it on my read terminal which is a raspberry pi based touch terminal i'm starting with a fresh install of raspberry pi os buster and we can then install grafana by entering the following commands into our terminal these commands will all be listed in my blog post linked in the video description as well once we've got grafana installed we'll need to start the application you'll see in your terminal window from the previous step that it explains to you how to start the service and verify that it's running so let's enter those commands you should then see a green message saying that it is running lastly we need to set it to automatically run on startup we can then access the grafana dashboard by opening up a new browser window and entering localhost on port 3000 you'll then need to enter your username and password the defaults are admin for both and you'll then be prompted to create a new password now that we're on the grafana user interface page we need to configure data source to be able to read the information stored in our influx db database so let's click on add your first data source we can then choose our data source tab which in our case is influx db and will then land on a configuration page like this this is a similar process to the one we followed to allow esp32 to write information to the database we're now just telling graffana how to read the information from the same database you can change the name of the data source if you'd like i'm going to leave it as influx db next we're going to change the query language to flux flux is more of a scripting style query format rather than a sql query format so it is a bit more versatile than influx ql the url is the same one we used in arduino code so we can just enter it or copy it across from our code you can leave most of the default settings selected for now let's add our username and password here then lastly we need to enter our database details now we need to add a token for grafana to be able to access our database this is also similar to the esp32 token except that that token could be a right only token and this could be a read-only token so let's open up influxdb to generate this token we'll go to tokens and then click on generate api token i'm going to create this as an all access token again but you can configure it as a read-only token if you'd like to then we just need to copy the generator token over to our configuration page i'm going to set the minimum time interval to 60 seconds as we're only writing to our database every 5 minutes and i'll leave the max series as 1000. if it has all worked correctly then you should get a green tick and it'll tell you how many buckets had been found so that's great we can now move on to creating our dashboard let's go back to our dashboards home page using the menu bar on the left we can then click on this box on the right hand side to create our first dashboard a dashboard is a page that is used to display a set of information we are going to create a single dashboard to display our three weather parameters each of the graphs or gauges we want to display can then be created in a panel so let's create our first graph by clicking on add a new panel we then get a page like this which gives us a number of options to set up what data is displayed and how it is displayed it might look quite intimidating the first time you land on this page but it's actually quite simple once you get the hang of it using the toolbar on the right side you can configure how you want the information displayed like the title access labels legends and colors at the top of the section you can choose the type of visualization using the toolbar at the bottom you can configure your database query which selects what information from your database is displayed let's enter our first query to get the temperature information from our database we start by telling it which bucket we can read the information from we then select the range to look at we're going to look at the past day we then need to add a filter to it to tell it which field to look at we're going to be using temperature first lastly we're going to name the set of data returned as results if you click on the graph area it'll run the query and you should now see some trained data being displayed as a time series graph we can then configure how this information is to be presented by making changes to the graphs parameters on the right try changing each of these fields to see what effect it has on the displayed graph you can't really mess it up too much and if you do just set it back to the original value or setting let's now do the same for humidity and pressure as well we've now got three graphs which show a time-based trend of our data but it's difficult to see what the current values are so let's create an instantaneous gauge for each of the parameters by following the same process we can use the same query as we've done previously but instead of one day we only need the data recorded in the past five minutes as this is how often we're taking new readings gauges are quite a useful visualization tool as you can set up limits and define colored thresholds or bands showing the normal and extreme conditions for each metric or field i'm really just scraping the surface of what you can do with each of these visualizations so you should really read up a bit more on what options are available and try customizing yours further the last thing i want to add is a bit more information on the higher low and average value for the past 24 hours we can do this quite easily by adding these values to the graphs legend search for each value in this text field or select them from the drop down list this dashboard looks fine if we're viewing it on the web but if we want to use it on something like a touch display then we don't want the browser and toolbars to be visible we just want to see the information on the dashboard we can do this by first hiding the taskbar and browser toolbars by setting the browser to full screen mode we can then hide the grafana toolbox by cycling through the view mode using the icon on the top right we also need to then resize the panels to suit the full screen mode on our device and that's it we're now done creating our first grafana dashboard from data we're recording to our influx tv database using our own sensors and microcontroller [Music] remember to save the dashboard before exiting or leaving it running this is a really basic example there are loads of tools and plugins available to create really powerful graphics and trends so do a bit of reading up on them and try creating some unique ones for your data let me know if you found this tutorial helpful in the comments section below and let me know what you're going to be using your new grafana dashboard to monitor thanks for watching please remember to like this video if you enjoyed it and subscribe for more tech and electronics projects tutorials and reviews [Music]

Channel: Michael Klements

Views: 75,412

Keywords: raspberrypi, influxdb, grafana, raspberrypi4, esp32, arduino, arduinoide, flux, fluxsql, dashboard

Id: 7M8MHa6W9w0

Length: 19min 32sec (1172 seconds)

Published: Thu Jan 27 2022