Take charge of your own Ambient weather data with Raspberry Pi, MQTT, and Home Assistant.

we've all experienced failure of a web connected component in our home automation setup due to api failure or some other internet related outage so today what i want to do is show you how i'm bringing in my weather data directly to home assistant using a raspberry pi an sdr receiver and mqtt this is a longer video but i hope you'll watch to the end so i show you how i get all the way from my weather station into a grafana graph so let's get [Music] started [Music] all right for this project you're going to need a few things to get started here you're going to need a raspberry pi of some sort and i have a raspberry pi 4 here and then you're going to need an sdr kind of receiver this is a nesdr mini 2 plus it's a frequency compensated crystal oscillator or temperature compensated crystal oscillator which means it won't drift so you need to make sure you have one of those you need a case to put it in if you choose to do that i just happen to buy the canna kit that has the case with it and then also you're going to need a decent power supply a decent power supply is to help power that usb dongle if you don't have a decent power supply then trying to power the usb dongle is not going to work very well this one outputs about three and a half amps so it's pretty decent powered it's also probably filtered so make sure you have a good power supply and then finally you need some sort of sd card this one came with a 32 gig evo samsung card so you need that to put your operating system and everything else on all right those are the parts lists okay so i'm going to assume that you have installed raspbian buster on your raspberry pi and that it's up and running and you're able to ssh to it now you can do this through a straight uh regular hdmi output to your monitor or whatever you want to do but you need to have access to the device obviously so that's the assumption here so we're going to start with installation of a couple of tools we need so first things first let's get rtl sdr installed so i'm going to make a directory called sdr i'm gonna change to that directory cd sdr and then i want to make sure that i have all my latest packages on my raspberry pi so i'm going to do an update and it's not actually getting my packages what it's doing is making sure i have all of the locations for all the files and everything so all my package locations and everything are up to date so that's what we're doing now let's go ahead and install these necessary packages apt get install we'll get git cmake live usb 1.0-0 developer version the dev version so we can build against the dev version and then build essential and it's asking me if i want to install all these the answer is yes so now we'll download everything and we'll get it installed okay so all that is installed so the next thing we're going to do is we are going to clone the rtl sdr packages so because we installed git we can use git so clone and then the location is git or the protocols get and the location is get.osmocom.org rtl sdr dot get and we're grabbing all the packages we're cloning cloning that repo all right so now we want to do is change to the rtl-s directory where we just downloaded everything to and we're going to make a directory called build and we're going to move to that directory so now we will make the file or make the executable for our tlstr we're going to send it some flags concerning the udav rules and the kernel driver these are super important to do or it's not going to work correctly all right so d install dash d install uw rules and dash the detached chrome driver so let's go ahead and make and compile and make these uh this package this executable and that was quick we're done with that now we're gonna um actually what we did before was we actually configured now we're gonna actually make it and now we're building all the objects that's complete so now we are going to make install we'll use the sudo for that so i can put it in the correct directories and we're going to run ld config as well all right so that is the installation of the rtl sdr binary executable whatever you want to call it so now what we're going to do is we're going to install the rtl 433 package and that's again the thing that's going to do the decoding for us whenever we receive a signal from our weather station or anything else there's probably 180 different things you can decode from the with this so first things first we are going to go back to our sdr directory and we're going to again we're going to clone a repo from github and this is the author of rtl430 murr bannon could be saying that wrong but here we go grabbing the github files and same thing we did before we're going to go into the rtl 433 directory this time and we're going to make a directory called build and we are going to change to that directory and we are going to go ahead and do our cmake this way but only if we spell build right compiling compiling that's done all right so again we'll do a make here all right that's done it took a couple of minutes because it's trying to or it's compiling objects or support for every single type of device that it can decode data for and there's quite a lot of them so it took a couple of minutes to do that all right now that it's built let's go ahead and install it so again sudo make install and ready to go now if uh everything worked out the way it should when we installed this package and the rtl sdr package we should be able to type rtl underscore 433 and start listening for for activity on the on that uh frequency and i'm going to use a frequency of 915 megahertz the 950 megahertz is where my weather station transmits it's an ism frequency and so let's uh give it a command and see what happens so rtl433 and the flag for frequency is dash small f or lowercase f and 915 m for the frequency and now you can see that it is it has found my micro tuner uh the sample rate is whatever that is something one million hertz um and then the tuner is set to auto gain and a frequency of 915 megahertz and we will wait and see if we hear any traffic on here here's our first bit of traffic what we're seeing is a fine offset wh-25 that is actually a little pressure temperature sensor that i have that's in the building here um it's set to display in celsius so what that tells me is this is all working like it's supposed to so let me do a quick little change here and we'll change the flag to show in fahrenheit since i'm in the us i want to do a dash c i believe it is customary and what that's going to do is it's going to start decoding and showing the output in fahrenheit instead of in celsius and you can see it's done that here we have still the final set wh 25 and a temperature of 74.5 degrees fahrenheit and then our pressure is going to be in inches of mercury instead of hesco hectopascals so i know that this transmits every one minute and four seconds so i'll get an update as it goes along here but what it also tells me is that my raspberry pi is working my sdr my little usb sdr is working and i have at least a decent or at least a working antenna so now what i can do is i can start sending data to my next step which will be the mqtt server so also you'll notice here that i just saw a bunch of traffic from it looks like a power meter i don't know whose power meter that is it's probably mine i know we have smart meters here in the us and in my neighborhood so it's quite likely that this is my meter i know that there's a solar panel set up on the house next door with two meters that are in the same proximity as my meter so it's very possible i'm hearing their meter as well but the point is you're seeing traffic here which is a good sign next up what we're going to do is we're going to configure home assistant mqtt so that we can take the data from rtl and put it into home assistant so let's get started on that all right now that we have the rtl 433 uh package installed on the raspberry pi what we're going to do next is we're going to come over to home assistant and we're going to configure the mqtt server to be ready to listen to packets coming in from the raspberry pi so let's start by going to supervisor and then we'll go to add-on store i'm going to search for mqtt and you get a couple of options here you'll get the official add-on and you'll get the community add-on i always lean towards the official add-ons rather than community if one exists so i don't have an issue if the community add-on goes away also in this community add-on you'll notice that this is deprecated which means it's basically going to go away at some point so again lean towards the official when you can i've installed the official mosquito broker already and in configuration it's all default so there's nothing here you can put login username and password pairs here but what we're going to do is we're going to use home assistance users to be able to authenticate the mqtt server all right so for that let's go back over to our configuration we'll go down to users i've got a few test users in here but i'll show you how i create a user pair use your password pair so the name is going to be mqtt test 3 because i have a couple already so mqtt test three will be the user and the password apc123 don't ever use abc123 in production ever ever ever in fact never do that ever that's bad going to use it here for testing and then i'll be deleting this after i'm done with the with the video filming so no issues there so we'll create that so now we have an mqtt test 3 user so that essentially basically or that essentially sets up mqtt to be ready to go on home assistant and what i'm going to do now is go back over to integrations go to mqtt and i'm going to get ready to listen to a topic i have nothing on my network that's sending me many any inq mqtt blah blah blah right now so i need to do some stuff over on the pie so that it will send some information over home assistant so for that let's go over to our terminal window and let's take a look at what we have here let's clear all the junk off the screen all right so what we're going to do is we're going to create a command line string that'll send mqtt data from my weather station and anything else actually that this rtl 433 hears over to home assistant so for that we've already installed the 433 binary we're going to give it a flag of dash capital c customary that gives us the fahrenheit and the inches of mercury pressure reading and then we're going to go ahead and give it the frequency we need a frequency so dash lowercase f 915 megahertz which is where my weather station happens to transmit and then i'm going to give it a flag of dash f which tells um rtl433 what kind of output we're going to want and for that output we're going to want mqtt so we're going to create a big mqtt string here and this is the host name of my home assistant instance port 1883 is one of the mqtt listening ports and because we are required to do user name authentication on that integrated mqtt server we will have to send it this username and password we created just a minute ago you can go without using username and password but you have to configure access control list within the mqtt configuration i haven't done that so i'm just going to go straight up with the username password it's the easy route versus the acls if you have devices that are going to send this mqtt server they don't have any authentication you're going to set up access control this for it to work so anyway there's my user is gonna be um what do we call it mqtt test three and then uh comma and then the password pass is going to be abc one two three um and then for what we're sending we're going to be sending events which are basically what the topic is going to be for us to listen on the other side so my events are going to be formatted this way the name of the program and and really as far as this first part goes i think you can use anything you want i'm just going to stick with rtl433 and then i'm going to give a flag here that gives me the model of the device that i'm getting events from and then i will also give it the id number of the events oops how do we do that let's let's see what we did here so test three somehow i deleted everything so pass equals abc123 and then events equal rtl underscore 433 and then a bracket slash model and then another close bracket another open bracket slash id close bracket and then closing quotes let's go ahead and hit enter and just uh cross our fingers and see what happens okay good all right so what you're seeing here is that we've got mqtt connected and our mqtt connection is established so you won't see any more output on this screen all the output is now being for fed over to our mqtt server at this ip address and port using this username and password so let's go back over to the browser and let's take a look and see if we're getting anything into our mqtt server so if you remember we used rtl slash underscore 433 slash model slash id number and we get the model and the id from the mqtt or from the weather station or other devices every device sends that information over and so we can distribute or we can pull it out and then build our our sensors off of that but i want to see everything that's coming into my rtl underscore 433 events so i'm going to do a rtl underscore 433 slash hashtag the hashtag basically says send me everything that starts with rtl 433 that's coming into the mqtt server and we'll start listening and right away you can start to see some stuff i find it interesting that i can just put a radio out there and start listening to things i don't know what this is i'm assuming a neighbor's security system um basically giving me the switch states of everything in the house or in the in this security box or whatever it is i don't know what it is specifically um so we go down and we look through and we see um it looks like we have a power meter uh more power meter stuff and more power meter stuff and then what we have here is a wh25 which is my temperature and pressure sensor that's a little standalone device i have here in the house and it's giving me temperature humidity pressure in inches of mercury and then some other information so keep in mind that this is all coming to us in json because rtl433 when it sends an uh an mqtt packet it formats it in json and then the mqtt server picks it up and and decodes it here so what it tells us number one is that our raspberry pi is working we're receiving data on the sdr the little receiver rtl 433 is processing and decoding that data it's feeding it out to the mqtt server and our mqtt server in-home assistant is picking all of that information up so now we need to figure out what we're going to do with this you've got to create some sensors now in order for home assistant to do something with this so let's go over to our terminal and i'm going to take you through some of the sensors that i've already created i've created one for temperature humidity rssi rain wind speed wind direction all the stuff you would typically get from a weather station and so you can see how we do this here the platform is mqtt that tells home assistant to look at the mqtt server for its information the state topic is what we talked about before it's this is an event rtl 433 underscore model or i'm sorry slash model slash id number when you're looking at your feed you have to determine what you're looking at and you have to create the sensor the state topic based on what you're you're looking at so i know that this is that little well actually this is my backyard weather station so i know that that's what that is so i created a sensor to be able to view the temperature data from that so i have a name of wh 65b temperature and that's the sensor package that's in the weather station my unit of measurements fahrenheit force update is true force update on these sensors means that even if the data is the same if it receives a value over mqtt and there's no change in the data it's still going to register that and it does that i do that because i want to graph this over time and if i don't put true and let's say the temperature stays at 70 degrees or 75 degrees for three or four hours then i'm going to have nothing on my graph for that three or four hours and it make it'll look to me like something has happened it's not getting data so i set this a force update true so i always get a data point regardless of whether it's the same data as before and then the value template is the money template here so what it's doing is it's looking at the state topic if we go back over here you'll see this packet has a has a temperature underscore f now keep in mind i'm looking at this this is a little bit different event or state topic it's the wh 25 216. it's the same principle you're going to get a packet you're going to get this state topic you're going to get the information here and so what i'm doing with the value template is i'm taking that value of temperature f out of this sensor or out of this packet and then what's happening is it's being decoded into an actual value and what this is going to give me is the actual temperature fahrenheit from this particular state topic so every time that comes in from my weather station it's going to read that it's going to create this value temperature template and it's going to put it somewhere you know in home assistant you've got your history ability to store things in history it'll do that if you're displaying it on a dashboard it'll display it on a dashboard in my case i'm sending it to an influx db through my home assistant and then i'm going to be graphing that on a grafana visualization so it it's the same thing for all of these again platform is mqtt here is the the outdoor weather station this time i'm looking at humidity the name can be anything you want it to be and if you look at developer tools you can actually search for these and of course developer tools is over here on the left hand side so you come down here to developer tools click that and then you get this screen here and i'm going to click on or i'm going to search for wh 25 i'm going to do 25 because that's um what i actually am getting from my feed right now so you can see if i go into my terminal and i look for that wh25 sensor so you can see here that i'm looking at this wh 25 216 right in this area and my sensors will be 20 wh25 temp f if we come over here to the two developer tools we'll see this wh 25 temp f and what it's doing with the names it's actually putting underscores lowercasing everything putting underscores so my wh 25 underscore temp underscore f is the same thing as my wh 25 temp f just they just put underscores and lowercase everything so whatever you name this is what your sensor name is going to be so let's go back up a little bit i have a couple of caveats to talk about so in these weather stations at least the one i have i don't get feels like or heat index and i don't get pressure those are all in the actual head unit or the display unit the display unit of the w the ws2902 doesn't transmit anything all it does is send stuff out to the various services like weather underground ambient weather some of those right so i can't display those natively from the data stream i'm getting from my weather station so we had to do some work on that my brother who's uh smarter than i am he actually worked with me on this and we were able to build this value template which is massively long but it's the heat index calculation that we get from these other sensors we take the temperature and humidity we do a lot of uh calculations on it it brings back our heat index so it is possible to take sensor data from your weather station and actually make more sensor data from it so when you create or after you've you've done the work to create the sensors you wouldn't want to definitely make sure that you go back over to your configuration and you're going to do server controls go to server controls here click on check configuration and make sure your configuration is valid for all the sensors you created and then restart home assistant now that we have data flowing in from our raspberry pi through our mqtt server into our sensors and home assistant it's time to do something useful with them so let me create a couple of graphs just to to plot the data so whenever you create a sensor and home assistant and assuming you have influx db setup and grafana setup and i've got other videos that i show that in you should be able to just start taking the data from the influx db and putting it in grafana it should automatically be going over there unless you've filtered that out so let's go to grafana and i'm going to create a new dashboard so dashboards manage and new dashboard and it first wants me to add a new panel which i will do and this is going to be a graph panel because what i'm going to plot is i'm going to plot the temperatures that i'm getting from the little sensor box that we're using that's inside my house and that's the wh25 topic that we talked about earlier all right so nothing on the panel right now and when you go into this you want to make sure you're using the correct data source correction up here correct data source so i'm using my default data source i'll just keep it at that that's where all my home assistant information is going into and then i'm going to select a measurement and these are all the measurements that are in the influx database that the influx database knows about in that default database if you choose a different database it might have different measurements in it but these are all the ones that it knows about and if you remember if we go back and look at our our our sensor setup here you'll notice that we specified unit of measurement that's how grafana knows what kind of measurement it is so we'll select a fahrenheit because i know that's what we're getting i know that fahrenheit's all the way at the bottom here so i'm going to choose degrees fahrenheit when you do that first of all you're going to get everything that has a degree fahrenheit reading in your whole database which we don't want we're going to further filter this using a where clause or where statement and have a couple choices i can either use a domain which i hardly ever ever do or i use entity id and the entity id is going to be filtered automatically by grafana for anything that contains this degrees fahrenheit measurement so if i select that tag value all of these particular entity ids have a fahrenheit measurement but i know that i created the wh 25 underscore temp underscore f sensor and that's when i want to measure or what i want to select for my measurement and you can see right away that we get data in this graph and let's uh name this you can see right here it says f dot mean that's because it's taking this and this and creating the default um legend for that so i'm just going to call this set fahrenheit you could leave it blank i guess you want to just leave it this way and it'll change that to fahrenheit and the reason i do fahrenheit here is i'm just going to say indoor temperature and when i select indoor temperature i will know that this line is fahrenheit you could if you had humidity or something else plotted on here you could also just change this title to indoor and then your two different uh graph lines would be different humidity fahrenheit and it would be specified via this legend that's all subjective however you want to name it however you want to put it together is up to you we can do a bunch of things with this the first thing i want to point out is you can see how often this is updating it should be one minute and four seconds for that device it'll send me a a an update through mqtt if you remember what i said earlier about the force and i'll show you here if we set force update to true then all of these spots right here are going to come across as points on the graph if you didn't set force update to true that sensor would only update during a change you would only see a dot here a dot here a dot here and a few over here you wouldn't see all this in between that's why if you're graphing this it's important to set that to force update true if you're just using a dashboard or something where you just want to display the latest temperature you probably don't want to use force update because what you're doing with force update you're also creating entries in your database and i'm doing one minute four seconds for this my weather station updates every 16 seconds and so i am going to keep all of the plots or all of the data points but if i'm just displaying it on a dashboard and i just want to know what the current temperature is and if it hasn't changed it's still the current temperature right uh that way i'm not filling up my database with useless information but i'll keep it forced true for plotting it over here on the side a bunch of different things you can do bars you can change this to bars you can do um your line widths you can do a uh area fill uh you could fill it if you wanted to area filling works with a line um my fill gradient would be something like that and what else do we have here if you want to connect all the points together then you would come down here and rather than having a null value would you connect it and it's going to generate a line that's where the fill comes in so you could turn the fill off you could turn the fill on make it less visible the gradient makes a difference again these are all just personal preference how you want your graph to look you can also turn on your points whenever you have the fill on and i set my point radius to about 0.5 this way if i have a line a connected line i still want to see how often the thing is updating and this tells me whether or not i have blanks or gaps in my data by looking at the points like this visually quickly um what else can we do we can do legends we can do the uh where's it at here we can show the values min and max so minimum was 72 maximum was uh 73. you can do an average you can do current uh you can set this as a table so it then makes it in table format you can put the table on the right-hand side so if you had a whole bunch of different sensors on one graph or one visualization you could put those and then make a table out of them i don't necessarily need to do that so i don't show any of that you can show the legend again if you want to uh we'll take it as table to the right and take off current so then you just get this label for what this is if you set the temperature here to temperature fahrenheit indoor temperature fahrenheit you can get rid of that label completely again i'm just showing you um a bunch of different options you can use for this if you want to look at the query the way this query actually looks you can click on query editor and now this is the actual influx query select mean value from temperature fahrenheit where the entity id is this and the time filter which is this time filter grouped by the interval and then the fill for this is null we're not filling any values in between the updates but it's very simple at least to me right so i'm used to using mysql and sql queries this is a influx query it's a little different but pretty close so that's what the query looks like if you were to write it out um all right so i'm pretty sure that covers getting the data all the way from the raspberry pi through to the mqtt server into home assistant and then into some sort of graph or useful information you can also go in and you can add um this same stuff now that you have a sensor if you want to go into your whatever favorite dashboard you have if you have displays around your house or something like that you can do this so i have currently this is my current outdoor weather station and that's coming through the ambient api once i'm comfortable with where my weather station signal is and the data i'm getting i will most likely replace this or at least add another another block next to it so that i can compare the two make sure that they're doing what they're supposed to um other than that i think that's all we have for today i appreciate you sticking through to the end of the video i know it's a longer one but there's a lot of steps to get this working and if you can calculate things like the heat index if you can calculate the pressure which there is actually a pressure calculation formula as well you can create a sensor similar to this one for pressure then you have all your data to yourself if you lose internet connection or you lose a connection to the api or the api goes down or for whatever reason api access is discontinued you still have access to your weather station data you can still put it in your your dashboards you can plot it you can store it and that's what i was going for if you have any questions leave those down below in the comment section make sure you hit the subscribe button hit that bell icon in case you want to be notified of new videos when i make them and if you feel like you want to help support me in in this endeavor throw some bucks in my kofi account or become a patron on patreon and i'll link those down below here in just a minute to the description so thanks for watching and we'll see on the next video

Channel: mostlychris

Views: 35,429

Keywords: Ambient Weather, Home Assistant, API, Grafana

Id: vRO9zzrlbPI

Length: 36min 39sec (2199 seconds)

Published: Thu Sep 10 2020