Tracking Utility Meters with SDR // MeterMon & RTL-SDR

hey everyone it's ryan with smart house and on this week's video we're going to show you how to set up a system that can read wirelessly for smart enabled meters now if you watch the first video in our energy series which is linked here above you saw how i installed a fairly inexpensive whole house energy monitor so i can see the electricity consumption in and out of my house now that's excellent but of course that doesn't help me for my natural gas or my water consumption i put together a system that can receive data that's wirelessly transmitted from my utility meters now in my case i have three main utilities that come into my house electricity natural gas and water the first one electricity i discovered uses an encrypted channel that i cannot access it actually uses a secure variant of zigbee to mesh together all the houses in my neighborhood to transmit my power consumption back to the utility so i was unable to retrieve data from that particular meter however i discovered my gas meter transmits in the clear and i'm able to pick that signal up out of the air so i'm able to get about every minute data coming from my gas meter my water meter on the other hand was actually battery operated and so after discovering at the water meter it was battery operated uh one of the comments said that you will just have to randomly pick out whenever it transmits and up until last week it was transmitting about once maybe twice a month after they came through my neighborhood and did some maintenance on the meters now it's transmitting every hour or more than that so i'm able to pick up that data it's not real-time data but it's able to see the consumption graphed over time so what we're going to be doing today is using what's called an sdr or a software defined radio so what is an sdr if you think of radios in the old school sense you've got a set of hardware that's been tuned to a specific frequency or band of frequencies that it can pick up and then receive data music whatever from that specific set of frequencies now with the new technology of software-defined radios you can define what frequency band that that hardware picks up within its tolerances so what i've done is i've picked up an inexpensive sdr to usb device off of amazon and i've implemented it in using a couple of projects that i found on github now all credit to the development goes to these developers i didn't develop anything i just kind of packaged different systems together and made it set up for my home assistant usage so today we're going to be using a couple of different projects the first project we're going to use is rtl sdr and this is a linux accessible application that allows to use the usb sdr in linux to be able to pick up a specific frequency now the cool thing about this tool is it can respond to network requests so we're actually going to set that up and have it running on our on a raspberry pi to receive all the incoming signals and then we're going to use a docker project called metermon to tell that application what specific frequencies to look for bring that into the docker container and then push it out over mqtt sounds complicated but once you get it set up it kind of runs on its own and you don't have to worry about it so i've been running this for about three months now and it's pretty stable i've had to restart the docker container a few times but other than that i haven't had to do any intervention until today when i wanted to record this video so now we've kind of looked over what the project's going to entail let's jump into the steps before we get into the installation of the meter mod software let's go over what's required for this project the first thing you're going to need is some sort of linux based computer i'm actually going to be setting this up on my spare raspberry pi well i say spare i had to actually take this out of my 3d printer but you'll need something like a raspberry pi a linux box anything that you want to leave running constantly that can take a usb connection in now i will say if you are a person that runs plex or has some sort of other device that uses a tv tuner you will not be able to have them run on the same box that's because we have to blacklist the tv tuner drivers because the sdr and tv tuners are very similar so you can't have them coexist on the same box that you're running a tv tuner or recording live tv on next you're going to need a software defined radio now this is the one actually out of my live instance and this was no antenna attached that's because i already have the antenna zip tied and i don't want to cut it all out so the final setup looks very similar to this i've actually got it zip tied to a pegboard in my closet this is the actual sdr uh connects over usb and has a an antenna port on the other side so a lot of these kits come with different antennas and i've actually got a link right here below the kit that i purchased from amazon so the final requirement is an mqtt broker now if you don't have one of those set up i've got a link up here in the cards to my esp presence video that takes you through how to set up an mqtt broker in home assistant it's very quick and easy if you want to use that link i've got a timestamp here where that particular section starts so with all the requirements out of the way let's jump into the install all right so the first step we need to do is get docker set up now if you already have docker set up on this linux box you're using you can skip the step and go on to the next one but if not i'll take you through the quick and easy way to set up docker on your device now this is a fresh out of the box raspberry pi all i've done on there is run sudo app update sudo app upgrade and rebooted the pi and that's a significant step if you have just done a app an upgraded update make sure you reboot or the script will fail so the first step we're going to do is go ahead and grab the convenience script off of the get docker home page so we'll run the following command here it's a curl command that's going to grab the getdocker.sh script and download it to your machine all right so next we're going to we're going to execute the getdocker.sh script now i'm getting this error message because i had already attempted to start the docker install before i rebooted so if you get this and docker's not functioning go ahead and let it continue and now it's going to execute the install script now the docker install will take a couple minutes depending on how fast of the pie that you're running it on is if you're using older raspberry pi 3 it might take a little bit longer my raspberry pi 4 took about a minute so so the install's done our the next step is for us to go ahead and give our pi so the default user on the raspberry pi the pi user access to the docker group so we're going to go down to the bottom here and type in sudo user mod dash a capital g docker and then your username is pi so that's done and then finally we're going to go ahead and test to make sure docker is working so let's run docker version now if we check the version on docker you can get this error message about permission denied while trying to connect that means that your permissions haven't been added to your user so you can either reboot the device or go ahead and log off and log back in again all right now that we've applied the group settings and rebooted to apply to our user account let's go ahead and make sure that docker is functioning and yes it is all right so next let's go ahead and run the docker run hello world just to make sure everything's working there it is all right so our docker is fully functional and ready to go so let's go ahead and get the software installed that we need all right so now docker is installed let's go ahead and get the prerequisite software set up and we will test our sdr to make sure it's functional so the first things we need to do is we need to blacklist the existing usb radio and tv drivers that come with linux so it linux is set up so you can plug in like a tv tuner card or a fm radio card and be able to use that to receive data well we need a blacklist though so it doesn't load those drivers and we can instead have it load the drivers we need for the sdr so the first step to do that is let's go ahead and all right so we're going to create a new file uh in our slash etc modprobe.d directory and this is going to allow us to blacklist those drivers so we're going to run sudo nano slash etc slash modprobe d slash blacklist rtl.conf then we're going to paste in the following lines and again these are available on the blog post you don't have to worry about typing them in you can just go to the blog post and copy paste them in to your window and save it once we get those pasted in we'll hit control x to exit and then hit yes to save now we'll go ahead and reboot the pi to make sure it accepts those changes okay so now if we look we can see that it has our realtek semiconductor device properly loaded so let's test it to make sure it's actually can see things so now that the device is properly loaded we need to install software that will allow that will allow the linux device to use that sdr and grab all of the active meters in your area and pass them on to the docker container we'll create later so we need to go ahead and run sudo apt install rtl-sdr this is going to download a tiny little application that we need to proceed all right now that's been installed let's go ahead and test it to make sure it can see our sdr and our sdr is functional all right so to go ahead and test to make sure your sdr is working let's run the command rtl underscore test and you should start seeing some information about being able to sample so as long as you don't get any main error messages other than the pll not locked everything's fine so we'll go ahead this means that we're ready to go let's go ahead and exit by hitting ctrl c and we'll proceed forward to the next section in order to have the radio active and available to the docker container we need to go ahead and insert a bit of code in our startup command so that this runs in the background so to add the startup command let's go ahead and do sudo nano slash etc rc.local and you should already have an ip address startup that comes up automatically when you do when you boot up your system but we'll go ahead at the top here we're going to add in the sleep 10 so it waits 10 seconds so we make sure we have a valid address and then everything's ready to go then above the fi we want to we want to have slash usr slash ben slash rtl underscore tcp space dash a space dollar sign underscore ip space ampersand now what this does is this is going to when the system starts up it's going to wait 10 seconds the pre-built script is going to go ahead and identify the ip address and store it in that variable the dollar sign underscore ip then we're going to execute our slash usr bin rtl underscore tcp and then that dash a indicates what address it listens on and then it's going to go ahead and insert the current ip address of the system and then that ampersand just lets linux know hey run this without a user intervention so i'll go and hit control x to save and then hit yes if prompted then we'll go to test this we're going to go ahead and reboot the system so sudo reboot so we want to make sure that this starts up every single time all right so now that our pi's back up again let's go ahead and make sure that it's functioning all right so to verify everything's running let's go ahead and run psa you x this is going to show us everything that's currently running on our system we're going to add a pipe grep to search and rtl underscore tcp and there we go we have a and there we go our system is executing it with the proper ip address so everything looks good there let's go ahead and get the metermon docker container set up and start scanning for meters all right so now we have docker set up and configured and we also now have our and we also now have our sdr functioning and ready to accept commands over the network so the next step let's go ahead and get metermon set up so we want to go ahead and do run a docker pull shot off metermon and then we want to make sure we're using the arm so since we're running on a raspberry pi i'm going to go ahead and run it with the arm tag at the end if you're running this on something like a standard server or a workstation you want to run it with the the amd64 image or you can also execute latest so we'll go ahead and let this finish pulling the arm version for my raspberry pi and then we'll get it set up all right so while we're waiting for it to install let's go ahead and take the docker run command that we're going to use to actually execute this and we're going to add in a few variables that we need so if you look here we've got the this is actually the example that he gives from the website um so you could copy it from here and modify it or you can go to the blog post and grab my modified version that has the uh corrected variables that has the correct variables that so the first thing we need to do is change our mqtt broker host to our ip address of our mqtt broker in my case 192.168.1.5 then since i require authentication in mqtt i need to go ahead and add two more extra environmental variables that will allow us to access it so for every environmental variable you need to add a dash e and then you can go to his website you can go to the you can go to the github site and grab the required variables that you need so in my case it's going to be mqtt username equals username and we're going to add another one for for password so we need another dash e mqtt password equals password now if we scroll over a little further we've got to define the rcl tcp server so this will be the ip address that you had in the last step 192 168 1.36 in my case now don't forget to add the port so 1 2 3 4 is the port number for it so all together it'll be the ip address of your device that we just set up the sdr on colon1234 now there are some other topics that we can set in here that we might go through later on so for example you can change and customize the mqtt topic prefix so right now it's going to come up as meter mon as the root topic you can change this to whatever you want you can also change the filters so later on i'll show you how to filter and only pick up what devices you care about but for right now to confirm it's working we want to just go ahead and capture everything so now that we've got those extra variables added in the last thing we need to do is scroll to the end here and change the tag to arm to make sure we're pulling the arm version because i'm on my raspberry pi for you if you're running this on a linux standard linux box then you can use amd64 so let's copy that whole command all right so let's go ahead and paste in the command that we wrote up earlier alright so there we go now we've got this is using the mqtt explorer app so again this is a free one you can download but now we'll start seeing there we go we now have some devices coming through now what you're going to notice here though is you're going to have a bunch of random devices popping up throughout here and they're not going to show up as separate topics we needed to find some extra variables here before so um but this confirms it's working so now we can see both gas so meter ids and consumption values coming through so we know everything is functioning everything's good now we just have to tune it for our specific devices all right so now that we've confirmed that everything is set up and ready to go let's go ahead and filter out only the devices that we care about so i'm going to pop over here into my notion document that i have with all of my house meter with all of my meter information now this picture here is a close-up of my battery-powered endpoint encoder that's attached to my meter and you'll notice the code here on the side 312-97404 that's the id that i need for the water meter to be able to receive data from it and if i scroll down here a little further i've got a picture of my gas meter with its id 66154761 now it's a little hard to read on this picture but that's the idea that it's going to be transmitting out along with this consumption data so we need those two values and we need to add those into our add those into our docker script so let's pop back into our docker script and let's go ahead and see how we add those in there so again this will be available in the blog post this is going to be called the full metermon docker script so will be at the bottom this will contain the filters and the additional keys that i use so the first environmental variable we need to add so dash e rtlmar underscore filter ids and that's the two ids that i filtered out from before so the water meter and the gas meter then another dash e and then this variable called meterman send by id what this is going to do is it's going to force meter bond to send a unique topic for each individual id that it receives so we can grab that topic and put it into home assistant and finally there's this rt rtl amr underscore unique equals false so what this does is it by default it will filter out duplicate transmissions from your meter now that means if you if you have not consumed anything it's going to ignore every updated information i actually want to see that because i want to see that i've consumed no gas so for example in my house it's the end of summer beginning of fall here we're not really using a lot of gas we do use gas primarily for our hot water heater whenever we take a shower or run the washing machine but i want to know that i'm not consuming any gas so even if the consumption value is exactly the same i don't want to see i want to see that update in home assistance so i went ahead and marked this as false so by adding those three additional filters we set this up so we can use it in home assistant so let's go ahead and before we copy this in head back over to our ssh environment and run docker ps we need to remove the existing docker container before we can re-add it again so let's go ahead and run docker stop and then the id of the docker container which is right here so docker stop the container id this will take a few seconds to stop and then we're going to run docker rm to remove the container with that container id all right so now if we run docker ps we've got no running docker images so let's head back over grab our command and paste that in and hit enter now if we go ahead and run docker ps our docker image is run so to check that let's hop over into our mqtt environment and make sure we're receiving some data now this may take a few minutes for your initial data to be populated because your your meter has to actually transmit during that time period so uh if you see here i've already got my current my gas meter is transmitted it's id has been filtered out so it's going to show up and its consumption is shown right here it like we have defined before it's created a unique topic for that particular id and provided a bunch of information about what it is so what we care about in home assistant is this consumption value as that goes up that's how we know how much gas we've consumed now because i live in the united states my unit is my unit is cubic feet this will come in as important later on when we actually add to home assistant because as of 2021.09 home assistant does not support any other unit of measure for gas other than cubic meters so we'll need to do some hacking to get that to work in the home assistant dashboard all right so now that we've got our docker container set up we're receiving data from our meters let's go ahead and switch over and set up our home assistant now before we do that i want to add one quick note you'll notice my gas my water meter hasn't shown up yet now that's because the water meter is battery operated and it's only going to transmit every hour or so it's defined by the water utility and there's no good way of waking the meter up to transmit so it's not going to provide you real-time usage data but it's nice to still know if you have a spike in usage when you're not expecting it so on mine obviously i know that it's going to be consuming water whenever i see the sprinklers run that's an expected increase now if i would see something like that happen and the sprinklers weren't running then i might want to look for a leak somewhere in the system all right let's go ahead and get it set up and home assistant now we're going to get this set up at home assistant we got to set up a couple of mqtt sensors to pull that data out but first if we switch back over here to my mqtt explorer you'll see i now have two topics in here one for the gas meter one for the water meter so everything's functional let's hop over here into my live environment and i'll show you how i have these sensors configured so if you look up here you'll notice that there's a gas meter reading and this is all that data that's being translated from the mqtt server and i also have a water meter reading which comes in as well one of the tricks like i mentioned is that if you wanted to display this information in the home assistant energy dashboard you cannot do that the data as it comes right out of the mqtt broker because it's in cubic feet instead of cubic meters so you'll see down here i have a virtual sensor set up to convert that information over to cubic meters so real quickly i'll show you how to get these set up and then how to add them to the energy dashboard so if we go into my visual studio code and into my sensors i'll show you the three sensors that i've established for those so here we have our gas meter mqtt sensor so platform is mqtt state topic is metermon dash id number so if we pop back over into our mqtt explorer you can find this by finding the specific topic you want and clicking this copy clipboard and you paste it in there as a state topic give it a title whatever you want it to be value template now the value template is going to be value underscore json consumption so if you look back over here you'll notice there are multiple json values that are available from the sensor from the topic you could pull any of these particular ones that you wanted but what we're looking for right now is json value of consumption so that's why we're using js value underscore formatter for json.consumption that's our value template our unit of measure is cubic feet because it's cubic feet and then the device class is gas now in order to correct the gas measurement into cubic meters so it can be displayed in the home assistant dashboard we need to create a template sensor and again you can grab this template sensor off my blog post so then we have the mqtt water meter very similar except for it's got a different reading for the number except it's got a different value for the state topic water meter reading value template again is value underscore json.consumption and finally the unit measurement is gallon and of course you can find that because it does transmit its unit of measure when it tells you its consumption as well so now that we have both of our mqtt sensors in here we would reboot home assistant and then we get those two sensors available in there so it'd be sensor dot water underscore meter underscore reading and sensor dot gas underscore meter underscore reading now to fix that metric issue and convert it over into the appropriate information i have set up a separate template now this isn't a template sensor this is actually a template so i am not not 100 sure the difference the the thing is you can define more data about the particular sensor in this format than you can using a standard template sensor this is either defined in the template section under your config file so it can be in your config.yaml or a separate templates.yaml so in this template.yaml i am defining a sensor giving me the name gas energymetric and its state is going to be the original gas meter reading sensor but i'm converting it to a float and multiplying it by the conversion factor to make it cubic meters it's going to have a icon of fire and its unit of measure is cubic meters and this is m with the three superscript over it now i set this up intending to add this directly into the energy dashboard but in the current version you can't do that you actually have to hack it and i'll show you how to do that in the last section and then finally the state class is needs to be totally increasing what this defines as is that your sensor is not going to be just giving an instantaneous value so it's not going to be showing the amount you're consuming per minute or per second it's going to give you a reading at a specific time this tells homelessness that it needs to do some statistical work in the back end to be able to tell you how much it's being used per slice so if your say your consumption value is a thousand this minute and a thousand and two the next minute then home assistant will know that you've used two cubic meters or cubic feet of gas and it will show that in the statistics panel so that's why this state class is super important and you cannot define a state class in a template sensor but you can define it in a template defined as a sensor i know it's confusing but it just works then finally the device class is gas so once you get all this done go ahead and save go to configuration server controls and make sure to check your configuration and restart now when it restarts you should have three new sensors that you can re that you can access the original gas meter sensor the original water meter sensor and then the converted gas meter into cubic meters obviously if you have cubic meters coming out of your meter then you don't need to worry about this conversion factor or if this has been superseded in a later version of home assistant you can ignore that as well so now that we're back in to home assistant we've got our sensors established so we can now see the gas meter reading the water meter reading and all the other extended information about it so finally i'll show you how to add this to the energy dashboard so adding to the energy dashboard like i mentioned before is a bit of a hack in 2011.09 it's the first version that they've allowed you to do gas but there seems to be a bug where you cannot add a virtual sensor or a sensor using cubic feet so you go to my energy dashboard you notice that there's no data in here right now it's because i've pulled that sensor out but let me show you real quick how to hack this in so if you go to visual studio code or whatever your your editor of choices and instead of grabbing one of these files here there's actually a hidden directory called dot storage that you'll need to go access so to get to that we go to file open file and it's going to be under your root config dot storage so it's hidden and then there's an energy file in here so click that energy file and it's going to open that up now if you notice right here i have added in this section and again this is going to be in the blog post too this section is added in directly under my electrical grid which i'm going to go to into next week so this is my electrical grid so in yours you more than likely if you haven't added any electrical grid this section will not be present so you can ignore that so just right under the energy sources you need to add in this set of uh formatted values which starts with the curly bracket type is gas stat energy from is the sensor that you've defined in the template that converts to metric so in my case it's gas energy metric the cost is null the intensity energy is null price is null now this number energy price is a calculated what i've done based on my setup here so it cost me 39 cents per cubic meter uh for my gas and i just put it in there as an average right now you can't set on season or off-season values it's just a static value so you can calculate this or you can leave this null if you don't care and save and i'll have my entire configuration file out there as well because that way you can reference it as a as a guide um you want to go ahead and save that and then when you restart home assistant your energy dashboard will start showing up with that particular value so i've got nothing here right now but if i go ahead and restart my home assistant instance the next time it comes up it should show that sensor value at the bottom and then after two hours after it can define the statistics then you should start seeing data across your dashboard all right so now after we've waited for a couple hours we've gone back into the energy dashboard now we see there's some data there's some live data coming in from the gas sensor so now you can integrate this into whatever home energy product you want or if you just want to define this as a graph that you show separate you can do that as well but now that data is in there so now you can use this data to see how much gas and water that you consume or electricity if you if your electrical meter supports it so hopefully that was super helpful i know a lot of people had asked for this specific project i know it was a complicated walkthrough so if you do have questions please feel free to leave them in the comments or or jump onto our discord server and i can try to answer your specific questions there as well so hopefully you found this super helpful uh let me next week we're going to be going into adding in my energy meter into the home assistant instance and then i'll show you kind of a full walk through of the dashboard so on the next video coming out later this week we're going to go ahead and show you how i set up my new emporia energy solution in home assistant and then added that to the energy dashboard i'll take you through some of the features you can do in the energy dashboard and also some of the limitations that i found that i'm trying to correct using some third-party tools thanks again for joining us if you missed the rest of the energy series click here to go to the energy playlist or if you haven't already please make sure you subscribe by clicking our icon here above thanks again have a great rest of your day

Channel: This Smart House

Views: 4,015

Keywords: sdr radio receiver, sdr radio antenna, sdr radio raspberry pi 4, sdr radio setup, sdr radio dongle, sdr dongle setup, how to use rtl sdr dongle, smart home, energy, utility, utility meter, gas meter, water meter, rtl-sdr, metermon, home assistant, energy dashboard, rtl sdr, software defined radio, energy tracking, electrical meter, smart tech, thissmarthouse, this smart house, green tefch, utility meter home assistant

Id: xOgEjd_2hnA

Length: 27min 18sec (1638 seconds)

Published: Mon Oct 04 2021