Home power monitoring using the Shelly EM with Grafana, InfluxDB, Node-RED and MQTT

good merry all christmas covered youtubers it's matt here with news of my new all singing all dancing house power consumption display so a quick point at the stage of the video i'm not going to cover the installation of node red influx db grafana or mqtt there's loads of youtube vids out there covering these topics so you can do all this stuff on a single raspberry pi um and if this stuff kind of scares you off then i would say don't be scared just jump straight in check out burns h a youtube channel where which i quite like because he cuts out the annoying stuff and gets straight to the point with a screen record of his installation procedures for most of the programs we're using here quick note influx is now in version two i'm still running version 1.8.3 i find the upgrade procedure it looks really annoying so i can't really be bothered to upgrade for now another quick note before we start i'm going to be careful in this video about my use of the words power and energy a lot of people unknowingly use these words to mean the same thing but as engineers we know they have different meanings and you have to remember to not use them interchangeably sorry in advance if i slip up here i'm going to explain a little bit more about how they are different in detail later in this video for the uninitiated anyway here you can see two channels of power monitoring one on the left and one on the right both showing the same breakdown of data i have a shelley em which stands for electricity monitor or meter i'm not sure which with two current clamps uh 150 amp and one 120 amp um i bought two different ones just so i could compare and contrast and honestly i don't know why you would go with the 50 amp current clamp unless you wanted something that was smaller physically to fit in a box for example um anyway so i've got two current clamps and we have two different power supplies that i wanted to monitor so i'll go through one of the sides and you can follow um the rationale behind why i've got these graphs and readings top left we have a gauge which shows current power usage in watts this displays the very latest reading which is updated every 30 seconds from the shelley gets sent from the shelley by mqtt and i use node red to basically insert this into our influx db database um this also gives me the opportunity to do other things with the data in node-red such as showing it on the node-red dashboard or storing it in node-red memory if i like um so here's an example is my device monitoring system which i've set up in node-red which aggregates all my tasmota sonoffs shelley's my own custom arduinos this device monitoring system only uses mqtt to collect all the data shown here and it can inform me when one of the endpoints goes down or comes up and i can see its status history to an extent i can see things like you know wi-fi signal firmware version even with the sonoff shows the hardware that it's installed on this little delete button here allows me to remove it from my monitoring system but um if it receives an mqtt message on the path that it's listening for um then it will just automatically re-add itself back which is how i want it anyway back to grafana to the right of that gauge we've got the green line graph of instantaneous power which is basically the historical data from that gauge for those of you don't know grafana we can change the time frame for any dashboard using this pull down in the top right or we can click and drag on the graph to zoom in on the selected period and there's also a little zoom out button as well and we can enter the time period manually if we choose um so now working down from the green line chart to that blue bar graph you will see the data seems to take a very similar shape to the green chart above but it is subtly different you can see the spikes line up um and the shape is pretty similar in fact the data uh comes from a different measurement within the shelley em all together um and it tells us something different to the to what the green chart tells us basically there is nothing clever about the green line chart at all every 30 seconds the shelley sends its current power usage i should say it's instantaneous power usage in watts and i'll just quickly switch to node-red to show you the thing that it's listening for we've got this little function here which checks um because each message sorry each reading from the shelley comes through as a separate message so i'm using this function and this join here to put them all into one message uh so that the output uh you know looks like this all of the measurements in one payload rather than separate messages um so that's what this function um does it just checks to make sure that we're listening because obviously um the shelley well not obviously but the shelley em also sends information about the relay state uh i don't i'm not using the relay and i'm not interested in it so this function will you know basically only return uh the message if uh you know the fifth element number it says four here that's um zero one two three and four if the fifth element in the array that's created by this split operation um iev path um is you know not undefined if it exists anyway you can have a look at that in more detail um i'll just whiz through this now so it kind of combines or joins the multiple messages into one message and that's what we get and then it sends it through to um through to the influx db output node um and then let's just quickly show you the influx database and this is what it looks like and you can see at the top these are the readings and so i've got zero energy zero power zero reactive power zero returned energy zero total and zero total return and zero voltage and the reason it's zero underscore that comes from this um node-red function that's me doing the zero underscore and the zero obviously refers to the first current clamp and the one refers to the second current clamp anyway back to griffana this green chart simply plots that data for any time window you select on the dashboard so you can imagine that if the wi-fi dropped out or you rebooted your server for the duration of that drop out there will be no data coming into the database therefore nothing to show in grafana and because it's a line graph it will just draw a straight line between the previous point and the next point in time be it one minute or ten days in fact in grafana you can specify whether you want it to draw a line um or whether you want it to yeah so you can interpolate between the previous and the next value and so it would be a you know a line that isn't horizontal or you can have it display nothing um so no no line at all that'll be a null value um or you can have it just kind of continue on the previous value for the duration where you had no readings so imagine if that were actually 10 days drop out and in the meantime someone taps into your power supply to boot up their nuclear power station a few doors down and they conveniently shut it down again just before your readings come back then you'd have absolutely no idea on this green graph that you've been bankrupted because it just wouldn't show that spike um on the micro level this can be happening between each 30 second reading and there can be power fluctuations you know and although the green graph shows accurately what the power usage is on those 30 second intervals it doesn't help you accurately work out what it might have been between the intervals now don't forget that you pay for electricity in units and one unit is a kilowatt hour don't forget as i mentioned before the power is not the same as the energy that would be like saying distance is the same as speed they are not energy is like the stuff you're pulling out of the power lines and power is basically how hard you're pulling it out or rather how quickly you're pulling it out so the line on that green graph shows how hard you're pulling the power out and the area under the green graph shows the total amount you pulled out for a given period of time um you know i don't if you remember maths at school but that's how graphs work um and the amount that you pulled out for a given period of time is of course what the energy company uses to work out how much to charge you for a given month of usage problem is the area under that green graph is never going to be an accurate measurement of the amount of energy used and the same way that if you looked at your speedo every five minutes when you're driving through a town center with roundabouts and traffic lights where you're stopping and starting this speedo measurement at regular intervals would not be good enough to actually accurately deduce how far you've driven so instantaneous power is great but what the shelley em also gives you is a thing called total power which is the equivalent of your myelometer or the counter built into your electricity meter itself and this is what you need to know to work out exactly how much money you spent on energy yesterday or over the last six hours or last week or whatever so this explains the difference between the blue and the green graph the blue graph uses the total energy measurement from the shelley which is like the myelometer and the shelly stores this in non-volatile memory like your car does so it persists across reboots in the same way that your myelometer you know doesn't just reset itself the next time you get in it so in order to show this graph we need to find the difference between the first reading in the time period and the last reading much like you would look at the difference on your car's mileage at the start and end of the journey when you have a period of no readings because of a wireless or server outage the blue graph will show zero for that period because the difference data is grouped by hour but the shelley em will still be counting energy so the next reading will jump up so hopefully you can understand why you'd actually want to see both the green and the blue graph because they tell you subtly different things let's look at an example of when i did turn the database server off for an hour or two this is on the 27th of december at around seven o'clock in fact um see on the right blue bar chart see if you can see um when that happened and it's really easy to spot it on the on the right hand one because the energy usage has been very very very consistent as you can see um over the last six seven days um have you spotted it anyway let's zoom in on this one here the green graph shows a nice smooth line when there are no readings and the blue graph shows this weird zero reading then um a double reading for the next hour because the thing came out just before the end of the following hour so it's you know kind of kicked it back in to that reading now the green graph should probably technically show zero values for when there are no readings and we can actually change this in graffana and you would do this by um you know this is how you do it you can either show the previous value when there's no values or you can show zero values or you can show null or if you like you can um kind of like interpolate between the previous value or it's not really interpolating is it um but you can go between the previous value and draw a straight line up to the next value moving on down to the purple graph it does the same as the blue graph in that it uses total power except the data is grouped by 24 hour period instead of hourly so this gives you the actual amount of energy used per day which is obviously pretty handy you could add more graphs for different time periods if you wanted but um don't forget you've got the time period selector in the top right and i only want a breakdown by day or by hour can't see of any reason to have any any other time periods for breakdowns of data than those now under that top left gauge you will see total for period in red again this uses the total power readings from the shelley not the instantaneous power and it works out the difference between the first and the last reading in the selected time window for the dashboard the number is basically the accumulated value for all the blue bars and that's because the blue bars are kilowatt hours by hour so each blue bar is for one hour of stuff and the height of the blue bar refers to how many kilowatts it was so yeah if you were to take all those blue bars and add them up end to end that is what that number would equal underneath we've got the orange representation i have simply set the co that's a cost representation i've simply set the cost to 14p per kilowatt hour pence that is um uk pennies um which is not actually correct because i'm pretty sure our power costs more our energy i should say cost more than 14 pence per kilowatt hour um uh but it's not far off i think might be 16 or 17p um and i can't be able to look it up but it's near enough and it can be changed later and so for you if you wanted to do this you need to look at your electricity bill to determine the cost per unit or cost per kilowatt hour and enter that in to this transform and here's how it does it um so i am just going to go into here and i'll show you the transform so what this transform does is it takes your query let's just look at that um this is a query of zero total that's the field we're selecting and now if we just scooch over to shelley em mqtt docs you'll see that total which is this one here um is the total energy in what hours so it's not actually it doesn't come in in kilowatt hours it comes in in watt hours that's the counter accumulated in devices non-volatile memory so that's in what hours so uh we need to convert this to kilowatt hours now i didn't actually have to do that with um this thing because i just gave it to it in watt hours and i've set the field type as um what hours and it just automatically displays it in kilowatt hours if the number is big enough um however for this one um i'm actually doing an operation on on that so it comes through in watt hours and my operation converts it into kilowatt hours by you know multiplying by 0.001 and then obviously that last um one four is there because it's 14 pence per kilowatt hour so that's point zero zero zero one four to give you the number of pounds um for the time period so that is what that is back to the dashboard so under that we have averages for hourly and daily use again that's calculated from total power and you need to be careful with these readings because if you look at the average daily energy whilst you have a time period selected that is less than a day then this figure is not going to be correct it's going to be nonsense and the same goes for average hourly use you've got to select at least a couple of hours time window for this number to be meaningful right so that's all the stuff and same here on the right hand side for my other current clamp sensor finally underneath we show the latest voltage reading from the sensor and the graph on the right shows the historical readings um now this graph looks like it's only one reading and not two lines but that's because the lines are exactly the same because the readings are exactly the same the voltage and we're on a i think it's the same phase of a three-phase supply and the voltage is bang the same for both of those um and it's also got a handy max min average and current reading and legend on the right again current here does not mean you know electrical current amperes it means the now reading i.e you know the like the current reading right now not electrical current um you'll notice i used um pergola power now or pergola what's now rather than calling it current energy as that would be confusing so i think that probably sums up the whole thing and yeah i've waffled quite a lot about um the graphs um but hopefully it's given a bit of insight into you know how you would look at energy usage when you actually come to exploring these visualizations you can start to see trends and patterns say for example here um we've got on my pergola power supply we've got a hot tub in there and the hot tub um has got a big old kick of a couple of kilowatts on and off every couple of hours but superimposed onto that we've got a an internal electric like fire electric heater um which um obviously has got a much shorter cycle so you can see it um you can see the hot tub without the electric heater inside there's two separate things i'm talking about you see the hot tub here without the electric heater on and the hot tub with the electric heater on and you see these kind of like spikes on top of the hot tub which is quite cool so you can see um so this is what people talk about profiling power usage and how and the electric companies that the supply your electricity supplier can sometimes guess what appliances you have and you know when you're in and when you're using them based on your usage and you know by the by looking at the shapes of these things you can really take an educated guess about what it is that you're actually doing in your home which is pretty scary that's why i'm personally a bit skeptical about the electrical power supplier um or you know your energy provider um installing one of those smart meters because essentially they're reporting much more detailed information they're not just reporting the counter like your myelometer is and that's the equivalent of your myelometer they're not just reporting that they're reporting all this stuff so they get to see all this stuff if you get a smart meter installed which is despite being a home automation nut i don't want that smart meter installed i'd rather have my own um you know monitoring set up like this if you've got any questions or anything like that um just fire them off at me in the youtube comments and i'll do my best to answer um but i hope you enjoyed watching this video thanks for watching

Channel: Mat Smith

Views: 15,197

Keywords:

Id: XevRGFjlIbU

Length: 18min 44sec (1124 seconds)

Published: Wed Dec 30 2020