Make Your Own Innovative Smart-Home Secretary

guys like I saying that this is cool is just I don't think it begins to explain the the Intrigue or how I'll put it this way there's always something this was a message that was etched into my mind as a teenager a kind and funny woman from my church spoke those words to me around our kitchen table one night as we were simply talking about life she said that I probably won't feel the impact of her words then cuz after all I was just a shelter teenager that didn't really understand what it meant to be an adult but then slowly as time pressed forward my responsibilities grew I had to learn how to manage my time with no parental supervision I had to manage the little money that I could script together to pay for gas and food while commuting to college and then my responsibilities grew even more navigating responsibilities as a son brother friend student musician intern boyfriend employee fiance freelancer husband engineer father the responsibilities keep growing and they keep changing and you might wonder like what does this have to do with Smart Homes and I'm getting to that cuz you see the list of things that I need to keep track of at this age of life is a lot and it's probably the same for a lot of you that are watching this I often find myself prioritizing like the important stuff and hoping to remember the more mundane trivial things of life I've tried lists calendar event reminders to-dos apps and to a degree they all work when I remember to look at them but I find that they never really last long as they don't really integrate into my life I find myself having to revolve around those lists and apps instead of those apps and tools revolving and adapting to me ideally I don't want to think about what's next or or manually go through a mundane list I would rather have something more akin to let's say a secretary right someone who can look at all of the mundane things in my house my calendars in my to-do lists and summarize the things that are important and possibly even add to them but I don't know about you right I don't have secretary money however with a bit of creativity and GPT we can build a secretary and it's a lot easier than you think all right okay so I'm going to go over several different levels of this summarization automation starting with level one now level one is the simplest and I'm simply showing you this because it will be like the ground floor for understanding the rest of them all of the information here is going to be hardcoded so you can see quickly what it's anticipating and how we want it to respond so let's start from the top we have here three different parts parts of this Automation and this is going to stay consistent for the rest of them you have the first part which is a trigger the trigger can be whatever you want in this case we're going to use the inject node you have the body of it and that's going to be these four nodes and these are the open AI intent nodes that I created this is what makes it very simple and easy to add GPT into the mix and to execute this type of automation the last part is the output and that's how we give ourselves the information so in our case we're simply going to use a debug node and just print this information out here on the side with that let's get into it we have in the system node which describes to the system how it should operate here I'm basically outlining that it's a personal assistant I'm telling it that for dishwashers just tell me the state when it's dirty I'm also telling it that for any event that is within 24 hours of the current time I want to know about it otherwise if it is outside of that 24-hour time or if it had passed don't mention anything about it and then I think and then the last major part of this is I gave it a list of all of the events and the states that I want it to be aware of in our case it's aware of the dishwasher and that its current state is dirty and it's aware of these three events I'm also passing at the time and I've set the time to January 15th 10 p.m. it's not January 15th it's some other time but uh that's what it's currently set at right now I also want you to notice that for the events two of them has passed already this is on the 15th from 4:00 to 5: this is on the 15th from 10:00 a.m. to 11:00 a.m. both of those according to our current time has passed the only one that's applicable is this Physical Therapy one that's happening the following day at 1:30 p.m. so when we run this we expected to talk about the dishwasher because it's in a dirty State and we also expect it to talk about the physical therapy event let's see what happens here we go so here in the payload we can see that it has that we have one event coming up within the next 24 hour and that's the physical therapy one also it's talking about the dishwasher and that it's sturdy and that it needs to be attended to this is perfect this is level one this was hard coded but you now see the gist of what needs to happen so now let's get on to level two so you can see what it looks like dynamically within level two we have three new nodes that you should look at the first one is this template node specifically it's called the render template node this node is special and it's pretty cool especially when using it with GPT because it allows you to essentially use yaml in order to find information the same way that you can look at the developer tools within home assistant and under the template section you can play around with the templates and kind of query for data this does exactly the same thing except all of that information that you use within that template node and the developer tools gets passed along within the automation as text that GPT can read you should see where this is going so for this one we're using the Now function which prints out the date dynamically so we can see the actual date and time and then we also see here the dishwasher we can grab the actual state of it and the only peculiar thing we have here is this events you'll see what I'm going to do with this shortly so this particular node is one that connects to my calendar and sends me back information from that calendar that I set up uh inside when you what you can see here is there's this config uh I don't need to really get into that right now because you can set it up like you can follow the instructions for it to set it up but it allows you to essentially run this and grab any c information that you need based off of these criterias the only thing here that is of note is that I'm telling it to grab all of the events that are within 16 hours of this current time if you want to know more about this node if you go to manage pallet it's called the IAL events this doesn't come by default within node red I'm only using this because this came before home assistant added calendars into its primary uh way of working um now that there's calendars I'm going to show you how that works but I'm going to show you how this particular node work just in case you don't find the native version robust enough at least you have this option the last node that I want to take note of is this new function node in it I'm looping through all of the calendar events so it's going to come once the calendar fetches the information it's going to come through this payload I'm mapping through the payload and I'm simply grabbing the summary which is basically the the description of that event as well as the date and this is going to have both the beginning and end date I think that's how it it uh creates it um and we're simply taking looking for that events within that yaml and just to point that out that yaml value is here right so it's taking all of this it's finding the current date it's finding the state of the dishwasher and it's printing this out and it's pushing it into this part um is pushing it into this property called a yaml and then we're grabbing that yaml and we're replacing that events with the events that we got from the calendar here that's why we've done it this way we're simply substituting it out since we can't get it from the actual yaml within home assistant if you remember in the system node in the previous one we've hardcoded this information because we now have this stuff within that yaml property we can dynamically add that to the system by simply referencing it here in single curly braces now when this particular node runs it's going to replace the ammo with all of that information so that way we can add in Dynamic information which is pretty cool I've altered the date time on the computer so that way it can give me a different time so that way the stuff in the calendar can show up appropriately uh but here's what it's going to look like so here it's saying that we have uh some Pest solution people coming through we have physical therapy that's happening and then we also have uh swimming that's happening as well and then music lessons and then also the dishwasher is dirty all of that is correct again I had to update the date time on the computer so that way it will show all of this information but um all of this is correct and this is being dynamically fetched from my calendar as I mentioned a little earlier 2.5 uses the actual in calendar integration within home assistant in order to do the same thing so if you don't want to use this particular node you can just swap that out right here with this call service node and reference the calendar so if we take a quick look right we're using in the domain called calendar we are going to use a service called list events uh don't use list events use get events instead so I have here the entity that's pointing to the calendar that I want to reference it's blurred out for obvious reasons and then here is something interesting the duration the duration is technically referencing the hours minutes seconds away from the current start time now if you don't specify start time it says here or in the documentations that it's going to use the current time so whatever this current time is it's just going to start there if nothing is provided and then this is referencing or saying that from the current time I want everything that's 24 hours from now realistically when we run this we should get exactly the same response or something similar okay so here it's saying that we have the pest solute the pest people music meeting for four and uh and the dishwasher so the one thing it didn't do is it did not omit um it did not add in the physical therapy the time for that one has probably elapsed in terms of its end time which is why it probably didn't show that up and we can kind of verify that stuff down here too as well if we look at the ammo something that I did want to point out is that because we're using it natively we no longer need to go and use the map that we saw before when we looked at the previous one right this merge events had this particular map that went through each of the the events and kind of reduce it down to something smaller uh we don't need to do that for this particular one because the calendar node is already condensed already so all of the information it passes is useful to us so I would say as you're going through this keep that in mind that the more information that you send to GPT one the more it has to do and the more likely it could get something wrong but um on the same side right you're going to be sending a lot more tokens which will burn through the amount that you would have to pay at the end of the month um so just kind of keep that in mind when we look at the render template node remember this is the node that allows us to write yaml we have additionally two Loops the first Loop is to Simply look for everything that has the name battery within its entity name and we're printing out the state of those batteries so the battery level and then we also have one that's looping through all of the lights and we're looking for lights that have the state of unavailable and we're printing all of those lights out with the addition of these two we now know the state of the dishwasher we're now going to substitute all of the events based off of what we're going to get back from the call service node we also are going to know the state of all of the batteries and we're going to know the state of all of the lights or at least we're going to know which lights are unavailable merging it is still the same we're only substituting out the events now that we've added more information we're going to have to get more specific with what we GPT so in the system node again same stuff from before same thing about the dishwasher same thing about the events but now we have here we're only asking for information about batteries that are 10% and lower and we're also asking for a list of all of the unavailable devices let's see what it says okay again it talked about the dishwasher it talks about the music meeting that's supposed to be happening uh at 400 p.m. it says that no batteries are below 10% and the following are unavailable let's move on level four as you know home assistant has added support for calendars as well as now to-dos specifically within the to-dos we're able to now query for that information so we're going to do that now with this particular call service node this one we're adding on top we're looking at the Domain to do we're going to use the get items this is a voicemail to-do that I've set up you'll see what that's about in another video further down we have here in the data status needs action which is simply telling it to look at all of the to-do items that haven't been addressed yet one important call out that I want to make for both the calendar call service and this to-do call service is that that output needs to be there by default you're not going to see anything there you're going to have to make sure that you set a property to this results that way once it resolves it will put that information within that property that you've set without it it's just going to come up blank and you're going to be very confused so we're going to be outputting the results into this particular property called to-dos and similar to the events take a look at this template we now have this to-dos right here we're going to do the same thing that we did with the events with to-dos where we're going to substitute suited out with the information that we got from the call service node everything else is the same so when we look at the merge events the events are being replaced as well as the to-dos are being replaced if this looks like a lot of code don't worry I'm going to make all of these flows available to you within shaone you can just simply copy and paste the Json and import it into your system so you can see the four different levels that I have here granted you're still going to have to update them because they're going to be pointing to things that you don't have but at least you can see the structure of it now notice that I said what should I know about I'm it sounds very generic and and plain but something that I ran across the other night was when I was testing this out when I told it to summarize here when I told it to summarize the following events in a simple to easy understand way it simply it prioritized the events and kind of ignored a lot of the other stuff so remember this particular open AI system system has a lot of information in it in terms of how it should respond and how it should interpret the information additionally I've added in here instructions for what to do with the to-dos for the to-dos I want them to mention any to-do that is due today yesterday or tomorrow and any to-do that is older than one day or older than yesterday to just simply State the number of to-dos that are that old so if I have a to-do that's let's say due today it's going to tell me about that specific to-do however if that to-do was supposed to be done let's say five days ago four days ago three days ago then it's not going to tell me about that specific to do it should simply say that hey you have one overdue to do or something along those lines now because there's a lot of instructions inside here if I get super specific with the instructions with inside the user node then it kind of negates the stuff here like it still uses information within the system node but it it kind of hones in on what the user is asking for so I this is like the balancing act that you're going to have to do with GPT this is not anything that has to do with AI intent but more so just the way llms work you have to kind of do this prompt Engineering in order to get what you're looking for additionally because you have to do this type of prompt engineering it also means that testing this is kind of hard and it also means that it's much easier to get flaky inconsistent results just due to the nature of what it is so let's see what it says so we're one expecting it to talk about the dishwasher being dirty we're going to expect it to talk about the events that's du for today I think at this point there's only one it's going to talk about the to-dos and just simply mention that there's a to-do pending this to-do is I think four or 5 days old according to the current date time that I've set this to and then it may talk about the battery States as well because we have that in there um and is there anything else I think those are the oh and not just the battery stage but it's also it should mention about anything that's unavailable so like any lights that are unavailable otherwise it may not say anything about it but I don't think there's any unavailable lights so let's just see what it does all right so talked about the dishwasher that is correct you have an event music starting today at 4 p.m. that is correct there's one to-do whose date is more than one day old perfect and then it's talk telling me about devices that are unavailable and these are all the unavailable things and remember I told it that for anything that's unavailable it's supposed to tell me about it or List it out if they aren't unavailable it shouldn't say anything and same thing for the battery States and I think all of this is good all of this is good for what it is but I I want to show you more I want to take this to another level I want you to get a good understanding as to how AI intent can enable you to do so much more this right here is the good night routine I also have a good morning routine and they're essentially the same but they do slightly different things where they differ the most is in their triggering and in their output and if you remember what I told you we have three different parts we have the trigger we have the body which does the work and then we have the output for the trigger for this the good night one is a little bit more involved we have here this bedtime State it's watching a helper that is a schedule Helper and I've set in that schedule helper all all the days for each day which is what time we need to go to bed and what time we wake up and based off of that time as that time elapse this will trigger off and on accordingly essentially we're looking at that particular uh State and we're waiting for it to trigger on so whenever that bedtime state turns on and then this whole entire automation will fire off we also have a manual trigger button here where I can press it on a dashboard and it starts this process as well just in case for some reason this fails um it's always nice to have a backup this is looking at the house and making sure that it's in the appropriate State and I listed the states here it needs to be in the protected warning or active State and typically it will be in the active state but this is to again to cover our bases and then if it's in one of those States what this is essentially signifying is that we had already ran this automation so the house is already in sleep mode if it is then we do nothing it stops right here however if it's not in sleep mode it will be in one of those other three states so we have to put the house in sleep mode so the automation can move forward from here it splits off into two and it does two things firstly it'll go start cleaning so it'll start the vacuum as you see here it'll turn off the lights um disable motion sensors so that way the house doesn't become super active at night we want to turn off a lot of the stuff the other part of this automation is the part that we've been working with from the beginning and that is giving a summary of everything that we need to know so this should look very familiar where we have the caler the render nodes or the render template and then all of the GPT or open AI uh nodes this particular node will basically send those events to wherever I am and I'll just read those responses out I'm not going to run this because it can get really involved turning off everything in the house when my wife is using everything at the moment but um just when I say it works it works pretty well now this is the part where I feel like this can get kind of um awe inspiring if you notice here we have the start vacuum if you've seen my previous video on the call and register intent nodes I did make mention of this particular node called the Linkin linkout nodes and it basically allows you to jump from one automation to another and the register intent and call intent nodes essentially does the same thing where we can call one thing and it'll jump to another Automation and start that instead we're doing that here and this makes sense here because this can start on its own so if let's say we never went to bed and we kept the house going then we have it so that way at a at 1:00 a.m. this will fire no matter what and it'll start the house or it'll start the robot vacuum to start cleaning at 1:00 a.m. now notice we have one automation that was meant for starting the whole good night sequence but then we have a separate automation that can technically live by itself like we want that to be its own thing we don't want it to be completely attached or completely dependent upon any of the other automations so the Linkin linkout nodes do a good job of bridging those two things and kind of keeping them decoupled but allowing you to utilize it I'm going to take this concept and apply it further to our summarization automation realistically our summarization automation can be used outside of any of these uh particular use cases I don't need it when I'm just going to bed or when I'm just waking up I could ask for it for in the middle of the day so I shouldn't have to be tied down to this particular trigger hence why these things are kind of built into pieces the trigger doesn't matter the trigger can be anything to be honest I can say hey what do I have on my calendar for today and it should go through this particular Automation and tell me these things so let's do that using the AI intent nodes that we have set up instead of running this particular one I'm going to show you on the one that we've been looking at from the beginning and that's this one what we can do is we can add here I have a register intent node and here I name it summarize my tasks and then I give it this particular description creates a summary of all the task event to-dos and states of all the devices in the smart home between the name and the description open AI should be able to look at it and know when it should be called it's good to have the stuff named pretty well and described pretty well so that way GPT has a good chance of using it when it's supposed to be used I'm going to make sure that it's included so that way open AI can see it I also have here a change node and I'm simply removing the payload data and template just in case because this can get called from anywhere we don't know what's going to be passed and what I do know is if any of these three are passed into this particular automation it can mess things up because all of these can be dynamically set and it looks at those particular properties to dynamically set those properties so it's best to just make sure that they're deleted before it comes in so that way it Ed what we have set here I'm going to attach it to the call service node that starts the to-dos I'm going to deploy it something that I'm not showing you behind the scenes is that for k k is set up so that way it does one of four things I'll actually show you I'll actually show you so here's k k is basically listening for any message that's coming from telegram because that's how I primarily communicate with K so whenever a message comes in from K it's going to immediately try to use use that message and Trigger the call service node or it's going to do the call service node if once the call service node resolves it's going to resolve in one of three ways from what I found out there could be more but for now I've just found these three and these three seems to work well one it can be action done which means that it was successful in that case nothing more needs to be done so it just simply sends out an answer the other part is that it can say query answer which means I have ask it a question that it could answer and so its job is done my job is done and it just stops the last one is if there's some kind of problem and that is an error if I give the assist because it's calling assist if assist is unable to help me or if there's some kind of error then that particular state is sent to me and from there I will now activate GPT I went through these scenarios with K versus assist home assistant assist versus my AI chap bot Nam K let's put them against each other and see who comes out on [Music] top you can see how both assist and K competes against each other um but essentially this is how I can get the best of both worlds I start with assist if assist isn't able to do it then GPT picks up the slack and for GPT we have these four outputs One commands if I send it a command it's going to try and call it dynamically using the call service node I have here questions if I simply ask it a question it's going to know that it's not a command and it's going to simply answer the question and post it back to me the other part is registered intense if I give it a command and it comes back as hey I'm using or I want to call a register intent then it needs to call this particular call register um call intent node by calling that I can call any of my register intents this is how we can make make all of these connections come together so with that let's do a quick test I'm going to tell K hey what's on the agenda for today I expect it to interpret what I'm saying to call that particular register intent node and by doing so trigger all of the automations that you've seen or that we've worked on beforehand and give me the output that I'm expecting let's watch let's watch the debug section and see what happens okay this is to be expected I'm expecting one more oh there it goes 154 and 154 is this one which means that it ran all the way through it made it all the way to here so if we take a quick look we're looking for payload object and here we go this is the same information that we've been seeing all the way up until this point in terms of what we have in our calendar what we have in terms of like the battery levels uh the fact that for let's see even does it talk about the the to-dos there's even one to-do whose date is more than one day old unavailable devices it it it worked it worked we were able to not only create a summarization automation we not only saw how it works in tandem with like complex triggers and outputs but we can see that by being able to decouple these things and have these sections kind of partitioned off especially if you know that they can be used by themselves how you can use the register intent and call intent to combine with GPT so GPT can now trigger these things on the Fly for you you don't need any manual trigger you could use a manual trigger and it's best that you do have one because these systems need it but here's how you can start to make your system more autonomous you can just say things or tell it things and it can simply do it and it does it well guys like I saying that this is cool is just I don't think it begins to explain the the Intrigue or how I'll put it this way every once in a while technology changes to a point where it introduces something so brand new and so novel that it makes it creates a new Wild West no one knows how to use it no one knows what to expect from it no one knows really what expectations to have on this and then as a result not many people like like it's a fat it's a hype and people kind of jump at it for a little bit but a lot of the people who aren't Enthusiast they fall off really quick like it's just a novel thing and they just kind of fall away but enthusiasts like me and you like we're in it and we're here basically the gold miners searching for this gold we're searching for the way that that all of this should be utilized and we're pioneering the expectations and the experience that will one day be the standard for everyone to use people will come back and look at this and use this as a way to learn from our mistakes to learn from our successes and create a smart hope ecosystem in which the masses that once were in this that but fell off will come back to use and use with more uh with use that they will use with more finality use with more assurity and it becomes a deao standard within life we're we're here this is it this is it we are exploring we in this together let me know what you think let me know what you feel about this by please please please go watch my other videos watch them I have so many ideas and they just keep pouring and and it's it's it feels I feel manic almost I feel manic because they just keep coming I talk to you guys and then more ideas come and I can't make them fast enough I can't do it fast enough I'll just keep going along watch my videos don't watch my videos it's fine I'm here even if no one else wants to be this is fun as hell so come along or don't we go places bye oh God [Music]

Channel: Technithusiast

Views: 7,127

Keywords: montaque, smart home, home automation, node red, home assistant, Smart-Home, Automation, DIY, Tutorial, Innovative Ideas, Home Assistant, Smart Technology, Home Automation, Productivity Tips, Task Management, Calendar Integration, Voice Assistant, IoT, Gadgets, Tech Guide, Smart Living, Home Innovation, Personal Assistant, Maker Projects, Smart Gadgets

Id: H38JbIAwQaE

Length: 33min 5sec (1985 seconds)

Published: Sun Feb 25 2024