The ULTIMATE Person AND Face Recognition NVR

hey guys and welcome back to another tech tutorial tuesday recently as in like two videos ago i made a video or a tutorial on how to use frigate in home assistant and frigate is essentially a dedicated or a total nvr solution that is built right into home assistant and has a lot of really nice features including object and person detection and i said in that video that i was going to be uh using frigate in my own um personal home assistant setup going forward and but the one thing the one downside of using frigate is that there's no face recognition built in at least not yet and we've done videos in the past on doing face recognition with deep stack and so i got to thinking what about if we could try and combine both of these pieces of software and use both of them and get the benefits of both of them simultaneously so get the object and the person detection from frigate as well as the full nvr being able to play back clips save snapshots etc etc and then also combine that with deep stack so that i could have face recognition and so i sat out about doing that and that is what we're going to be looking at today i'll show you the automations and all of the config required to make that work but first i'm going to tell you about today's sponsor skillshare skillshare is an online learning community with thousands of classes for creators who want to learn a new skill or even brush up on some existing ones all of their classes are specifically tailored for learning meaning there is no ads and there are always launching new premium classes to their existing library you can choose from a wide range of categories and with skillshare premium you get unlimited access so that you can learn at your own pace i've just completed a skillshare original class with the one and only marquez brownlee called youtube success script shoot and edit with mkbhd which is helping me to develop my video making skills to make better content but they also have classes about arduino raspberry pi 3d printing internet of things and even networking skillshare is less than 10 a month with an annual subscription and the first 1 000 of you to click the link in the video description will get a free trial of premium membership so you can explore your creativity thank you again to skillshare for sponsoring this video okay so let's talk a little bit about how this setup actually works and it's not um overly complicated essentially we have frigates running continuously as our nvr it's doing its thing it's saving snapshots uh saving clips and it's running our person and object detection and when frigate detects a person in the scene it's then in turn going to trigger deep stack to run its face recognition and so that it can try and work out who the face of the person in the scene is deep sac is now going to run it's going to return a match hopefully and if it does it's also going to return the image with that match and we can then save that image and in turn attach it to a notification using an automation and send that to the the mobile phone of the person so what we're going to do is we're going to attach that image straight to a notification straight to your phone and pick up the notification that way before we jump in this is definitely more of an intermediate sort of advanced level video and there are a few things you're going to need first things you're going to need frigate to be up and running and installed with a configuration and configured the way you want it i've done a dedicated an entire dedicated video on that you can find that up here so you're going to need to have that up and working first we're not going to be doing any of the installation or configuration in this video the next thing you're going to need is to have deep stack also up and running particularly the face recognition portion of that video again i've done a dedicated entire video on how to get that up and working and you'll find that up here also we are going to be doing a little bit of tweaking of the configuration but make sure you have it up and installed and working first finally you're going to need what you're going to need you're going to need some very good hardware in order to run this running either of these solutions by itself is demanding enough as it is so you can imagine that running both of them at the same time is also going to be pretty taxing so have some good hardware and remember you can use the google coral stick if you have one of those you can use that to accelerate uh frigate and that is really going to help offload it and so you only really need to worry about the deep stack portion of it um but remember both of these services can be run on separate devices so they don't all need to be running on the same box you can access them from different ip addresses different different hardware so you don't need to have them all running on the same the same platform and you can also have them separate to your home assistant instance so just something to be aware of but you're gonna need some decent hardware in order to run this with that said let's jump in and we will get the basic and deep stat configuration sorted all right so let's very briefly go through the configuration that i've got for uh deep stack and also forget so if we head into the code and then we'll take a look at the frigate config first and jump into our frigate file so you can see i've got my mqtt details at the top um i've set up my camera again we're using the raspberry pi camera that's over on my left hand side i've set the width height fps all that exact same stuff that we did in the dedicated video um i'm also tracking person so i'm doing person detection only i've got my snapshots uh running so there are saving and with the bounding box and i've got them set to retain two and i've also got the clips enabled so that is my very basic um frigate configuration it's pretty much identical to the the previous dedicated video we did on that so if you want to see exactly what all of these options are make sure to go through that video if we jump back into my deep stat configuration which is this section here the image processing and this image processing block if we run through that i have um the deepstack face platform so i'm using face rather than object here and obviously we're doing object detection in frigate um it's also it's running on the exact same machine we have the port timeout and the detect only that is important um to set that to false rather than true if you have detect only on that is going to just re it's going to look for faces in the image but it's not actually going to attempt to recognize them so it's important to set that to false so that it does actual face recognition um the next thing all these other options are pretty simple except down at the bottom you'll notice we have um a camera we haven't yet set the camera entity so the source that um deep stack is going to use in order to run the face detection if we jump into the configuration so we're going to come back here in a second so if we jump into configuration integrations and then we look at the entities available inside frigate you'll see that we get obviously quite a few entities we covered these in the previous video but you'll notice things like we have the pi cam so if i click on there and you'll see that that is my stream now running it's a little bit delayed um on this screen but for the proper version it does work well so there's my rtsp stream running um and we also have other things like we have this pie cam person that is running so that's active because i'm currently in the frame if we open this one you'll see it's detecting me in the image right there so that is looking good we also have this pi cam person which changes as people um enter into the frame and we have this pie cam person motion so it's currently that pi cam person motion means that it's picking up at least one person inside the frame and that will change to off when we when there is no person in the frame so um that is going to become useful later on in the automation you'll see if i cover up the camera frame just now so you see there if i cover the frame or move it out of frame that the person changes to um to zero and the the person motion becomes clear so and then if i remove my hand from the frame so that's detecting me again you'll see that it becomes detected and one object so those entities are going to become very useful and when we come to the automation but let's jump back into the configuration a second so the source that we want to use for our entity id is going to be camera and pycam you could also use the pi cam person but i'm going to use the full rtsp stream which is pi cam so there we go if we enter pi cam and that is going to be the exact name that you've set in the frigate configuration so we jump back into the frigate that is the that corresponds to this name up here and i also want to add one more thing into the deep stack configuration and that is going to be to save the images to a folder so we're going to you remember with the frigate video and the all of the clips are available to browse in the media browser so i can go into the media browser and view all of the clips so for example if we jump down into the um to the last one you'll see there when i just covered my hand over the the camera um you'll see that it it saves the eclipse onto the recording so we want to do the same with deep stack we want to save the per or the face or the person that that it detects as um as images in the media browser so that we can view them if we want to so back in the configuration we're going to add just one more line into the config we're going to want to add save file folder folder and what we're going to do is we're going to save that again into the media browser folder so it's going to be slash media we're going to create a new folder for deep slack images you could save them inside the frigate folder if you wanted to but we're going to create a new folder and it's up to you which one you choose so deep stack and then we'll call this snapshots and that looks good so remember to set your camera to the frigate camera and also set your save file folder this if we go and try and restart home assistant now it's going to complain that that folder doesn't exist because it doesn't so let's create that folder first if we jump over into terminal and then i just simply need to make that folder so i'm going to do mk dir for make directory and we enter dash p so the dash p is going to create any folders so if you have it's basically going to create the entire folder structure for you so if we do slash media and then it was deep stack and then snapshots okay and now if we so that is the the snapshots folder created if we jump back into media browser into local media and you'll now see we have this deepstat folder here and we have snapshots inside of it obviously we don't have any files in there yet because we haven't run it so let's go ahead and restart um home assistant for those changes to take effect with home assistant restarted we're then going to want to and just test out deep stack make sure it's working properly so we're going to manually trigger it for now so if you head over into dev tools and then services and then search for the image processing service and remember you're going to have to have taught deep stack all of your faces or the faces you want to recognize you're going to have to have went through the um the face learning process for deep stack and make sure to follow that that previous video we did um we covered how to do learn all of your faces into deep stack uh we covered that in great detail so make sure to follow that first so that you have all of your faces already registered i've already went through that process so i'm going to select the image processing.scan service select my entity which there should only be one and then hit the call service button and now um the camera should have recognized my face if i was in frame and we can check that by going back to states and then search for image processing and you'll see there it picked up me in the frame so it got my name there and also got the confidence level the confidence level will be a little bit low because the camera is off to one side at the moment so let's check what image or what the image looked like that it actually got so if we go out over into a media browser and back into our deep stack snapshots folder there should now be a couple of images so these images will be the same but the face latest will obviously be the last image that deep stacked on so if we open that and you'll see it got me uh there pretty well so and that's looking pretty good now we're going to create the automation for running deep stack um on demand when it gets a person detected in frigate so what we can do is if we head over um into our automations and then we're going to create a new automation so we'll just start with an empty automation and then we'll just say um run deep stack on person detected for the trigger we're going to choose um a state and then we're going to choose remember earlier we had and when we were inside the frigate integrations we saw that we had um a couple of different entities and one of those was the person detected motion um entity so if we search for person and you'll see up the top we have our binary sensor for somebody um being detected in the frame so if we click on that a binary sensor is really nice to work with because there's just an on and on an on and on off state so what we're going to basically do is um set the two to on so what this will do is it will um trigger this automation to run when this binary sensor or when a person is detected um from frigate so set that to on we don't want any conditions for this one and then in the actions we're going to do a call service and we're going to do the image processing service so the exact same as we just did inside the developer tools we just need to select an entity and that is looking pretty good so that will now run every time so if we save that and what i'm going to do is just jump out of frame for a second so you see there that my automation is triggered it hasn't it hasn't run yet because my um my camera has been detecting motion constantly so if i jump out of frame for a second so as soon as i uncover the camera here it's gonna run the deep stack service so if i remove um the cover from the camera and now if we jump back over into automations you'll see that our um our automation has now been triggered and if we head back into the media browser and into local media deep stack snapshots and look at the latest face you'll see there that it did in fact get me um the image is a little bit blown out just because when you take the take your hand off of the camera it all of the light suddenly comes in and it takes a little second to readjust so it's actually even more impressive that it did get it um whilst everything is a little bit blown out so you can see just how accurate and how um well deepstaff works so that is looking really good and the last thing we need to do is actually just to um create uh or add the image into a notification so that we can get it straight to our phone so if we head back into the automation and edit the automation we're going to add to just one more action that's going to be call service for notify i'm going to select my mobile device i'm going to be showing you android here and unfortunately i don't have an ios device so i can't show you the ios notifications but i've done an entire dedicated video again on how to do actionable notifications on both android and ios if you want a little bit more information make sure to check out those videos and then in the so i'm going to choose the three dots and we're going to it in yaml and then we're gonna so we're gonna include some templates in this because i want to include the name of the person it recognized as well as the confidence level so we'll go ahead and we'll choose data or enter data underscore template and then on the next line down we're going to enter message on a multi-line and we're going to use a nice simple template in order to include to the name of the person as well as the confidence um and i'll show you briefly it will open a new tab i'll show you briefly and what the image processing sensor looks like so basically i want to get this name of the the first um person as well as the confidence level we're going to use some templates to do that again watch the template video if you want to see how exactly i go through this how i work out how to use templates and stuff like this but for this purpose we're going to do state underscore attribute and then we're going to enter image i'll just go with processing image processing dot face and then as the attribute if we jump back we want to get faces so faces is the one we want to get so if i enter faces and then i want to close my bracket and then i want to get the first face that it detected so i enter zero remember computers count uh from zero so we entered zero and then again i need to get the attribute which is going to be name in this case so that's the first one so if we enter zero dot name and then we can simply close our brackets and then we're gonna write a little message say um so that's gonna give me the name of the person so that's going to be lewis it was detected with confidence and then again we're just going to copy this um template because it's almost the exact same copy that paste it in here i'm going to change name to if we go back we'll get the the attribute which is confidence let me just put that on a new line so it's easier to see so that is what it looks like for our little message um so that's going to say lewis was detected with a confidence of 80 next thing we're going to do is just to add our image so it's relatively simple the next line down under message we're going to add data and then on the next line down under that to make sure to get your indentation correct we're going to do image and then the path to the image so remember earlier we had the uh i think it was called face underscore latest dot jpg and that's always going to be the last image image that it scanned which is the one we want so all we need to do is enter um in quotes slash media slash local a little bit confusing there because you don't see that local folder when you're in uh the media browser but you do need to include it so it's gonna be slash media slash local and then deep stack snapshots why do i keep typing like that snapshots slash face underscore latest dot jpeg oh back up here in the confidence we're also just gonna round this um otherwise it's gonna give us if we look back in the in the entities it's gonna give us these three decimal points after that and i don't really need them so i'm going to round it to one decimal point like so and that is looking pretty good if we now save that wait for it to trigger and you'll need to move out a frame so that the the binary sensor goes back to the off state and then comes on and triggers so if we wait a second for that to happen now this time when i release it it should send a notification straight to this phone and there we go within a couple of seconds we get a really nice notification with our image in it and it looks like this i'm not sure how clear it'll be um but it's it's uh it's drawn the box around my face and it's also giving us our confidence level and along with the name from the template so that is how to integrate deep stack with frigates so that you can trigger and run face recognition on demand using frigate as the trigger finally it's worth saying that this isn't sort of the final solution that i'm going to be using it's not quite perfect there are some things that do need tweaking and i'm going to continue to and sort of refine them and get them make them a little bit better in the the next week or so before i implement it um but this should get you most of the way there if you want to copy along and do what i've done here and then sort of fine-tune it for your setup and your environment but that is about all the time we have for this video let me know if you guys enjoyed it hopefully and you learned something hopefully it was useful and let me know in the comments if you plan on integrating this or setting this up with your own um your own setup and your own home assistant i'm really interested to see where you guys use this sort of thing it's always very interesting to me so let me know where you you're using it if you plan on integrating it and if you want to support the channel you can do so by becoming a patreon on patreon and your support allows me to keep on making videos just like this thank you to all my current patreon supporters your support is very much appreciated make sure to drop this video a like and get subscribed and i will see you in the next video

Channel: Everything Smart Home

Views: 24,097

Keywords: home assistant, home assistant frigate, home assistant object detection, home assistant person detection, home assistant face recognition, home assistant face detection, home assistant nvr, object detection, person detection, face recognition, face detection, deepstack, smart home, smart home nvr, artificial intelligence camera, ai camera, ai nvr, smart nvr, home assistant setup, home automation, home assistant notifications

Id: 9bMQg20Iy4k

Length: 21min 41sec (1301 seconds)

Published: Tue May 11 2021