Raspberry Pi LESSON 60: Finding and Tracking Faces In OpenCV with Haar Cascades

hello guys this is Paul mccorter with toptechboy.com and we're here today with episode number 60. in our incredible new tutorial Series where you're teaching your Raspberry Pi who's boss what I will need you to do is pour yourself a nice tall glass of ice cold coffee fat would be straight up black coffee poured over ice no sugar no sweeteners none needed and as you're pouring your coffee as always I want to give a shout out to our friends over at Sun founder Sun pounder is actually sponsoring this most excellent series of video lessons and in this class we are using the sun founder Raphael kit for Raspberry Pi now most of you guys probably already have your gear but if you don't look down in the description there is a link over to Amazon and you can hop on over there and pick your kit up and believe me your life and my life are going to be a whole lot easier if we are working on identical Hardware but enough of this Shameless self-promotion let's talk about what I'm going to teach you today and we have got a great lesson lined up for you today now you know in the last few lessons in the last five or six lessons we've become very familiar with opencv and we're very comfortable in thinking about the frame that we grab from the Raspberry Pi camera as being a set of data and we've been able to go in and look at that frame and find and track an object of Interest so far our object of Interest has been something that's a bright color and we have been finding and tracking and all of that sort of stuff based on color well that was just to introduce you to the concept get you comfortable with the idea comfortable of working with that frame but today we're going to take things to a whole new level because today instead of just tracking something based on color what we're going to do is we're going to grab that frame we're going to analyze it and we're going to see if we can find box and track a face pretty cool I hope it sounds like fun to you I am having a whole lot of fun with you with this stuff okay opencv actually has some pretty sophisticated artificial intelligence features built in and what I'm going to show you is I'm going to show you today how to begin to unlock some of those features an opencv actually has some models of a face so what we can do is we can go out we can get the frame and then we can analyze the frame looking to see if there is anything in it similar to a face so there's some pre-trained models now what the challenge is the challenge is to find those pre-trained models and that's what we're going to show you today so enough of this introductory talk what I think I will have you do here is let's jump on over to the Raspberry Pi View and what we want to do is we want to go ahead and start with that program we want to start with that program that fires up the camera so you can come to the most excellent www.toptechboy.com might be helpful if I got out of your way at this point and then what we're going to do is we're going to come over here and using the happy little search icon up here you're going to search on something like calculating frames per second in opencv and then you're going to come down here and then you're going to grab this close code by clicking on those two little Pages you can see it copied it we're going to come back over to thawny and then we're going to paste that into thawny a couple of things I want to do I want to come in here I don't like calling this m I like to call it frame that wrong keyboard I like to I like to call that frame because we're grabbing a frame so I'm going to come here I am I'm going to call it frame and then here see the put text we're going to put it on the frame and then CV2 I am show we're going to call that frame okay A little quick adjustment there now the other thing that you see is I've got this mounted on the pan tilt hat and you guys we've been playing with that in the last few lessons if you're just jumping into this lesson and want to be able to actively track something there is a link down below to this uh to this pan tilt hat if you don't have the coin for the pan tilt hat ninety percent of what I'm doing you could probably kind of get by if you just got one of those little blue servos and taped or glued your uh your Servo to that you would be able to at least do the parts of the lesson that are pan okay the other thing is that when I put it on this you can see that the camera is upside down so I need to go ahead and compensate for that to flip it back around and we can do that by just saying frame you only do this if your camera is upside down frame is equal to c v two dot flip what do I want to flip I want to flip the frame and I want to do a minus one flip and what that does is it rotates 180 degrees okay sound good let's go ahead and run this thing and see if it works good excellent look at that almost 30 frames per second on a 1280 by 720 1280 by 720 frame so that is doing pretty darn good okay now if we are going to go in and look for a face we have to find that pre-trained face model and that is called an opencv a har h-a-a-r a hard Cascade we need to find the hard Cascade or the model for the face and believe me if I tried to navigate down in the folders and you tried to follow me it's very likely your folders could be named differently and you're going to get completely lost so I'm going to show you a little Pro way a little Pro way that you can find where your hard Cascades are so just follow along and it will work no matter where they are so what I want you to do is you see we are down here in the python shell so we can enter pipe on commands one at a time I need to say import OS for operating system now I want to find the path to the CV2 folder so what I'm going to do is I'm going to say my path so I'm just creating a variable path and that is going to be equal to OS the library that we just imported Dot I want it to give me the path dot I want it to also give me the directory name okay of what of OS dot path dot a b s for absolute path like that and what CV2 dot underscore underscore file underscore underscore two underscores on each side for a total of four close the add path and then close the door name and then hit enter and now let's just put path and boom that has given me the path to the opencv folder for my system you will get it for your system and for for example you probably are not going to be pjm you might be Pi here you might be your username there so now let's go see if we can find that so what I am going to do is let's open up our file explorer here like this and then you can see that we are already in slash home slash pjm or you would be like slash home slash Pi or slash home slash Joe but you're probably already there so now what folder do we want to go to we want to go to the dot local folder uh oh no dot local folder I'm going to come back here to the icon view like that you see no dot local folder so what do you do right Mouse click show hidden and what do you see giddy up you see your dot local you see your dot local okay so what I want to do is I want to go inside of dot local next is what lib so let's go inside of lib next to SWOT python 3.9 and then site packages right and then CV2 so we are now in the CV2 folder that's a huge amount of success now where are these models they are inside of data so if we look there are all of the models let me go back to the uh list view so you can see there's like a hard Cascade for the eye the eye with eye glasses frontal face frontal face all uh full body so there's all these different models of things you can find okay we'll talk about that later but let's go back up what I want to do is I want to make it more easy to load one of those models because if we did it right now we would have to put in this very huge long difficult to type in path and I want to put and then we'd forget how to do this next time so I want to put these models somewhere that they're a little bit easier to point to and a little bit easier to find so go up to data right Mouse click and copy okay now I need you to go to your python folder the folder that you use to store your python programs and so for me let's see I'm going to come back here and I'm going to I am going to turn off I don't like seeing the hidden files so I'm going to come here I'm going to right Mouse click and I'm not going to show hidden anymore okay because it kind of uh it kind of uh crowds things up it's too too much too much information okay so now what I'm going to do is I copied that data folder I inside of documents I have a python folder where I keep my programs and then I am going to right Mouse click I am going to right Mouse click and I am going to paste and now I have that data folder why when I run a Python program and then inside that program tell it to look for a file it looks relative to the folder that you are in the the the the folder that that program is in that is what the starting point is so if I put these models there it's going to be a lot easier to find so I'm going to rename this though I'm going to come in and I'm going to rename it h a a r because if we come back in a month right we're going to what what was that oh it's already there so let me I'm sorry I'd done this before and so it was already there okay but when you do it now you should have this folder called har and if we look in there there are all of our different models okay so now we know where they are they are inside of my default python folder and then it's inside of a folder called heart so we'll need to remember that now we're going to come over here and we're going to see if we can start doing a little artificial intelligence and a little bit of face finding and so the first thing that we want to do is we're probably I think a good thing would be to come here and before we go in this while loop this is where we are going to start doing our magic okay so the first thing I need to do is I need to create a I need to create an object for my model so I'm going to call my model face I'm going to call it face Cascade like that and what is that going to be equal to it's going to be equal to CV2 dot Cascade classifier uppercase C Cascade uppercase C classifier now I have to point it to the model so I open a string first of all I'm going to put a dot that dot means start in my current working folder that's the folder where this program is saved right that's my python folder so I'm starting it dot my python folder then I need to go down to which folder Park and then I need to put the name of the file so let's look at these and you see all of these different models the one I want to use is this hard Cascade underscore frontal underscore default dot XML okay so let's see if we can put that in h-a-a-r-c-a-s-c-a-d-e hard Cascade underscore frontal face underscore default dot XML close the string and then close the uh the classifier like that okay so we have that done now so I've created my model I've created an object for my face model now we need to use it okay now what you got to see is this is becoming very computationally intense and when I have a big 1280 by 720 frame that is a huge file and it's even Huger it's Huger because it is color and so it's you know what I want to do is I want to make it a little smaller so my program will run a little bit faster and so what I'm going to do here after I grab grab my frame I'm going to create a new frame frame gray like that okay I'm going to create a new frame frame Gray and that is going to be equal to cv2.cvt color like that I'm going to convert the color and what do I want to convert the color of frame and then how do I want to do it CV2 Dot color cd2 dot color and what do I want to do I want to go from BG R to gray okay because a grayscale image will be a lot smaller and therefore our program can run a lot faster [Music] I better do it after I better do it I think I better do it after the flip so just to make sure things don't get confused I'm going to do it after the flip like that I think that would be smaller all right now I'm going to go in and I'm going to see if I can find faces all right so what I'm going to do is faces is equal to face under a face Cascade that is the object that I created right and then I want to do a DOT detect multi scale okay detect multi-scale what do I want to look at I want to look at my frame gray because it's going to go a lot faster if I do that now I have to give it a couple of parameters and I'm not going to talk about these parameters very much you can play around with them and you can study them but I don't want to spend an hour talking about them the first number is the scale factor and that's just saying as you're looking for faces if there's multiple people there might be a person closer and a person further away and their face might not be the same size and so we got kind of got to give it a range of scale factors and 1.3 works pretty well and then the next thing is kind of like how many nearest neighbors need to find a face before you call it a face select if this sees a face and this sees a face and this sees a face you know it's the rectangle is moving around how many times do you need to see it in nearest neighbors before you call it a face and five works pretty well you can play around with not specifying the parameters using default ones play around with it study it but I'm just trying to get us where we're finding faces now so I'm not going to talk really any more about that okay so now what we should have is we should have this data structure called faces and that data structure called faces that data structure called faces should return us data if it finds a face and so what I'm going to do is let's just sort of see what happens I'm going to try to point this at myself a little bit more and what I'm going to do is just print faces like that so we'll see what we get back all right and so I'm going to come in here and I'm going to run this thing hold your breath I it ran okay now look at this we are getting data we are getting data so I hope if I quit this that doesn't disappear my data doesn't disappear okay good it didn't what do we see we see an array inside of an array okay and we see four numbers now let me let me kind of draw this out because it's really important that we understand this Why didn't it just give me the four numbers why did it give me an array inside of an array okay that is because let me I'll just kind of draw up this data structure for you okay the inside array is it found a face okay it found a face and then what it is giving me is it is giving me the x y coordinate of the upper left corner of the bounding box of the face and then it is giving me the width of the bounding box and it is giving me the height of the bounding box and so what it is giving you here is X comma y comma W comma height like that all right but it is giving it to you inside of another array why what if you found three faces well this would be face one and then it would have those again for phase two comma those again for face three and then it puts all of those in an array so what would I have faces of 0 would be that first face x y w h like that and then faces of one would be the next one okay but what does that mean that means even if I have one face I can't just go in and say faces right I have to index faces even if there's only one face I've got to refer to then what faces index 0. okay hopefully that makes sense but your life really will be easier if you understand these data structures hmm so here we are so what was the 618 that was the X position up the upper left corner of the box that would go around my face 98 would be the Y value 286 was the width and 286 was the height well if we have all of that if we have all of that what would be a pretty cool thing for us to do now what would be a pretty cool thing for us to do now draw a box around it okay now the way I'm going to handle the the array in a array I'm going to say faces is the group of arrays of the individual face singulars okay so what I'm going to do is I'm going to say 4 face in faces face singular and faces plural so if there's five faces it'll step through all faces if there is only one face it'll just go to that one face but four face in faces now what am I going to do I'm going to say x comma y comma W comma h is equal to that individual face so it's going to take that array the first element is going to go in X the second element is going to go in y the third element in W and the fourth element in h so now I have what I need to do a what CV2 dot rectangle okay cd2 dot rectangle I want to put the rectangle on what on the frame back on the original frame okay and then what's the upper left corner it is the Tuple X comma Y close the Tuple what's the bottom right corner it would be X plus W and H Plus uh comma y plus h okay then let's give it a color I think I'll go with the blue box today what do you think 255 comma 0 comma zero and then how heavy do I want that box let's make it a 3 and then I'll go like that could it really be that easy now the cool thing is if there are no faces it won't even go into this so we don't have to worry about the program crashing if it doesn't find anything okay I will really need you to hold your breath this time ah look at that boom I'm human do you realize how heartwarming it is that opencv recognizes me as a person that I am not so obscure that it doesn't recognize me as a human being look at that okay one thing is I want you to notice is look at that it found it and it is tracking me it's a lie it is working what do I also want you to see what I also want you to see is we have taken a big hit and we've gone from almost 30 frames per second to six frames per second because we are really doing a lot of computation we are really doing a lot of computation what if we came here and we went 640 by 360 like that okay so you see now I've gotten my frames per second back up higher okay I've gotten my frames per sec second back up higher all right so that is uh that is what is needed and I'm going to go back I'm going to go back to 1280 was that right or was I yeah 1280 by 720. I thought that was right okay there I am and we are getting about six frames per second okay now guys this is what your homework assignment is going to be your homework assignment is going to be and give me just a second here I need to take a look at this and make sure that I am telling you right okay this is what your homework assignment is your homework assignment is I showed you how to find and box a face what your homework assignment is to go in and find and box not only the face but also find and box the eyes okay find and box the eyes now I did to get the eyes to work good I will give you I will give you a hint I use the uh hard Cascade underscore I okay and then also I played around with those parameters and the scale factor remember when you do the detect multi-scale I did a scale factor of about 2.25 and then I did the number of nearest rectangles at 25. now you can play around with those but just don't think that if you don't use any parameters at all there when you do your detect multiscale that it's going to find the eyes good you'll have to play around with those a little bit but what your whole work assignment is is to go in find the face box the face find the eyes box the eyes and then report your frames per second report your frames per second make a video put it on YouTube show that you're finding the face you're finding the eyes and show your frames per second because you can see what we're starting to do we're starting to really tax our CPU on the Raspberry Pi we're not getting 30 frames per second we're getting down to a few frames per second and there's still a lot more work that we want to do and so we are really starting to squeeze the processing capability of the Raspberry Pi okay guys I hope that you are having as much fun taking these lessons as I am making them I am just really really excited about where we're going here let's see my camera is a little bit off there it should pop back here and uh it should pop back here in a second there it is okay uh let's see how Alan people are doing we haven't checked in on them in a while you want to check in and see how they're doing let's see ah it looks like the boat is departing huh the boat is departing I just love watching my friends these people are my friends and they're my neighbors and they're out getting their work done let's see Lady is probably planting her garden up there you can see the banana trees are looking good okay I got digressed I I got I got sidetracked there so what is your homework your homework is to find the face find the eyes report it on YouTube and show your frames per second and don't cheat by going down to really little things do the 1280 by 720 and let's see who can get a 1280 by 720 let's see who can get the highest frames per second guys I hope you are having as much fun taking these lessons as I am making them if you enjoyed the video give us a thumbs up if you haven't already subscribe to the channel when you subscribe to the channel make sure that you ring that Bell so you'll get notification of future video drops and as always share this video with other people because the world needs more people doing coding and fewer people sitting around watching silly cat videos Paul McCarter with toptechboy.com I will talk to you guys later [Music]

Channel: Paul McWhorter

Views: 4,149

Keywords:

Id: jcdMW3z_QrI

Length: 29min 21sec (1761 seconds)

Published: Thu May 04 2023