AI on the Jetson Nano LESSON 35: Tracking Faces in OpenCV with pan/tilt Camera

[Music] hello guys this is Palma quarter from top tech boy comm and we are here today with lesson number 35 in our incredible new tutorial series where you are learning artificial intelligence on the Jetson nano I am going to need you to pour yourself a nice big mug of iced coffee also because today is a particularly fun and important lesson you may also for yourself a nice cup of hot coffee and now get ready to learn some cool new stuff hey oops I just touched my face you're not supposed to touch your face I touched my face you're not supposed to touch your face okay trying to be careful here in the midst of the corona virus outbreak hopefully you guys are staying safe and I need to say that you're going to hear me with a little bit of a queasy voice today might hear me coffee few times never fear I do not have corona virus but I am highly allergic to Clorox into Bleach it's the one thing that I am allergic to gives me a little bit of asthma and guess what happens in the midst of a apocalyptic coronavirus pandemic zombie apocalypse the cleaning ladies go around and spray Clorox all day long on everything they just making me a little bit wheezy so apologize if I coffee little bit but I want to keep making these lessons because I know a lot of you are stuck at home and you probably want some cool stuff to do so we are going to move forward with these lessons okay I will need you to get out your jet son nano gear and let me get a couple of things out of the way here I need you to go and get your jets and nano gear out I have got to clean this up a little bit okay there we go get ready to learn some cool new stuff hey as always I want to give you guys a shout out who are helping out for at patreon your help is a huge encouragement to me for you guys that are not helping out yet look down in the description below there's a link over to my patreon account think about hopping on over there and hooking a brother up I want to let you know I'm getting a lot of kind of offers from some of these online educational services wanting me to kind of work for them and make these videos behind a paywall I don't want to do that I want these videos to be free for everyone okay but you guys that can if you would help me out I would really appreciate it enough of this shameless self-promotion and let's talk about what we are going to learn today what your assignment was in lesson number 34 we had set up OpenCV on the Jetson Nano where it would track your face so it would put a box around your face a box around your eyes and that was working but what the assignment was was for you to go away and for you to put together a system that would track the person's face by using a pan tilt camera so that the person's face was always in the middle of the field of view okay and now what this is going to do is if you haven't done some of the earlier lessons you're probably going to need to go back and do those and let me kind of show you the setup that we are using the hardware setup the hardware setup that we have I probably at this point need to move out of your way the hardware setup that we have is we have a camera either the Raspberry Pi camera or a suitable webcam and we have that hooked up to a pan tilt servo okay now I showed you how to hook this up in lesson number 30 I kind of I had the links to the hardware and then I also went through a build where I showed you how to put this together and then I believe it was probably in lesson 31 I showed you how to use this servo driver board on the Jetson Nano so if you have not been following a on all of these lessons at home you need to go ahead and look at lesson number 30 and 31 and I'm gonna try to get my focus back with a little luck okay that looks pretty good want to keep this stuff in focus as we are doing this so in summary if you have not followed along in all of these lessons you need to go to lesson number 32 build your pan tilt camera and then lesson number 31 on controlling the servos and if you can do those two things you should be in a pretty good shape you should be in a pretty good shape to follow along on this lesson let me check out here okay so let's let's go ahead and jump into this and so what I am going to need you to do it with all these different screens and all this different stuff takes me just a second to get everything set up so what I'm going to need you to do is I'm going to need you to open up your visual code studio and then we are going to create a new program and so I'm keeping things in papri oh I need to move to a view that you can see there okay so we're now in Visual Studio and what I need you to do is create a new lesson so it's gonna be in the folder OpenCV and then I'm going to create a new program and what was the last one that we had done it looks like it was open cv-22 so this is going to be open CV 23 and then I'm gonna put face pan-tilt dot T IL t dot hi okay we have a fresh new program there and I'm going to go ahead and make the view bigger of the code and now we are gonna go to our browser and I need you to go to the most excellent wwp tech boy calm and then search on the lesson starting the Raspberry Pi camera so this will sort of fire up and make the camera work so I will come down here and copy this code so we don't have to start everything completely from scratch copy and then we will go back to visual studio and we will paste this in and now what we've got to do is we have to start coding now again this should be familiar to you from lesson number 30 in lesson number 31 and so if you really are confused but on what I'm showing you here you need to go back to lesson number 30 and lesson number 31 now in today's lesson I am gonna be using the webcam and so I'm gonna come down here and I'm going to uncomment out the webcam and on mine this should be set to one okay and then we always like to move that window after we show it so after we show the window I'm gonna go ahead and do a little bookkeeping cv2 dot move window and what wouldn't oh do I want to move I want to move nano cam and where do I want to move it I want to move it to zero zero comma zero I never remember whether that should be in parenthesis or not so let's go ahead and run this and make sure we get a happy little window move window move window yes spelling is still important I didn't move it what did I do wrong let's quit out of that let's see one name and then X comma Y that sure looks right to me when name is nano kam and then I go to zero comma zero and it didn't move it so let's try it again it's kind of bad to be having these cut types of problems so early on do you guys see what I'm doing wrong move window nano kam let me just put this in there again this is kind of this is kind of perplexing I want to do move okay and I'll select that and then the parameter is the wind name nano kam and then I want to go comma zero comma zero right mouse click run Python file in terminal ok have no idea of why it didn't work the first time maybe you guys saw it but that looks good that's a pretty good picture too I like that ok let's go ahead and let's start getting reading to code now so our goal is just to have that pan/tilt camera track a face so if we're gonna do that the first thing that we're going to need to do is after importing and printing the version that we are working on and a few of these things here what I'm going to do is go ahead and we want to get those servos ready to go so we're going to have to from add fruit underscore servo and it's guessing it so I'm going to take the guess import servo kit so that's importing the library that will control the servos again if you don't have that library installed you can learn how to do that back in lesson number 31 ok so we've got the servo library we've got to kind of set that up now and so I'm gonna say my kit I'm gonna create the object my kit is going to be equal to servo kit and then I need to tell it that this particular kit has 16 channels so channels is equal to 16 that looks good alright now I got to kind of get those things up ready to go so I'm going to initialize some variables and I'm just going to say always start with a pan equal to 90 and a tilt equal to 90 why because if you can from zero to 180 90 is kind of in the middle so I'm going to set the servo to begin with sort of in the nine in the middle and so let's go ahead and put the servo in position so I'll say skit dot servo which servo will serve o0 and servo zero is the pan one and so then I'm going to say angle is equal to pan and so that will put the pan servo at the ninety degree position and then I'm going to say kit dot servo one that's the other servo angle what angle do I want I want to put that in tilt okay now I'm going to go ahead and run this because I want to make sure things are happy and if we run it as we go it's a lot easier to debug so let's go ahead and run this and see if it moves that servo okay taking a second here takes a second I think with the servos to initialize them and so that library takes a second okay you hopefully heard that little sound and you can see that it moved the camera to a different position so we're talking to the servos boom good milestone good milestone there okay let's go ahead and quit out of that and then let's go down and in our while loop let's see we are reading in a frame okay and then but okay in the while loop we're reading in a frame yeah let's see here sorry about that we are reading in a frame and then as soon as we went read in a frame if we are going to be tracking things what is it that we want to do we want to do our image recognition and we want to do that sort of stuff in greyscale so we're going to create an image called gray and it is going to be C V 2 dot C V T color and then what which image do we want to convert well frame that's what we just read in and then C v2 dot color oops and I froze up over here sorry about that again I have a USB bandwidth problem we come back over here USB bandwidth problem and I'm back live now okay sorry about that guys color and then underscore b g r2 b gr - what i'm just going to come down here and select - gray and i need to move out of you guys this way here there we go okay hopefully you didn't miss any of that color BGR - gray all right so that looks good so now we have a grayscale image and now what we want to do is we want to go ahead and create our face cascade okay so we're gonna say faces is equal to face underscored cascade okay dot detect malt multi okay I'm not sure why that's not that looks good I'm not sure why it's not guessing I like it when it pops up because that usually means I'm topping it right face face underscore cascade dot detect multi multi scale and then I want to detect on what image grade that I just created and then remember we got to put in a couple of parameters that it helps it find a face 1 point 3 comma 5 all right like that okay guys I think - to make this a little quicker because this is gonna be a long video anyway I'm just gonna detect the face I'm not going to detect the two eyes and I'm gonna detect the face and I'm gonna track on that I'm just gonna run this to make sure we don't have a problem yet it's not gonna do anything but I just want to make sure that we are running without crashing looks happy so far fast-paced cascade is not defined I know what I did not do I know what I did not do up here before we do this we've got to define the Cascade so face underscored cascade I forgot to do this face cascade is equal to C v2 dot cascade classifier okay it's recognizing that that makes me happy now we need to give the path all the way to the Cascade so we're going to come back over here to our file explorer I find it easier to just copy the entire path name and so we are going to come over here and navigate we're gonna go to you know once we've opened up our file explorer we're gonna go to desktop and then go to pipe row in cascade and then on this face I'm going to go ahead and I am going to look at properties okay and then there is the path so I'm gonna copy the path and then that will give me the path so I'm gonna paste it and then I just need to put the file in my file was face dot XML and remember we did this in the last lesson setting these things up and I think I'll go ahead and do the eyes okay this is going to be a long lesson but we will detect the eyes as well because it's good practice so I'm gonna copy that line because the next one will be almost the same and I'm gonna paste it and this one instead of face is going to be what I remember last time we downloaded those two hard Cascades and so this one will be eye now we should be able to detect faces and we should be able to detect eyes now when I come in and say faces and then say face cascade it knows what face cascade is alright because I'd forgotten to define that last time so let's go ahead and run it and make sure that it's happy on long programs like this if you do all the coding and then run it it's sometimes very hard to find your errors and so that looks good no errors everybody's happy and so we're gonna queue out of that so now we have recognized faces okay now what we want to do is we want to step through the faces that have been recognized so I'm gonna say 4 X comma Y comma W comma H all right 4 X comma Y comma W comma H in faces okay so faces is going to return to you a list of arrays and each array is going to have the X and y of the corner of your face in the X plus width and y plus h of the other corner so got the four numbers that you need for the two corners of the bounding box of the face that was detected okay now what I'm going to do at the bottom here since we're trying to track something I'm only going to treat there was more than one face I don't want it to be moving back and forth between the faces trying to track both of them so I'm going to go ahead and put a break in here so it will only do the first one so after it does the first one then it will break in if for some reason there was two faces it would not try to find both of them okay now I've got a box around the face what I need is I need to know the center of that box okay and so how would we define the center of the box well I'm gonna say like the the X Center and what the X Center is going to be equal to its going to be equal to the X plus half the width so it's going to be the X plus the width divided by two does that make sense if this is the x position of the corner then I would move over by the width divided by two and that would be the middle in X and then similarly Y Center is going to be Y plus what H divided by two I think that makes sense does that make sense all right now those are the center positions what I have to do is I have to now find the error right if I want if I want to be in a certain position and I'm in a different position I've got to find the error and so I'm gonna have an error in the pan and I'm going to have an error in the tilt okay those are two things that I have to figure out the air in the pan and the error in the tilt okay well what I need to know is I need to take that center that that X Center and I need to - width / - okay and same thing the error in the tilt is going to be equal to the object Y okay and then minus the height / - so you can see that that x position this X center and Y Center is going to change every time and then every time I'm going to create an error alright so as I move around that error is going to move around because it's I move around the center of my face changes okay The Sitter of last face changes so I hope that that makes I hope that that makes sense and what this actually is this is the width of the display and this is the height of the display and so I'm I know that I am NOT explaining this really well but if you think the whole display the whole camera has a certain width the whole camera has a certain height all right it has a certain width and it has a certain height and the center of the image is the width divided by 2 in the height divided by 2 that's the sitter of the camera and then this X center and Y sitter that's the center of my face and the difference between those two that's the error and that's what I'm trying I'm having to drive out now with that we got to make sure that we defined X center and Y center which we haven't done yet so I need to go back and I need to do that and for my camera for my camera I need to check and see what it is okay for this camera the width the display width is 640 in the display height is 480 all right so I need to use those down here a different variable display width and display height now let me make sure that I did that right no I didn't display width display height okay guys I apologize I haven't explained this perfectly but what I want is I want the center of my face to be in the center of the camera view the camera is 640 wide and 480 high and so if I take half of that and half of that I get the center of the screen now my face is moving but I've got the X plus half the width and the y plus half the height so this is the center of my face X plus half the width Y plus half the height that puts it here now the error is the center of the screen to the center of my face and that is the error that I want to drive to zero by moving the servos around if that doesn't make sense I think I explained it clearly there the last time and if that doesn't make sense you know stop it and listen to it again the one thing you got to be careful of is you've got to make sure that your display width for whatever camera you have is set correctly so if you're firing up the Raspberry Pi and you're fine firing it up at a different screen resolution the the PI camera or you you're using a webcam with a different screen resolution you've got to make sure that you set it for the camera you're using okay I think that I am back on track here and now we can continue to move forward okay so now I have the errors I have I have the error in X and I have the error in Y okay now what I got to do is I've got to start driving those errors out so what I'm going to say is if the absolute value of the error the absolute value of the error in pan let's say what if that is greater than 15 pixels okay if the error is greater than 15 pixels what do I want to do well I need to say that that the error is big so I've got to subtract something so I've got to say pan is equal to pan minus the error pan and I'm going to divide by 50 okay if I just subtracted it out it would swing that thing too hard and you would overshoot it so I want it to kind of go in small increments and so I'm going to try saying the pan is equal to pan - error pan over 50 so I'm driving out one fiftieth of the error okay now why am i subtracting because the the absolute value is greater than 15 so I want to make it smaller so I'm going to subtract right now what I can do next is I can say I need to back out of that I can say if the absolute error absolute value of the error if the absolute value of the error and tilt is greater than 15 then what I want to do is I want to say tilt I want to say tilt is equal to tilt - error tilt / 50 okay so because I'm doing the absolute value I only have to do two if statements because if the error is negative then it will change the sign in the error pan so you see error pan could be positive it could be negative but it's always going to be right so it's kind of like I'm doing two commands with one if statement because I'm using the absolute value if that doesn't make sense stop and think about it a little bit that if the error is positive then it's going to be subtracting a positive number if the error is negative it's going to be subtracting a negative number which is going to be adding CC this will work for whether you're too far this way or too far that way and same thing with the second if statement okay now the thing is I've just got to make sure that the the servo can go from zero to 180 and if I go outside that range I need to stop so that I don't like get an error message so I've got to put some error checking in here so I'm gonna say if pan is if pan is greater than 180 then what I'm just gonna say is make hand equal to 180 and that way it could never be more than 180 you're kind of pinning it there okay and then what I could do is I could say print and I could say print what I would print hand out of range just so the person knows there's a problem okay so I think that would work I think the easiest thing we're gonna have to do this several times and so what I'm gonna do is I'm just gonna copy and then I'm gonna paste and that did not paste very well let's see I need to take that back ok so if pan is now less than zero I would make pan equal to zero and same thing pan out of range now these two are going to be the same only it's going to be tilt so I'm going to copy those and I've got to kind of fix that for tilt so then I'm gonna say control V and I think that did better that time no it did ok so now I will say if tilt is greater than 180 then make tilt 180 and if tilt is less than zero then make tilt equals zero so now I've got things pinned on pan and tilt I got them pinned between 0 and 180 ok so that I think will be good so we got all that taken care of so let's see what we want to do now we probably want to go ahead now and move the servos so just like we did before I'm gonna say kit dot servo and I'll start with the pins I'll start with the pan servo so I'll say kill kit servo and that was servo 0 is the pan 1 and it's gonna be equal to pan and we calculated pan up there just a couple of lines ago alright and then what we're gonna do is we are gonna say kit dot servo and now we're gonna do the tilt servo which is servo 1 and then we are gonna say that is equal to tilt and we just calculated tilt again a few lines ago alright so that should move the servo now just for fun I'm sorry this is gonna be a long lesson but I do want to go ahead and I want to put the eyes on there and so that'll just take us a second so what I'm going to do is I'm going to create a region of triste from our gray you know so we're gonna be making a gray region of interest because we're looking for eyes and if we're looking for something we need to be using a grayscale and that is going to be based on our gray image which we created above and we're gonna go the region of interest is from y to y plus h okay and then we're gonna go from X to X plus width now what you got to remember is is that when we're doing this with these brackets we're treating gray as an array and when you have arrays you go rows first and then columns and if I go rows first that is the Y value so it's rows commas that's why X so you got to be careful if you're doing this as points you go X comma Wat but if you're treating it as rows and columns it's more like you go Y comma X and we've run into that before and I just hope you guys will remember that okay and then we're gonna go ahead and make an roi color cuz that's what we'll draw on and the roi color is going to be equal to the that was the frame I believe frame and then it's going to be the same thing that same range all right that looks good so now we're gonna go ahead and we're gonna find our eyes and so we're gonna say eyes is equal to I cascade i cascade and then we're gonna do a detect multi scale detect multi scale and then what do we want to detect in roi underscore gray and then do you remember that for these eye detection we don't need to put any parameters it kind of finds the eyes without putting any parameters in and now we're going to need to go for and we're going to need to go from the eye at i why i w I H okay X Y WH just like before but I put the e in front of it so it'll be different than the X Y WH that we use for faces and so we're now we're going to step through the eyes here alright and we'll want to do both eyes and so in eyes so it finds the eyes now we're gonna step through the eyes and it's only looking for the eyes inside the face okay now we'll say see the two dot rectangle see v2 dot rectangle and then where do I want to put the rectangle I want to put it on roi color because I want to shut this to show up on the color image and then we're gonna go where do we go II X comma E Y that's the upper left of the found eye and then I need to go to the bottom right of the found eye which would be e x + E width comma and then I'm gonna go e y + e h UI + eh and I think I'm missing a parenthesis right did I miss that exe why okay and then e^x + CW alright so that is going to be then the corners of the eye and then you'll go through the for loop another time and it will do the other eye okay we got to put a color on there so let's go ahead and let's make the eyes blue 255 comma zero comma zero and then let's put a width of two on it so that is going to box the eyes this should make sense from that other from that lesson that we did last time this is very similar to the lesson we did last time okay then I think what we will do is we will then show the faces men this is kind of scary isn't it all right I need to kind of see if I can give you guys a view I want to try to give you a guy's a view of the camera so let's see if I can move this I want you guys to see the camera moving so let's see if I can kind of show you a little bit more of the action and then I'm gonna kind of put this more over here pointing that way and let's see if I can bring this up a little bit okay hopefully that will work we'll have to play around with this a little bit but I guess the the work at hand is to just see if this programs gonna run the suspense builds I should have made you guys hold your breath ooh invalid syntax Oh ro I gray did you guys see my error there on ROI gray look at that that should have been a lot that should have been a plus did you guys catch that up let me show you where you can see what I'm doing right that should have been a plus there and then from X to X plus W alright that was kind of a big deal so let me go ahead and run this still invalid syntax cv to rectangle I think I had on the rectangle statement I had an extra closed parenthesis it's one of those where it puts it in for you and then I put it in and therefore we both put it in so we have too many all right let's run this so far so good okay boom look at that good news is the program didn't crash the bad news is we've got no idea whether this thing is going to work or not so let me see if I can okay that's kind of interesting so let's see if it will find my face [Music] that was kind of awkward when it saw my face when it saw my face it crashed okay width is not defined so there was something that I should have done there where it said width in light I probably use the wrong variable name which line is that line 28 did you guys catch that when I made that mistake in line 28 okay so I say error pan I think that should be W no this is the X Center I got to think what error pan is this would be W divided by 2 I do believe and this was why Center I think all right let's think about this on the error pan it's the difference between the center and okay X Center and y Center is the center of my face okay and the center of the screen would be display width and display height I believe now I am completely confused alright X Center is the center of my face the center of my face would be X plus W over 2 that's the center of my face y center is the Y center of my face and that's why plus h divided by 2 the error is X center - the display width / 2 so I need to make this one display width display height okay where is the center of the display well it's the width divided by 2 and it's the height divided by 2 and the error is from where the X Center is - I mean which is the center of my face - the center of the display does that make sense so let me let me get full screen here and go through this and I'm sorry that I'm not explaining this better I have the display width the display width divided by 2 is the center of the display the display height divided by 2 is the center of the display now as my face moves this is still the center of the display this is the position of my face the center of my face is the X plus the width of my face divided by 2 and the y plus the height divided by 2 so now X Center Y Center is the center of my face and display width divided by 2 and display height divided by 2 is the center of the screen and if I subtract this from this then that is the error okay I hope that made sense stop and listen to what I said again if it doesn't make sense but let's try that and see if that will run so far so good okay it's running now let's hope that my face doesn't crash it this time okay error in line 46 I must have not oh okay okay okay I see what I did did you guys catch that this should be dot angle all right I didn't give it that parameter dot angle do you guys catch that I will need you to hold your breath this time I think part of our problems is we're not holding our breath so let's try that please please run this time it's hard to hold your breath this long when it takes so long to load the library all right I won't give a boom yet but at least it is running so let me come over here with the face it's still crashing when ah look at that it did you guys catch it it was in line 33 line 33 I had a typo error tilt okay so tilt is equal to tilt - error tilt over 50 I hope you guys caught that okay let's run it this time for real I think it's gonna run this time it kind of hurts my feelings when every time it sees my face the program crashes I'm just saying okay let's see here okay so we're coming over here to give it a face it is tracking me boom boom look at that now we didn't put one thing we didn't do is we didn't put the rectangle around the face and that would probably be useful so let's do that let's go ahead and let's go ahead and put a rectangle in here on the face and so we need to come back over here and Forex and faces okay we do all of that and where would be a good place to do that [Applause] let's see I think we would do it right after right after we entered right after we enter this for loop let's go ahead here and say see v2 dot rectangle and then the rectangle is going to be on frame and it is going to be at X comma Y right which we get right above there and then it goes to the other corner which is going to be X plus width X plus W comma y plus h I really think this is gonna work and then we're gonna make it let's go ahead and make it red so it would be zero comma zero comma 255 and then we are going to go at about a two okay about two line width so let's try running this again this is going to work this time okay there it goes alright so now let's give it a face to look at all right look at that we got our rectangle boom boom okay double chest bump okay I think that is engineers we really need to celebrate our successes war why shouldn't the professional athletes do a double chest bump in not us engineers I think we should start adopting this and in fact you could start doing this at work and for other reasons too to celebrate successes but look at that swinging around this way tracking in real-time aha Jetson nano artificial intelligence tracking my face I need to create a website www.rockymountainatvmc.com I kind of got a little bit too far ahead of it okay let's make sure if I stand up it will track I went a little bit too fast okay it's tracking up tracking down the one thing I don't like about it with this camera it looks like the cameras trying to do it looks like the cameras trying to do an auto white balance and so what happens is when it sees the green screen behind me mmm it tries to do an auto white balance so I wish that I'd need to play going and figure out the command where I can turn that off okay turn that off all right guys I am just super duper super excited about this and like if it can find me again okay got a little bit out of range okay there it is alright guys I am just super super super excited about this project this is really kind of we have done a lot in the last 35 lessons and I hope you guys feel like that you've learned a lot I mean you are really doing some artificial intelligence now and I think this is a pretty a pretty cool project okay what we've done so far is we've used pre trained models now the same type of thing you can go back and you can look at those cascades that I showed you on the open CV github you can download other of those cascades and you can track on other things and you know you can kind of play around with that but you're to the point now of doing some pretty sophisticated stuff so I've got a pause now and figure out where we're going next with these lessons but I really am having a lot of fun with this and hope that we can continue to do more lessons okay guys again I really appreciate you guys that are helping me out on patreon it really means a whole lot to me and allows me to continue making these videos if you guys aren't doing that yet think about thinking think about helping me out and then also you guys if you would give me a thumbs up that helps a lot think about subscribing to this channel think about sharing this with other people sharing it with other people get more people involved in this work all right Palmer quarter from top tech boy comm I will talk to you guys later

Channel: Paul McWhorter

Views: 5,865

Keywords:

Id: dINNPMLSBWo

Length: 53min 1sec (3181 seconds)

Published: Sat Apr 18 2020