Real Time AI GESTURE RECOGNITION with Tensorflow.JS + React.JS + Fingerpose

[Music] what's happening guys my name is nicolas renaud and in this video we're going to be handling real-time gesture recognition so you'll be able to use your computer's webcam to detect different gestures in real time and see them displayed to the screen let's take a deeper look as to what we're going to be going through so first up what is gesture recognition or gesture estimation well it's a technique that allows us to take a hand pose model so one that for example detects different joints in our hand and work out what that particular gesture is at a point in time so say for example we make our hand into a thumbs up it's able to detect that this particular gesture is a thumbs up likewise if we change our hand to a p sign or to our victory sign it's able to work out that this is actually a victory gesture now this particular form of gesture recognition can be used for example in game play can also be used for sign language detection and a whole bunch of other great use cases let's take a look at what we're going to be covering today so the first thing that we're going to be covering in this video is installing the finger pose library now finger pose is an awesome library because it allows you to take a hand pose model from tensorflow.js and convert that into an actual gesture so work out what the different gestures that your particular hand is conveying at a point in time now from our finger pose model we're actually going to be able to detect gestures from hand poses in real time and we're also going to show how you can use that to control your application so in this case we're going to be making different hand gestures and you'll see an emoji change which represents that particular gesture in the app on the page in real time so let's take a look as to how we're actually going to be doing this so the first thing that we're going to be doing is we're going to be using our computer's webcam to grab a real-time video stream with that video stream we're going to be passing that to tensorflow.js to determine our different hand poses and then from that we're going to take those hand poses and pass them to finger pose to detect our different gestures in real time ready to do it let's get to it alrighty so in this video we're going to be focused on gesture recognition but specifically we're going to be extracting gestures from our hands now in order to do that we're going to be leveraging the hand pose detection code that we wrote in a previous video so if you want the link to that or if you want to walk through that video end to end the link will be somewhere up there and if it's not showing up there just check the description below it'll all be there so what we're going to do is we're going to leverage this code to build a real-time hand pose detector so that's already done and dusted then what we can do is we can take that particular application and extend it in order to perform gesture recognition so this will take a regular hand pose model which detects the different joints in our hands and take it one step further so we can detect different gestures so for example thumbs up and peace or victory so what we're going to do is start off by copying this github repo and again the link to this will be in the description below as will all the code for this video so you can pick it up and get going hella fast so we're gonna grab this code we'll grab this link up here and we're just going to open up a new terminal now in this case my terminal is already pointing to this gesture recognition folder here so in this case if you needed to open up a terminal just navigate to that so to download this we're just going to type in git clone and then the name of our reaper so what we've typed in is git clone and then the name of our repo so specifically this link here and we're just going to hit enter to download that and that's now done now what we're going to do is just rename this because it's no longer going to be hand posed it's going to be gesture recognition and we're going to open up that folder and start making it a couple of tweaks so let's open this up alrighty so i've opened it up in my favorite code editor but feel free to use whichever code editor you prefer so we're mainly going to be using javascript for this and specifically react so again if you want a little bit of a more in-depth walkthrough as to how to set all this up by all means check the link in the description below for the full hand pose model we're going to take this and run with it now if we open up our source folder the main components that we're going to be working with are all in here now this is all from our hand pose model so these steps are specifically for our hand pose model for our gesture recognition model we've got a couple of key new steps so let's paste those in let's make this a little bit bigger all right so we've got five key steps for our gesture recognition model now the first thing that we need to do is go on ahead and install our core dependencies for our hand pose model so we can do that using the npm install command then what we're going to do is install finger pose so this is the library that's going to allow us to perform gesture recognition so again the link to this will be in the description below but you can see that by using our hand pose model we're able to detect different gestures now in this case we're going to be doing something similar to this but we're going to be writing it from scratch now in order to do that we're going to need to handle our state so this is going to allow us to work with our emojis and specifically we're going to import that in we're then going to import our different photos or our emojis build our detect function or update this detect function here so this is the function that actually makes our detections for our hand pose then we're going to handle our control flow so in this case we're going to be displaying emojis but if you wanted to do something else with your application so say you're building a game and you wanted to detect different directions you could actually point in that direction and detect those gestures so that's a little bit of custom gesture recognition but we'll take a look at that a little bit later on now what we're going to do is get started so we're going to first up install our dependencies and install finger pose so let's go ahead and do that first thing i'm going to do is open up a terminal and to install our baseline dependencies so everything that we've got inside of package.json we're just going to type in npm install and that's going to go on ahead and install our dependencies it should take a couple of minutes just let it run and we'll be back in a sec alrighty so those are our baseline dependencies installed now the next dependency that we need to install is finger pose so in order to do that we just need to let's just clear this so we can see it a little bit better so to do that we just need to type in npm install finger pose and then go on ahead and run that so that's now done and dusted now if we wanted to we could actually spin this app up and actually test out our baseline hand pose model so if we type in npm start this is going to open up a new browser and within that we're going to have our pre-built hand pose model that's able to detect the different joints in our finger we're then going to extend this and handle our gesture recognition so let's let that start up and run and then we'll start making some updates okay so we can see that our camera is up and running and if we give it a couple of seconds what should happen is our hand pose model will load and you can see we've still got our thanos hand pose model again if you want to see how we built this from scratch by all means check out the other video now what we're going to do however is we're going to take this model and we're going to allow it to detect different gestures so let's stop our application and then start making those updates okay so we can now mark step one as done so we've gone and installed our baseline dependencies and we've gone and installed finger pose the next thing that we need to do is import use date so i've wrote add new state but really we're going to be adding the use state hook and this is going to allow us to handle our application state later on so this is pretty easy all we need to do is go inside these curly braces here and import use state as well perfect that's step one now done all right now the next thing that we need to do is we need to actually import some of our other dependencies so specifically we're going to import the images that later on we're going to use for our control flow so in this case our thumbs up emoji and our piece emoji or our victory emoji and we also need to import finger pose so this is that line down here so let's go ahead and import those couple of things oh i should also point out that we're going to actually need to grab those images again this is all going to be in the github repo and i'll include links to it so first up what we're going to do is we're going to paste our two images into our source folder and then we can import them so if we go into our source folder i'm going to grab those images and i'm going to paste them in here so you can see we've got two different images so i've got a thumbs up emoji and we've also got a victory or peace emoji so now what we can do is actually import those into our app so let's go on ahead and do that okay so those are our new things imported so specifically what we've actually written there is import star as fp from finger pose so this is our finger pose library and it's going to allow us to work with our gestures then what we've also done is we've imported victory from victory.png so this is importing this image here we've also imported thumbs up from thumbs-up.png and again this is that image there now what we can go and do is mark that step as done so the next thing that we need to do is actually update our detect function for gesture handling so this is a couple of lines of code that we're going to add into this function here so let's move this down so we can see it a little bit better so this is our detect function that we wrote in the last video what we're going to do is after we've done our hand pose detections we're going to also append to that and actually perform our gesture detections so let's go ahead and start doing that okay so what we first started out by doing is just checking that we've actually got some detection so if hand dot length is greater than zero so we're making sure that we're actually detecting a hand within our frame or within our video then what we're doing is we're setting up a new gesture estimator so this is from our finger pose model that we imported up here so in this particular case we're bringing in our victory gesture which is a piece and we're also bringing up a thumbs gesture which is going to be a thumbs up now if you wanted to in this particular space you can actually configure new gestures so if you wanted to do custom gestures you can actually bring them in here so i'll include a link in the description below as to how you can go on about and setting that up but if you want to see a video on it by all means drop a mention in the comments below and we'll take a look at what we can build there alrighty so we've imported our two gestures now what we actually want to do is go and estimate our gestures so let's do that okay so that's the core update to actually go and estimate our gestures so what we've actually done is we've created a new variable called gesture we've made this asynchronous so we're going to wait for the estimation and then what we're actually doing is we're grabbing our gesture estimator that we set up up here and we're using the estimate method and passing through our hand pose prediction which we had up here so what we're basically saying is that we're setting up a gesture estimator that knows about thumbs up and our victory gestures and to that we're passing our hand pose detections then what we're doing is we're passing through our minimum confidence level so in this case we want it to be pretty confident that it's detecting a thumbs up and a peace sign and then we're going to be logging that out so initially we're not going to be handling the emojis on the screen we're just going to be logging out the different gestures that we're seeing so let's go on and kickstart our app and take a look to see whether or not we're actually logging out any gestures and we're just going to open up our console so in this case because we're console logging out we should be able to see our gestures being logged all right so we can see our hand pose model running at the moment so it's going off but it's not detecting anything now in this case you can see as soon as i pull my hands up if i actually give it a thumbs up you can see that within this particular array here we're detecting the thumbs up gesture awesome now if we go and do a p sign let's take a look you can see it's slowly detecting that so you need to have the thumbs up oh your fingers pointing upwards in order for that to detect if we open up our gestures you can see it's detected victory with a confidence of 8.14 so we're now detecting gestures so that's about it so we're now detecting gestures that if you wanted to stop there you could but what we're actually going to do is provide a little bit of control flow so with these gestures we actually want to do something within our application so what we can do is we can actually use these gestures that we've detected and display the relevant emoji to our application so let's go ahead and start making the updates to go and show that alright again we're going to just stop it so our computer doesn't go a little crazy because it is quite resource heavy now what we're going to do is we're going to make the necessary updates in our detect function and we're also going to start working with our different images that we imported here all right so the first thing that we're going to do now is we're going to set up a state hook so this is basically going to allow us to handle our application state so what we want this to do is if we detect a gesture we want our application state to reflect that gesture so what this basically means is that we're going to have a variable that represents what gesture's been detected and then our app's going to be able to work with it so to do that we're going to be using our use state lifecycle hook that is available within react so let's go on ahead and start that off all right so that's our state hook done so what we've done is we've created two new variables so emoji and set emoji and we've specified our use state hook so what this basically going to mean is that we can access our gesture state or our emoji state using this particular variable there and we can set that state using the function set emoji so eventually once we detect our gesture we're going to be using set emoji to set our emoji state so this means that when we detect a peace sign or a thumbs up if we go and read our emoji variable we're going to be able to get which gesture was detected then that particular variable is going to be able to be used to handle our application or display our emoji within our page then we've also set up a images object so this is basically going to map through to the image that we want to bring up and this is going to be used a little bit later on but we've set it up for now now what we actually want to go and do is start working with this emoji state here and this is going to be done within this little gesture section here so this is going to be where we actually go and set our application state for that particular gesture so let's do that alrighty so that's our gesture handling done now let's actually step through what we've gone and written there so first up we're checking that we've actually got a gesture so we're checking that it's not undefined and that we've got a length greater than zero so we want to actually make sure that we've got some gestures detected then what we're doing is we're grabbing the confidence of each one of those detected gestures so in this case if there's more than one gesture detected we want to bring back the gesture that has the highest confidence so what we're first doing is we're mapping through them so we're getting our gesture at gesture we're grabbing we're traversing through that object and then we're going to map through then what we're going to do is we're going to extract our prediction confidence so what this is going to mean is that you've got an array called confidence and within that you've got the confidence of the different detection so in this case if you had more gestures up here we'd have an array of all the different confidences of those particular gestures if they've been detected so say for example it detects victory and thumbs up you want to predict the one that's most confident so this is that first step and then what we're doing is we're grabbing the maximum confidence so we're using math.max dot apply and to that we're passing through our confidence array and then we're just going and grabbing the index of that so this is going to bring back which gesture had the maximum confidence and then we're going back into our gestures and we're pulling out the one with the max confidence and then grabbing that name so this is basically just a little bit of pre-processing to go and grab the gesture that has the highest confidence and then we're grabbing that name then remember we said that we're going to be setting our state so to that we're using the set emoji function and basically this is going to be set to the gesture that's got the highest confidence so it might either be peace or victory or thumbs up and then what we're going to do is we're going to console log that out so we can actually see our gesture emoji name so let's go and run it again and just make sure that we're actually setting our state appropriately and then we're actually picking up the right gesture so again we can run npm start this is going to start up our app and we should be able to see that in a second and again we're just going to go inspect and check our console all right so we're up and running now if we go and hit peace uh all right we've got a bit of an error there let's go back and check that hold up this should be gestures there let's try that again all right so we've got peace got thumbs up there you go so you can see thumbs up popping up there if we do the victory sign that detection's a little bit unclear let's make that a little bit clearer you can see we're getting victory we're getting thumbs up let's actually stop some of the other stuff console logging to see that a little bit better so if we comment out hand pose model loaded up here and if we comment out console log hand so we don't need that and console.log gesture we don't need that we just need to have the console logging out our emoji so let's restart that and so if we step back in now if we go throw up victory it's detecting victory go and do a thumbs up we're getting thumbs up there you go so we're doing quite a fair few detection so it's running pretty quickly but again you can see that we're detecting those different gestures now the last thing that we actually want to do is we actually want to go and handle our application state and do something with this so in this case we're going to be showing up an emoji that matches to our particular state and this is where our different emoji pictures are going to come in handy so let's stop our app make those updates and then we can take a look at that alrighty so what we're going to do is right down here in our canvas section we're going to add our emoji handler so in this case we're just going to have a ternary operator which is basically checking if we've actually got an emoji and then if we do then we're going to display our image so let's go on and set that up okay so what we've actually gone and done there is we've included this ternary operator here this is basically just like an if else check so what we're checking first up is that our emoji is not set to null and remember this is our emoji state that we've got up here then what we're doing is if it's not set to null we're going to be returning this image up here so basically it's a standard image tag and we're setting our source equal to our images object and specifically we're passing through the emoji that we want so this is going to go up to here and it's going to return the image that we need so in this case if we pass through our thumbs up key we're going to be returning our thumbs up image and if we pass through our victory key we're going to be returning a victory image now if we scroll on back then the last thing that we're doing is we're just setting some styling and if we detect nothing then we're going to be returning nothing as well alrighty so now what we can go and do is start this up and hopefully when we detect our emojis we're going to be able to see the result now again this is running in near real time so you're going to see it change pretty quickly and we're going to see a number of results come through so there are ways to perform tune this and if you know of any by all means drop a mention in the comments below and we can take a look so we're going to kick this off and take a look at our emoji and gesture detector oh we should probably save that as well so let's save and then wait for that to compile and there you go so our hand pose model is working and you can see we've got our little detection there so if we put throw up victory that's working if we change our detection to thumb you can see that's changing in real time so this basically allows us to take our hand pose model and make those updates so it's actually using our hand pose model and resetting our emoji so there you go so now we've got some control flow based off our hand pose detection model so again if you throw it up in the correct direction it should work and there you go it's resetting and that about wraps it up so we've done quite a fair bit there so if we go back to our to-do list we've updated our function for our gesture handling we've set up our hook and our emoji object and last but not least we detected and drew our emoji to the screen so we did quite a fair bit now if you wanted to you can extend this further so as i was mentioning before you can actually bring in your own custom gestures so this would allow you to take this a little bit further and actually customize it to what you actually need but for now what we're able to do is we're able to use our app throw up our different emojis and our apps updating in real time so if we throw up our thumbs up there you go you can see it updating and if we go back to victory boom pretty cool and that about wraps up this video thanks so much for tuning in guys hopefully you found this video useful if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get notified when i release new videos and let me know in the comments below what gestures you're thinking of tuning thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 67,276

Keywords: gesture recognition, javascript, tensorflow js, react js

Id: 9MTiQMxTXPE

Length: 24min 4sec (1444 seconds)

Published: Sun Oct 11 2020