Finger Counter using Hand Tracking | Computer Vision | OpenCV Python 2021

hey everyone welcome to my channel in this video we are going to learn how to count fingers we will first look into hand tracking and then we will use the hand landmarks to count the fingers and all of this will be happening in real time and it requires close to no installations and configurations if you would like to create real-world computer vision apps do check out my premium course in the link in the description below here we will learn how to create apps such as object detection augmented reality document scanner and more so let's get started so here we are in our pycharm project and you can see that this is the same project that we used in our previous two videos in the first video we looked at the bare minimum code that is required to do the hand tracking parts and once that was done we created that into a module so that we do not have to write the code again and again and it will be easy for us to create new projects so this will be one of those examples where we create a project out of this module another example that we did earlier was the volume hand control and you can see here that this was the code that we did earlier so all of this is available on my website and now what we will do also there is another folder here you can see that it says finger images so basically what this is that we have the images of different fingers so when it is one when it is two three four five and one it is zero so we can have very specific ones as well where you have the index and the pinky finger up so then you can have only the thumb up and you can have all sorts of different scenarios but for simplicity we are just going to use these six scenarios but if you want to add more later on you can do that and it will pretty much use the same code and you can keep adding on to it okay so once we are in our project we will go to file settings and we will make sure that everything is installed now because this is the same project i know that the packages are already installed but if somebody is doing this for the first time then i will show you what you have to do so here you will write opencv dash python and you will install this and then you will go to media pipe media pipe and then you will install this so both of these are installed so we don't have to worry about that and then we will go to our project we will create a new file and this time around we are going to call it finger something finger counting projects let's say so the first thing we will do is to import our cv2 the opencv library then we will import time and do we need anything else we will also import os so i will tell you why we need os later on now the first thing as always we will turn on our webcam so here we will write cap is equals to cv2 dot video capture and we are going to give in device number one most probably for you it will be device number zero and then you have the option of giving the size so you can write for example cap dot set and here we are going to write that this is number three number three is for the width so i can write the width of the cam and then i can write cap dot set and the heights of the cam so we need to define the width and height so the width of the cam and the height of the camera is equals to 640 by 480. so we can write it like this and what else so then we have to write our while loop so while true then we are going to write here success and image is equals to cv2 dot cap dot read so it will read our frame then we write cb2 dot i am show and inside that we write image and then img and at the end of the day we have to give it a delay so cb2 dot weight key as one so this will give it a one millisecond delay so that we can see our images okay so what else okay the spellings here are wrong success okay so let's try this out and see if we are on the right track so this is my webcam and you can see my hand and that seems good okay so next we are going to do something new here and that will be to import our images so we have all these images so what we need to do is we need to get them one by one and then we want to store them so that later on whenever we have the certain amount of fingers shown then we can display that image so we need to store them first so how do you store it you will use os so what we can do is we can write here that let's say our list my list is equals to os dot list directory so we want to list the directory we want to list all the files that are present in finger images so we will say that our folder path is equals to finger finger images so i will copy this and i will paste it here so now if we print this out you will see that we get a complete list there you go so we get all the names so one thing to note here is that i have put them in order and the last one is zero so when there are no fingers then it will be zero here now you might say why didn't you put it here in the beginning and there is a reason why and i will show you later on why this is the reason what is the reason so then we are going to create a list of images so we can say list of images or we can say let's say overlay because we want to overlay this image on our main image so we will say overlay list is equals to empty and now we can loop through our list so we can say that for image path in our list we want to create we want to import our image so we will write here image is equals to cv2 dot i am read and then we have to given the path of the image so this is the path of the image so it is in finger images and it is one dot jpg so one dot jpg is basically this i am path and then our folder path is basically finger images so we can write here f and then we can write here folder path okay f needs to be small folder path and then we can write slash and then we can write our image path so we can write here image path impact okay so now if you're confused let me show you what this would look like so you can go here and we can print this and we can skip the import or we can import doesn't really matter so there you go so for each of the images you get this finger images at one dot gpg then finger images 2.jpg and so on so this way we get all the paths and we can simply import now we have imported it let's keep it there we have imported it but we didn't save it so we need to save it in our list so we will say overlay list dot append and we want to impend our image so that will give us our image list now to confirm that everything is working fine we can write here length of our overlay list and we can write here print so if that list is six then we should be good to go and there you go so the value is 6 so this means we have imported all these images and we are good to go so how exactly do we overlay an image now the thing is that image itself is a matrix so what we can do is we can define that our new image we want to put in our old image based on this location so what we can say is that our image is equals to so if i wanted the elements the first element of this list i would say overlay list 1 or overlay list 0 right so the same way if i want to target a specific region of my image then i can write that target uh space in this bracket so this is also called slicing so what we will do is we will give in the height first the range of the height and then we will give the range of the width so we will say that i want to put my image this overlay image whatever it is let's say we are using overlay list at number zero so we are using the first image so i want to put that at zero wait what happens uh i think insert is pressed or yeah okay 0 and then 200 this is the limit of my height and then 0 to 200 this is the limit of my width so now now the reason i'm putting 200 is because these images are 200 so the size of this image is basically 200 by 200 so we can automate that i will show you how to do that too so let's try this and see if it works so i will run this and there you go so now you can see the image number one is displayed at 200 by 200 so this is 0 0 and this is 200 200 so our image is now displayed properly if i wanted to shift this i can write for example hundred here and then hundred here so we are getting an error that we have hundred and two hundred uh okay so the problem is that the image is of size 200 by 200 and they are saying that you cannot convert it to 100 by 100 so here hours if i'm increasing 100 here i need to increase 100 here as well so that it maintains that 200 size so if we run this now then you can see that the image has shifted down so this is how you can place your image within our original image so here we are going to write 0 and 200 and 0 and 200 but now we want to automate this so let's say you don't have a 200 by 200 image you have something else you're the size of the image is something else and even the size of each image could be different so what can we do then what we can do is we can write here that for example we are using overlay list zero okay so we will write here overlay list 0 dot shape so this will give us that shape and we can store it in height width and channel because we have uh these three things and then we can replace this with height and we can replace this with width so if we run this now then it will give us the same effect because we don't have to worry about uh the size of the image it will put it on the corner so this is good now we understand how to overlay our finger image on the original image now what we can do is we can display the frame rate so here we can write current time is equals to root time dot time and then we have to write fbs is equals to 1 divided by current time minus the previous time and the previous time we have to declare up here in we will put it as zero and then we will say that our previous time is equals to the current time so this will update every loop and then we can simply write cv2 dot put text we will write in our image and then we will write the then we will write the fps as an integer and we will write here fbs and what else then we will write the what do you call the location so we can put 470 and then we will write cv2 dot font any of these and then we will have the scale and then the color and then the thickness so let's try this out and see if we get a good image whoa that is really big so let's change this to font plane there you go so now it is quite good and we can see that it is working so that is good so the next thing would be to actually go into our hand tracking part so here we are going to go up and we will import we will import our module which is the hand tracking module and we will import it as h m so this is what we will do or let's say htm hand tracking module and then we will create not here we will go down and here we are going to create a detector so we will write here detector is equals to htm dot hand detector and we will not give it any values or should we we can give it the detection confidence so we can keep it a little bit higher so 0.75 let's say later on if we get some errors we can change that too so that is good so here we are going to write that we want our detector to find the hands and we will send in our image and we will just ask it ask it to return our image so if we go to the hand tracking module you will remember that this is our class hand detector class inside that we have a method called find hands and it just needs an image and it will output the image with the drawing so we can write here image we will return image back and if we print this it should draw our hand so there you go so we have our hand and it is drawing nicely so that should be good and then what we can do is we can create a list of the landmarks that we detect so we will say that detector dot find position and we want it to find the position within our image and we want to draw as false because we are already drawing so we don't want to draw again so we will write here that drawing is false and then we can print our lm list to see if we are getting something so right now it's empty and when i bring in my hand you can see that the list fills up so that is good and now what we can do is we can write here that if the length of our lm list is not equals to zero then we are going to do something so that something could be anything so what we are trying to do is that we are trying to get the tip of our fingers and based on that tip we can decide whether our fingers are open or closed so here is the website of media pipe and we can see that these are our landmarks so what we have to do is we have to first of all get all these points so we need point number four point number eights 12 16 and 20. so we need to use these and then we need to check whether these are below let's say number six or number seven i think it's better to take number six you can even use number five but i think number five will be too much so you can say that if number eight is below number six then it is uh what he called closed then the finger is closed and if it is above six then the finger is open so what we can do is we can pick one of these and we can try it out and then we can apply it to the rest of them so let's say we pick our index finger so this is eight and 6. so what we will do is we will write here that if the lm list at number 8 we will get the value of the y not the x so y is the third element so it will be 0 1 and 2 so we will write here 2. so if the point number 8 is less than the point number point number six then it means it is open so in that case we will right here print index finger open so because we are using the opencv orientation so up means lower values so our image is starting from the top so the maximum value at the maximum height is zero so to check for example here our value is 50 and here our value is 100 then it means our finger is open if it's the opposite if it's 100 here and then 50 here then it is it means it is closed so here we are going to try out and see if it says finger open so here right now it's saying finger open if i close it you will see that it stops saying actually let me remove the print it's quite annoying so let's run it again so right now it will say index finger open if i stop if i close then it will stop and if i open it again it will say index finger open so this is how you can tell if the finger is open or not so now we need the tip points for each one of these so for each finger we need a point now we could write a lot of if statements and if you are using a lot of different types for example if you have the pinky finger and the index one up as well and then you consider that a different gesture than your two fingers of any kind for example the index finger and the middle finger up so if you want to differentiate between these then you have to create lots of different images lots of different scenarios but here we are only going to use six scenarios so we can simply use a for loop so what we will do is we will create a list here and we will call it tip ids and this tip ids will be basically number four which is for the thumb number eight for the index number 12 for the middle finger then 16 for the ring finger and then 20 for the pinky finger so these are the tips and then what we can do is we can put a for loop here and then we can change this value so here we can say for let's say id in range what is the range from 0 to 5 we are going to repeat this and here we are going to put in the value so the idea is that we have our id number four so we can write here tip ids at number id okay and the other value so here you can see it is eight and here it is six so it is minus two so whatever value we have here minus 2 so this minus 2. so this should basically loop and it should tell us for each one of these but once it tells us that if the finger is open or not we need to save that so here we can write fingers fingers is equals to empty and here we can write fingers dot append we can append either one or we can append zero so if the finger is open then we will append one if the finger is closed then we will append zero so here we will write prince fingers so let's see how that works out so if we have our hand in you can see all the fingers are open if i close all of them close except for the thumb so we will discuss the thumb but let's try the other ones out so here we have one the index finger then the middle finger then the ring finger and then the pinky finger so you can see it is really good i can do one two three and four there you go so that is amazing so what i was saying earlier is that for example if you have this pose it will show you that the index is open and the pinky is open but for images we are still going to use this image that it will show us that there are two fingers open because we are not using all the scenarios now one thing you have noted is that the in the thumb is an issue and the reason is that the tip of the thumb is not acting the same way so we are using the tip at this point so this is the point where we are using the tip and we are saying that when it is below this point because -2 will be here at this point so this basically is never going below that if i really push my finger maybe not even then so that is not a good way to check it so how can we check it can we say that if this is below instead we can write minus one even in that case it is very hard to bring it below that maybe now it will say zero but it will be very hard so how can you tell if the thumb is closed or not so for the thumb what we actually do is naturally when we are closing we put it on the side we don't bring it down we put it on the side so we can check whether this point here at the top is on the left of the second point or on the right so right now it's on the right now it's on the left so when it comes to the left side we will say it is closed and when it's on the right side we will say it's open so this is the idea so let's see how we can do that so what we will do is we will keep this for loop for the four fingers so we will just make it one to five and then over here we are going to create another loop uh not another loop just an if statement and we will say that the id number one which is the x axis is less than this point id number one and then we will check minus one not the uh two values below only one value below only one landmark below so if you are not clear about this i'm talking about this point number three so if this point is on this side then we will consider it closed if this point is on this side of the point so here it is on the left then we will consider it open so keep in mind for the left and for the right hand it will be different so if you are doing it for the right hand uh it will be the same what i am doing if you are doing it for the left hand then you can uh do it the other way around now you might say that if i am using it with the left hand with this code it will not work yes it will not work but there is a possibility of checking whether it's a left hand or a right hand and then based on that you can change your parameters you can change the if statement uh based on that so it is not something very complicated it's very simple but right now we will just focus on the right hand so here what we will do is uh okay we have already written the code so we don't need to do anything else so we can write here thumb thumb and we can write here four fingers so that is the idea so let's try this out if it works or not okay we have another must be integers or slices okay uh oh there's no id my bad so there's no id we have to get our id by itself so this will be number zero so let's run that again okay then we have another issue uh tip ids zero one okay i think there's an issue with the brackets so there should be a bracket here so tip ids add zero yeah and then the first part of it so the bracket should be here and over here uh it will be tip ids minus one and there you go so it should be like that okay so now you can see it is zero when it should be one so it is basically opposite so we need to make it greater than so now you can see all of them are one if i put my thumb if i close it you can see it says zero so that's how easy it is so now if i put all of them closed you can see it says zero and if i open i can open one by one two three four and five and i can get all the detections so that is good now the next thing we have to do we have to change our image so to change the image first of all we need to know how many fingers did we actually get so to do that we can actually let's go up here let's comment this and instead we are going to write here that our total fingers total total fingers is equals to fingers dot count so this is a method in our list so we have a list in which we can use the count method to count the number of values present of what of the number one so basically we are saying find how many uh ones do you have i think the indentation of this is wrong so we need to fix that okay so then we can print the total fingers let's try that oh it has no attribute counts because c is supposed to be small okay so let's try that here we have 5 0 one two three four five four three two one zero there you go so now it's looking good and all we have to do now is to change our image so how can we change the image so here we have the code for changing so we will bring it in the loop or what is it in the if statement so we will bring it in the if statement and then we have to change the value over here so we have to put a value based on the total fingers so already we have laid down if it is finger number one it should be element number zero so we can take this total fingers wait what happened total fingers minus one so if it is one it will become zero so it will take this image if it is two it will become one and it will take this image so this is how it will work so let's see if it changes we are taking the shape of zero we should take the shape of the same image even though all of them are same it doesn't matter at this point but overall it could be different ok so we have 5 now and then we have zero one two three four five so that's how easy it is one two three four five and it looks like an animation if i do it fast and it's running real time so it looks really good now you might say we didn't add zero we didn't we actually never said go till six so this value of total fingers can be let's say at five maximum right and then 5 minus 1 is 4. how is it going to 6 that does not make any sense right so this is why i put it here as 6. what happens is that when the value of total fingers is 0 it gives the value of -1 and in python what we have is that if we write minus 1 of the list it will take the last element so the last element is the value number 0. so that's why i put the image at the end so it will take that minus one value and whenever it is zero it will become minus one and it will take the last element which is the sixth element so this is uh the results so you can see here whenever it is zero it gives us that a fifth image which is image number six so this is good and we are pretty much done the only thing we can do is we can add a rectangle to show the count so that it is a little more appealing you can say so here we can write cv2 dot rectangle we will put in an image and then we have to give it the uh the points the starting point and the ending point uh 170 so i have tried out this before so i know the values that will work properly so i'm directly inputting those so then we can put the color let's say green and then we can write cv2 dot filled so if we run this uh you will see whenever we have our image or whenever we have our detection we get that green rectangle so this is only coming when we have the hand because it is in the if statement so if we put it outside the if statement it will it will always appear but i prefer that it disappears whenever the detection is not there so then we are going to put the text we will write cv2 dot put text and inside that we will add our image and we will change the text to total fingers and then we will give in the location and then we have to give in the font so we will pick the fonts plain font and then we have to give in the scale we will keep it really big so we can see two five five zero and zero and then we will put 25 so this is the thickness so let's try that and we'll bring in our hands and if we close it's 0 1 2 3 4 5. i will do it again 1 2 3 4 and 5. one two three four five so as you can see it works really well and the detection is really great but again if i use these two uh two fingers it will still show me the image of two and if i let's say put these three up it will still show me the three default image that we have so if you want to change these you will have to put if statements for each of these you will have to have an if statement rather than a loop and you can define if this is the case then this should be the image if this is the case then this should be the image so each of the fingers can be like a binary so if that is 0 0 1 1 then you do this if it's 1 1 1 you do this 1 1 0 0 you do this and so on so you can do it like that so this is basically the idea of our finger detection you can see that it is very easy to do now it might not be the best way to do this because uh with rotation or change in angles you might get bad results but for constrained environment or where you have a known environment this method can be very easy to implement and it could work very well so this is it for today's video i hope you have learned something new if you like the video give it a thumbs up and don't forget to subscribe and share it with your friends and i will see you in the next one

Channel: Murtaza's Workshop - Robotics and AI

Views: 80,952

Keywords:

Id: p5Z_GGRCI5s

Length: 38min 55sec (2335 seconds)

Published: Sat Apr 03 2021