Advance Gesture Volume Control [Part 2] | OpenCV Python | Computer Vision

hey everyone welcome to my channel in this video we will create an advanced gesture volume controller using computer vision this is the second part of the gesture volume control project here we are going to make the system smoother and use the pinky finger to set the final volume we will also add some filtrations to make it more stable and usable in the real world application if you would like to create real world computer vision apps do check out my premium course in which we learn how to create apps such as object detection augmented reality document scanner and a lot more the link is in the description below so without further ado let's get started so we are here in our pyjama project and this is the same project that we used in the hand tracking project in the virtual painter in the finger counter so we have done quite a bit of projects earlier and this is the same project that we did our volume hand control as well so we have the files here so we have we don't need the rest of them but we will need the volume hand control and the hand tracking module so if we open hand uh volume control you will see that this is our code that we did last time this time around we are going to make some changes we are going to add a few more things to make it a little bit better a little bit smoother and a little bit more usable so that is our main idea so what we are going to do we have everything set up and if you are viewing this video for the first time you have to actually install media pipe and opencv python so you can go and install from here you can write opencv python and then you can write and then you have to hit install and then you have to write media pipe media pipe and then you will hit install so these are the two libraries that we need and the third thing is your hand tracking module that will be required in the same folder so that is the main idea now i highly recommend that you go and check out the first video before actually watching this so volume hand control make sure you check that video out because if you don't uh view that video before this it will be a little bit confusing what we are doing okay so now let's divide our work first into some categories and then we are going to uh write the code for each one of these so first of all we are going to find our hand so we will write here find hand and then what we will do is we will filter it based on size so here we will go down and we will write here filter filter paste on size now this is because when we go back when when we are really far away when we are measuring the distance between the index finger and the thumb it will give you very small values so they will not be usable now the other thing that i thought about this was normalizing these values so normalizing the complete hand so that means that for example if the size of my hand is let's say 400 i will resize it to 500 and every time it doesn't matter what the size of the hand is i will always resize it to 500 and then based on that i will check the distance between my finger and thumb now the thing is that the size actually keeps changing so the width and the height they keep changing i even tried with a little bit of ratios but the results were not very smooth it does work so no matter if you are far away or you are close it will detect the same size and the values will be very much similar if you are very close or you are very far but the issue was that it was not very stable so i will not go ahead and do that instead we are going to filter it only based on size so if we have let's say the size of uh 300 then we do not use it it should be let's say between 400 and 500 so these are just random values that i'm referring to but later on we will see actual values how can we do that but to do this we need the bounding box of our hand now we did not do this in hand tracking and we did not do this in any of the other projects so we will look at how we can get the bounding box of this as well because the media pipe library as far as i know they are not supplying this information by default you have to add a little bit of code to actually find the bounding box of the hands so we will do that too then after we do the filtration then we are going to find the distance find the distance between the index and thumb so this we have already done but the thing that we will change is that we are going to convert all of this into a method in our class because we are doing a lot of different projects and every now and then we will need distance between two fingers it could be index or thumb it could be pinky and index it could be any different number of fingers so what we will do is we will create a method so that we just input two values and it will find the distance between these two fingers so that will be very easy to do okay and then after that we are going to convert our volume convert volume uh from the length to actual volume now here we are going to also make it smoother so we will reduce resolution to make smoother make it smoother so for example instead of incrementing with um one we can increment with five we can increment with ten so for example it will go from zero to five to ten to fifteen to twenty and so on so this way it will not change with a very small change in the distance it will be much smoother and much more usable okay then we are going to check which of the fingers are up check fingers up again we are going to create a method for this now if you haven't watched the virtual painter uh not the virtual painter the finger counter so in the finger counter project we had this um what do you call method of finding which fingers are up and in the virtual painter what we did was we created a method out of it in our class so if we go to our hand tracking module you will see that no this is the minimum where is the hand tracking this one okay so here you can see this is our method that will check which fingers are up and it will just return us a list of numbers so it will have five numbers and it will have five values and each value represents whether the finger is up or not so we are going to check that but why are we checking this because we will set the volume only when we close our finger uh the pinky finger so our pinky finger will be the one that will set the volume so we will change the volume with our index and thumb but we will set the volume with pinky because setting the volume is very hard because you will have to remove your hand instantly to keep the same volume so it is very hard to do that and there are other ways but this is one of the easiest ways and this is one of the fastest ways you can use so we are going to check which fingers are up and after that we are going to set we will check if pinky is down set volume so that's the idea and after that we will simply do all the drawings at the end so we will write here drawings so this is the main idea so we will go step by step and at the end we also have the frame rate frame so we are going to look at that so these are the steps that we are going to follow today and we will go one by one and see how we can achieve this now the first thing we will do is to find the hand so we have already done that now we have to filter based on size now filtration based on size is based on the bounding box but we don't have the information of the bounding box so what can we do the idea is that we will go to our hand tracking module and we will go to find position here we are getting all the different points of our hand so what we need to do is we need to find the maximum x and the minimum x and the maximum y and the minimum y of any of the points so what we will do is we will put all of these in a list and then we will check the maximum value and the minimum value of that list okay let me explain it a little bit further by just starting to code we will write here x list so this is the list that will contain all the values of x and this is the list that will contain all the values of y now we could do this with numpy as well with a little bit more efficient code but we are going to do with simple lists because it is easy and straightforward so we will put all the values so here whenever we have an x and y value we will write x list dot append and we will write x cx and then we will write y list append we are going to write cy so this is the idea we are appending all the values and once the for loop is done we will go down here once the for loop is done we are going to check the minimum and maximum values so here we will write x minimum and x maximum is equals to minimum of the x list and the max of the x list then the same thing we will do with our y and we will just make it y y y and y there you go so now this is giving us the x minimum x maximum y minimum and y maximum so this is basically our bounding box so we can write here bounding box is equals to x minimum then y minimum and then x max and then y max so and now we can also return the bounding box information so this is basically the idea so now if we go back to our volume control uh actually you know what let's keep it like this i will copy all of this and i will paste it into a new one a new file and we will call it uh volume hand control and advance advance okay oh did i put a space in between that's bad okay so now we will paste it here and we will do it with this rather than the volume control okay so here we are going to write lm list and we we are returning the bounding box information as well so bounding box we will write it like this and then inside here we are going to print the value of the bounding box so we'll print here bounding box value so let's try this out let's run the volume hand advanced okay so we are getting an error because uh we are not referencing it before wait what is it okay it's here how the problem is here because wait there is a problem here as well append uh other than that the problem is that we are we are sending it back but we are not declaring it so we need to write here bounding box is empty so let's run it again there you go so now if i bring in my hand you will see we are getting the bounding box information this is the bounding box information so let's draw it so we will do here that or we can put this this functionality here we can write that if if draw because we already have a draw flag and we are using it where here we are using it already so we can use it again if draw so we will write here cv2 dot rectangle and then we will send in our image we will write our bounding box and then we will write the color so it will be 0 five five and zero let's say and then the thickness is two so let's run this and see if it draws around our hand uh nothing why didn't it draw so it is checking draw is true or false oh it's false here that's why it's not drawing so let's make it true there you go so now you can see it is getting about something is wrong here what something is very very wrong i think the bounding box it it expects width and height and here we are giving the actual values so maybe that's the issue so instead of doing this let's write 0 and let's write one and then let's write all of this again this will be two and this will be three so let's try that yeah so that was the issue so here you can see we are getting our bounding box now and it is drawing these huge circles which look very weird we can reduce the value of these circles let's say five there you go so now we are getting our bounding box but you can see it is at the exact point so if you see my pinky finger it is at the exact same point so that might be a little bit inconvenient so what we can do is we can add some values here so we can say minus 20 and minus 20 and here we can say plus 20 and plus 20. so we can try that again and now you can see there is a distance so it's not exactly at the same point so it looks a little bit better so this is how we can get our bounding box and you can see it's quite fast and it's good okay so what we will need next um is the actual value let's just remove the print that we have earlier and we will go back here and now what we will do is we will take the the last two values and we will multiply them or no we need the width and the height so we need to subtract them and then we can multiply so what we will do is we will say that our bounding box so we basically need the area we can go up here and we can declare here area is equals to zero let's say and over here we are going to say that our width of our let's say box and our height of our box is equals to bounding box at uh first of all we will take the second value for the width so it will be 0 1 and 2. so two minus bounding box at zero and then the same thing bounding box at three minus bounding box at one so this will give us the width and the height um should we just multi let's just multiply it here area area is equals to this and we just multiply it here making sure that it is first calculated here and then multiplied okay so that will give us the area and we can simply print this area to see what kind of values do we get so there you go so now we are getting around if i make it smaller it's 20 000 something 50 31 okay let's let's go ahead and let's divide it by 100 or let's double divide so that it is a whole number so there you go so now you have 500 400 if i come closer so now we can decide when do we want to use these functions so here maybe this is way too far maybe this around 500 to 1000 maybe so these are the values that we should use if it is before that we should not use so this is what we can do here we can write that if the area is basically in between 500 and 1000 then we will do the rest of the things so everything here we will do uh except for the drawing we can draw except for the drawing we will do everything if the area is in between here so what we can do is we can write here print yes and we will print yes whenever it is in that range we are doing this just to check so there we go so if we are far away it will not say anything if we are close then it says yes and if you are too close it will say no yeah so this is this is a good range but here you can see uh we might need to reduce the value a little bit so let's say 350 because this is the issue when you are changing this uh when you're moving your fingers the size changes yeah so we will have to make it even smaller maybe so yeah here it's fine it's let's make it 250. yeah so if it's really far it will not work if it's close like this it's too close it will not work if it's here then it will work so this is the perfect spot that we want to use i think that is pretty good so this is our first part where we do the filtration based on size now we have to find the distance between the index and the thumb so we did this earlier using this method but what we can do is we can create this into a method within our class so we have this class where we have find hands we have the method find position and fingers up and now we are going to create a method called find distance so we will write here def find distance distance and we are going to give in the point number one then point number two then the image and then we have the draw is equals to true so these are the uh this is the flag so then the idea here is that we are going to get the values internally we are not going to send it out and then get back in this is just an id so it will be for example id number 10 id number eight id number four so we don't have to get the actual points the actual x and y position so how can we do this we will go to our volume control and we are going to cut all of this and we are going to go back to hand tracking and we will paste it here and here we are going to say that our self dot lm list is basically this and then we will just replace the self again and again so this is the basic idea and the rest should work fine there's no math because we didn't import it so we will write here import math and we can go down again and the rest is fine so here this is all for drawing so we can write here if draw so if we want to draw then we do all of this so that is the basic idea do we need to draw something else no okay so what we can do is now we can return these values so we can say here that return length and return the image on which we have drawn and then we can also return x and y so we can return all these value x1 y1 and then x2 in case we want to use them it's not compulsory but in case we want to use them we can write these x then y2 and then cx and then cy so we can return all these values and here in the volume hand control we can go here and here it says find distance between index and thumb so we can write here that our detector dot fine distance detector dot point distance and we want to find the distance between which index so actually we didn't uh define here so here instead of 4 it should be p1 and here also p1 and here it should be p2 and here it should be p2 l1 so yeah so then it was 4 and 8 so here we are going to write four and eight and then we will write our image and we are getting back a lot of different things so we are going to receive all of these and we will write length length gth length and then we are receiving the image then we are receiving the info so we can uh simply write that this is info should we write what kind of info we are getting let's say line info line info so that is good now let's try to print the length and see if we get it so let's remove that so yeah we are getting the length and we got an error as well so cx is not defined where is it not defined here uh ah okay because we are not uh earlier we had the cx and cy here now it is in line info so cx and cy is basically over here it's the zero one two three fourth and fifth element so we will get the fourth and fifth element and instead of cx and cy we will write here line info at four and line info at five so you can see it's already uh useful that we got the line in for as well so now you will see there is no error so that is quite good okay then the next step now that we have the length we have to convert the volume and this we did earlier uh over here so we are going to take all of this and we will cut it and we will go up and we will paste it here converts volume let's separate this a little bit so we can see uh now the thing is that we don't need to use the minimum and maximum volume because we know that it is a percentage now we are going to use the linear uh what you call scale rather than the logarithmic scale so in the previous video we just applied a linear scale onto the logarithmic scale and it does not convert well so in order to rectify that we are going to remove this and there is a function within this library that allows us to get linear values so we can write here that volume volume dot set set master volume level scaler i know it's a long name but that's the idea so so instead of writing this set master volume level we will write set master volume level scalar so that will allow us to send percentage values rather than just these converted values so what we will do is we will send our volume percentage and then again we will write none in front of that so if we remove this now and we run this you will see that uh let's open up the volume you will see that okay there's an error uh master volume scale the parameter is incorrect oh yeah so you have to divide it by hundred so this value is normalized between zero and one so you have to divide it by hundred okay so let's try that so now you can see it is much smoother than before so we can have zero we can have okay we need to fix the 100 value because we changed all this but now it is much smoother than before because we are able to easily move around and we can check it on our volume as well and now it is reflecting exactly the same as on the volume earlier when we were reaching at the end it will move it will start moving but now it is moving evenly everywhere so that is the idea so we can remove both of these but the idea is we are not going to set our volume volume now we will set our volume when we have the pinky finger up so we can put it here if pinky finger actually when it's down so we will set the volume so we can put it here later on we will add some code so the next step is to reduce the resolution now why do we want to reduce the resolution to make it smoother so if you are changing each and every single volume value so you are going from 58 to 59 to 60 to 61 then it's a little bit harder to set these values so what we will do is uh in windows i think the increment is of 2 yeah so you can see here the increment is of 2 so it goes from 40 to 42 to 46 44 then 46 and so on so what we can do is we can put a value of smoothness we can call it smooth ness is equals to let's say 10 and here we can write volume percentage is equals to smoothness smoothness multiplied by we are going to round the value and we will divide our volume percentage with smoothness so that will give us uh what you call smooth values with increments of 10. if we want increments of 2 like windows we can write a 2 if we don't want any increments uh just increment of 1 we can write it like this so let's try with 5 first so now you can see it is much smoother so you can see 40 45 55 60 65 and so on there you go so we can try it with 10. now it is way way way smoother it's very satisfying as well okay so i think we will keep it on this let's keep it at 10. then we are going to check which fingers are up so we have already created this function over here fingers up and as i mentioned before this was available in the finger counter we first wrote the code in finger counter and then we created this uh what you call method in the virtual painter project so it is very simple you don't have to go and see the whole video to understand this the idea here is that we have a list of fingers and it will contain five values each value can be zero or one so if it is one it means it is up if it's zero it means it is down then we what we do is we check if the points that is above the finger let's talk about the finger first if the point that is above the finger let's say the tip point it goes below the point that is two values before it if it does that it means it is down then we write uh this one is for up by the way so we are checking the opposite thing if it is up and then if it is not then we will put zero so this is the idea and here it is left and right so if you want to go into the details of this do check out the finger counter project so i have explained this in detail over there then it will return this finger list so what we can do is to understand this a little bit further we can write here fingers is equals to detector dot finger fingers up and we will print fingers so let's remove the length and let's print that so here you can see uh okay the thumb is wrong but uh yeah i remember i didn't change that uh if we go to fingers here and we reverse this it should work with the thumb as well again we are not using the thumb really so it doesn't matter in this case so what we care about is the fingers uh actually the pinky finger alone so here you can see it goes to zero here it is one zero and one i can do with the rest of them so all of them are zero one two three four and five so we can do that and what we need is the pinky finger only so whenever we put the pinky finger down we will set the volume so we will go here and we are going to check so here we are getting the fingers and that is good for us and that is good enough for us and now we are going to go and check if the finger pinky finger is down so here we will write if uh we will write not fingers at four this is the last element uh not fingers just means that if fingers is equals to false so you can also write it like this is equals to false is the same thing so if it is false then we are going to set our master volume so this is the idea and what we can also do is we can um we can change the color if you remember whenever the length was less than 50 we were changing the color to green let me show you here so if the length is less than 50 we change the color to green now instead of doing this we are going to change the color to green when we set the volume so it is an indication that you have already set the volume so we will remove this condition so here now if i put my pinky finger down you can see it turns green you can see that and now it will set that volume so here if i check my volume so right now if i change it will not change but right now it's 50 and if i put it down it becomes 50. if i put it at 70 or 60 it becomes that so if i go to 30 and do that you can see it changes so my hand is moving a little bit so let's say i put it at 40 and i do it goes to 40 30 goes to 30 then 20 then 10 if i keep it down and then change it will keep changing the value will keep changing why is it flickering that much ah okay so you can remove this flicker by going to the detector and you will see that the max max hands is equals to one so it will not try to find another hand there you go so now if we do that and if we rotate you can see uh the volume let me open the volume first so the volume will keep changing if my pinky finger is down but as soon as i put it up it will set the volume so this is idea so we go down and we set go up and reset so that is the basic idea so that works well you need a little bit of practice to actually get used to of it but it seems quite good now what we can do next is we can change the color uh and we can add some indications for the drawing by the way the frame rate we have already done so we can put the frame rate over here so we can remove it from here and we can write it here that this is for the frame rate or is it no this is from the frame rate this part this part is for the frame rate and we can remove all of this and this part here is for the drawings so drawings is up to us if we want to do it when we detect the hand we can push it in and it will only do when the hand is there or we can always do it whether the hand is there or not if we take it out of the if statement so we can keep it outside for now and what we can do is we can change this a little bit uh first of all we are going to add let's say another one of these so we will set our master volume so whenever we set the volume we are going to display uh or we can display it always so for example we want to check what is our volume so rather than going and checking on the windows we can write here that our current volume is equals to volume dot get master volume level level and scalar so this will give us the value we will multiply it by hundred and we will make it an integer or should be rounded let's round it uh no round will not work properly let's let's just do the integer okay so we will write here current volume and then we will write here current volume and we can write here uh volume set so this is the volume that has been set so we can write that and we can change the location of this to let's say 400 and 50. let's try that yeah that looks good so right now the volume is 69 and if we go here and we do this it changes to 40 we do this changes to 30 go up changes to 40 we go to 60 it changes to 60 and you can see uh one more thing we can do is we can change the color of this as well whenever it is set so it gives a better indication that the volume is set so to do this we will go up here and we will right here we will write here that our volume uh let's say color volume color volume is equals to initially it is blue so we will write blue and then whenever it is detected or whenever it is set we are going to change it to green so this is the case where it is set so we are going to put here and we are going to write 255 and we will write here zero and instead of using a fixed color over here we are going to use color volume so let's try this so right now it's blue i bring in my hand and if i set it you see it becomes green but then it doesn't go back so we need to write an else statement here we will write else and we will write color volume is equals to 2 5 5 0 and 0. so it will become blue again so there you go so if i do this you can see it sets the volume and it goes back so that is pretty good now if you want you can add a delay here you can write for example uh time time dot sleep so that it doesn't move around a lot i i don't like this but you can if you want to add it you can add it to so if i do this it will delay a little bit and it will set it a little bit better so while we are setting it we won't we will not move a lot so that's the idea but i don't like to put this here so this is basically the idea do we need to change anything else so here is check fingers if pinky up it fits down drawings and that is i think we are pretty much done so again you can play with the resolution you can play with the volume and uh you can play with these uh displays so you can change their colors you can change their size and when the volume is set and what not so this is quite good let's try it again so here my volume is 20 at this point if i do this it becomes 50. actually let me open up the actual volume as well so you can see it's actually changing so here it is let's say 70 or 80 and we do this it becomes 80. here it is 30 or 40 we change it to 40 then this is 60 changes to 60 30 changes to 30. there you go so it takes a little bit of practice but you can do it and now we can also see that it doesn't go to the max so this you can change this thing you can change uh by changing the size so here we can change instead of 300 we can put let's say 280 if we are facing some issues so let's try that okay it's still an issue so let's make it 250. so let's try 250. yeah it goes but it's still a little bit hard especially if you're at the back uh let's make it 200 so it is easier all the way around yeah so even if you are at the back or at the front you can go till 100. so again you can change this based on your area as well so now you have that flexibility so here you can see we are setting different volumes so that is the main idea so sometimes it gives one value less because it is rounding off so this basically means 90. so if we check here it will be 90. yeah so this is basically today's project i hope you have learned something new here uh it is quite satisfying to actually produce something that could be useful so if you like this video give it a thumbs up and share it with your friends and don't forget to comment if you don't understand anything and there's also a discord channel that you can go and if you have any questions you can put there as well and somebody will definitely help you out there if i'm not able to answer so this is it for today's video and i will see you in the next one

Channel: Murtaza's Workshop - Robotics and AI

Views: 28,460

Keywords:

Id: 9ZRqc4EaPRU

Length: 44min 56sec (2696 seconds)

Published: Thu Apr 29 2021