Gesture Recognition & Tracking with ESP32 Camera & OpenCV || Make Gesture Controlled Mouse

hi everyone welcome to how to electronics this is a video again about the esp32 cab module and image processing using python and opencv this time we'll learn about the method to detect hands and fingers using python and esp32 camera module later using this technology we will develop a gesture control virtual mouse the sp 32 camera module along with a python program can be used to control the mouse tracking and clicking operations wirelessly to get started one must have sound knowledge of python image processing embedded systems as well as internet of things first we will understand how to control the mouse tracking and clicking and we'll also see what all requirements are needed to run the python program we'll first test the whole python script with a webcam or internal camera of laptop in the second part we'll use an esp32 cam module and run the python code so the esp32 cam will be used as an input device instead of pc camera or any other external camera so without wasting any further more time let's get started do you want professional pcb like this one that looks so good then use the service of next pcb you can select the board size any solder color marks that you want including something like red and green you can select the pcb thickness and the pcb could be from 2 to 32 layers for some more complex design the finish quality is so good and if you want better connectivity you could also select some gold finish for the pads the ordering process is so easy just go to nextpcb.com then quite now insert your design setting upload your gradle file and order now and receive the pcb in couple of days welcome back again this is an esp32 best camera module developed by ai thinker the controller is based on 32-bit cpu and has a combined wi-fi plus bluetooth or ble chip this is the ob 2640 camera for photography video or any other image processing application the board also supports an sd card up to 4gb which can be used to store images and video the board doesn't have an onboard programmer so in order to program this board you can use any type of usb to ttl converter module please follow this connection diagram for programming and use the programming method and different modes of operation has been explained in earlier videos let's move back to the project part let's begin with a simple demo using the webcam of your computer in order to get started we need to have python 3.8.7 library after that a system needs to have a webcam or a built-in laptop camera these are only the basics for testing to find out whether our python code is working or not apart from this we need esp32 cam wi-fi module then arduino ide installed on your pc a local wi-fi network is required so that the esp32 cam can communicate so let's set up the software partner first we need to install python version 3.8.7 so go to this official website of python and from here download the windows x86 executable file when installing make sure to tick the option a to path once all the installation completes go to the code the code is available in our website article now to use this code make sure to create a folder on your computer and save this python code as a track hand file as you can see there are few libraries in this code like opencv another one is media pipe the time and math library is something that comes in built with python after that we are also using numpy in order to install this simply open command prompt and write pip install opencv dash python as this is already installed in my system in case if you have an install you can install it similarly we can also do for the media pipe for that type pip install media pipe so again all the requirements are satisfied and it's already installed in my system then we also need to install numpy for that type pip install numpy now something to note down when installing the python or any other libraries you can create a virtual environment for that go to command prompt and type peep install virtual env if someone has already installed it and it is not working then try to uninstall using this command and then again install it after that we just need to write this code which is virtual enb slash path and the location where the virtual environments need to be created apart from this there is also an autopilot library which needs to be installed to do that type pip install auto pi in my case it's already downloaded so all the libraries has been installed successfully moving to the project part this is just simple hand tracking code through this we'll try to understand how things are working so what we have done is we have created a class name as hand detector in this class we have a method to find the hand this will try to detect where the hands are using the media pipe library so the media pipe has some objects which will try to detect some hands and finger first we should try to detect where the hand is then we will find where the position is here we have a method to determine where the fingers are and here we can also calculate the distance between fingers now let's run the code and see whether it works or not this might take some time when you run it first so it has run successfully so this is my hand which is perfectly detected by the camera so let's understand how we will be doing our whole processing this is my index finger so i'll be moving my index finger to move the cursor so the cursor moves as my finger moves see this moving finger here now if i wanna click my middle finger should be going down like this this way i will be clicking things so let's get started with testing now in order to test this we will use another code which is available in website article in this code we'll import the cm track hand file here we need to do special modification according to the system requirements because my webcam is 1280 cross 720 pixels which i need to change apart from this everything needs to be kept similar now in case if you are facing error your webcam is not able to capture change this number from zero to one the number zero is for internal camera and one is for the external camera so let's run this code again so here you see the image window with the dimension 1280 cross 720 pixels moving the finger in any direction will move the cursor as well slide it left or slide it right or maybe slide it up and down let me try and minimize this video for that move the cursor on the minimize icon and use your both finger to click on it so it's done now i'll try to close this done again now let's reopen it so it has been reopened again this way you can have full control now i can also fully screen this so everything looks uh working so far this is my other hand and i'm not using any mouse right now all right you can try this sample program as well i believe you might have understood this to move further will require an esp32 device this is the esp32 camera module that i would be using instead of computer camera the connection and the metro to program has been explained in the article already so you can follow it now go to the arduino ide and make sure you have installed esp32 cam library from here go to the wi-fi cam so in this code replace the wi-fi ssid and password from here with the same wifi network that your pc is connected to so moving to the code part we are running on two servers that are the high gpg on low jpg jpg is image format connect your esp32 cam to the pc go to tools select esp32 developer module and then select com port then upload the code before uploading connect the io zero jumper to the ground and put the device in programming mode once uploaded remove the io zero jumper and press the reset button it should show the camera ok message it means the device is fine now here you can see the ip address of the es broadcasted by esp32 cam module copy this url and paste it to any of the web browser let's change to the high jpg hit enter so the camera has detected my hand if i replace it the camera will give another frame so the sp32 cam is working and is ready to transmit data now copy the ip address and we'll do some modifications to the code again i have given the modified code in the website articles hence you need to copy the final code in this section modify the wcam and scan variable according to arduino code so it is here as 800 by 600 similarly i'll change the frame rate to 30. now let's run the code so here it is the image is processed from sp32 camera and the device is perfectly able to detect handy finger the coordinates changes as per the hand gestures and i am able to minimize the window or move the cursor left right or use this gesture feature as virtual mouse this is really amazing that you can use the image recognition feature using sp32 cam and opencv all the information related to the project can be found on website article of how to electronics the esp32 cam library python code and all other details are found here you can read the article if something is not clear from the video you can also comment down in the video section below thank you so much for watching see you in the next episode

Channel: How To Electronics

Views: 32,294

Keywords: gesture recognition opencv python, gesture recognition using mediapipe, gesture recognition technology, esp32 gesture recognition, esp32 cam gesture recognition, esp32 cam opencv, gesture training, gesture controlled mouse using opencv, gesture controlled mouse, esp32 camera, esp32 cam projects, esp32 cam arduino, esp32 cam python, esp32 camera module project

Id: xjuTEowOWvo

Length: 13min 0sec (780 seconds)

Published: Tue Feb 08 2022