AI ROBOT ARM using Python Arduino OpenCV CVZone | Computer Vision

hey everyone welcome to my channel in this video we are going to create a robot arm and then add artificial intelligence to it we will first 3d print all the parts and then assemble the arm then we will use computer vision to track our hand and replicate that on the arm and all of this will be done with just 20 lines of python code that's right with our new cv zone library building robotics and computer vision applications has become very convenient if you would like to create real world computer vision apps do check out my premium course in which we learn how to create apps such as object detection augmented reality document scanner and a lot more the link is in the description below so without further ado let's get started so the robot arm we are using today is by the in move project this is the in move robot as you can see this is the website in move dot fr and this is designed by a french designer by the name gael lagawin so he designed this and he open source this for everyone to use so thanks to him and we can see that he has listed everything down to make it easy for everyone to use so here you can go to build yours build your index and here you can see finger starter hand arm and forearm bicep shoulder and so on so we will go to hand and forearm and here you are going to find all the steps to actually create this and you can also find all the 3d printed files within this website as well and if you don't want to go through all of this do check out his shop as well and there he's also selling this arm if you want to purchase that so at least he's selling the the hand rather than the arm i believe there was arm at some point as well i'm not sure but anyways you can get a cheap printer and you can print it yourself and nowadays printers are very very cheap and you can easily use them right out of the box so we are going to build this now i printed this almost one year back so i will show you all the process of how i did this and then later on we are going to look at the electronics and then we will go on to the coding part [Music] hmm [Music] [Music] [Applause] [Music] [Music] oh [Music] all [Music] [Music] so [Music] so [Music] [Music] so [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] so let's start by understanding the concept of our serial communication so the idea is fairly simple we are simply going to send zeros and ones based on which finger is open and which finger is closed so whenever a finger is open we are going to send one so as you can see here and whenever it is closed we are going to send zero so the string that we will send from our python code will be zero zero zero zero zero and then it will have a dollar sign in front of it so at the starting it will be a dollar sign and then you have these five digits corresponding to whether each finger is open or closed so if for example pinky is closed it will be zero if it's open it will be one now in our previous project the last time we did this so earlier we had the issue that if we open any finger we cannot tell which finger is open we just open one finger the index finger and that's it so in this in this scenario we are not going to do that for each individual case we are going to open and close so you can literally make any sign or anything with your hand and it will represent that exact same gesture with the with the robot arm so this is a kind of an upgrade and it is much faster than what we did before so this is basically the serial communication idea so what we will do is we will open up our arduino and here let me make it a little bit bigger so you can see better so here we are going to first of all save this so we will write here robot arm finger counter gesture whatever you want to write you can write here and we will save it now first of all we are going to include uh or let's do the serial part first then we can go on to the servo part so first of all we are going to write here integer values receive so whatever values we receive we are going to put them in an array and this array will be the number of values received so we are making it a variable so that if you wanted to use it in other projects you can do that too now in order to use a variable like this within an array uh in the definition we will have to define here define number of valves receive why is this v small we are using camel casing so we will use it like that okay so this is the idea so then we can put the value here that for example in this case we are receiving five values okay and then we also need to define one more thing which is the digits per value received now the good thing about this code is that you can use it later on as well for other projects so for example if you are receiving let's say a value from the analog sensor which has a range from 0 to 1000 then you can have four digits so here instead of one you can just write four and let's say you have three sensors connected you can write here we are receiving three values from three different sensors so it makes it very easy to work with different projects that's why we are writing it like this but here for now we are going to use five and one okay so once that is done we are going to define our string length so our string length basically what we are referring to is that dollar then we have zero zero zero zero zero so this is our string length which is 0 1 2 3 4 5. so how can we do that we can simply calculate it because we have the number of values receiving and we have the digits per value so we can simply calculate it so we can write here integer string length is equals to number of values received multiplied by digits per val value or val received plus one so here basically we are saying we multiply this with this and that will give us all these which will be five but then we also have a dollar sign so we will add plus one so this is what we are doing so far so very simple very easy let's put it here there you go then we are going to initialize a counter this counter will be used to actually see or iterate through all these different characters so we will be using that later on and then we will also initialize a flag which will be counter starts and we will keep it as false so whenever we receive the dollar sign this dollar sign we will start our counter so that's the idea and once we start the counter we are going to keep adding to our counter we will do plus plus and that will keep adding to our counter and we will know uh we have to go till this string length that's why we are using the string length as well okay then we will add a string this string will be received string so when we receive the string basically what we are doing is we are collecting it all together and then we will break it down into substrings and these substrings we are going to store in our value received so it might be a little bit confusing at this point but don't worry when we go ahead i will explain every step okay then in the void setup do we need to initialize anything for the cdl yes we need to write cdl.begin serial.begin and we have to write the baud rates 9600 and then what else then for now we don't need to do anything else here and then we can skip the loop for now later on we can use it here we are going to create a new function we do not uh intend to output anything or return anything so we will write void we will write here receive data uh i believe i wrote the spelling here wrong receive this ring okay did i write anything else wrong um not so far hopefully okay so then we will add our brackets to start and end and then we are going to write while while serial dot available away label then we are going to go ahead okay so we will write pretty much everything here inside this okay so what is the next step the next step is to read the character so the the data that is coming from the python script is not actually as a complete string it will come character by character so we need to read each character so we will write here character c is equals to serial dot read and we are going to read this character then we need to check if our character is dollar so this is the character we are looking for so we need to check here that if the character is equals to dollars then we are going to say counter did i put a bracket because we have a lot of brackets it can get confusing very easily so here we will write counter starts is equals to true so now this counter will start and then we need to check if counter start this basically means is equals to true we don't need to write that if we just leave it like that it means the same thing and then then we are going to check if we have reached our string length so we will write here if counter is less than our string length then we are going to count right so we will do the count plus plus and all of that actually let's put them together we can format that later okay so basically what we are saying is as long as we have not reached till the end keep counting one by one and each time we are going to get this string and we will add it to this string sorry this character we will add to this string okay the spelling here is wrong s t r i n g okay so what we will do is we will write here that our received string our received string is equals to string received string plus c so what does that mean that means that whatever our receive string was earlier so earlier it is nothing so it is empty so we will add our string to it our character to it that we received just now so it could be dollar sign uh it could be the 0 or 1 whatever it is we will add to it and then to make sure it is string we will write here string so that is the main idea and then we need to add to our counter so we will say counter plus plus so this will add to the counter and it will keep iterating so this is as long as this statement is not reached so then we can write here that if our counter is greater than or equal to string length then we are going to actually store our values in where is it in the values received so this we will store it in this array so here we are going to write so how do we actually put it in this array let me explain we can write here values received let's say at zero is equals to our received string received string dot substring so we can write substring and then this is basically a function or you can say a method that we can use to actually split our string so right now at this point our received string is basically this and what we need to do is we need to go ahead and split it one by one like this so we will give it some values so we are going to say that for example start from zero and go till 1. so we can say it like this and then what we will do is because these are numbers we can put dots to int because later on we are going to use these values to send the command to the servo so we need it in integer rather than string so we can write it like this but the problem here is that i will have to copy this and i will have to paste it uh five times because we have five values that we want to receive right so i will have to paste this five times and then i will just change this value to one two and so on and i will change these numbers to one and two and so on but there are two problems here one is that let's say i have more than five values then what then i will put more that is uh not a good way to do this that is kind of hard coding so what we can do is we can use a for loop so the for loop will loop as many times as we have the values and the second thing is we need to update these values so the problem here is that not every time we will have one digit per value so here we have one digit per value but later on we might have two we might have three so again we cannot hard code any values here so we will use the digits per value here as well so we can remove all of this and we can write here that for integer i is equals to zero we are going to write i is less than number number of values received and then we will do i plus plus so while it is less than this value keep adding to our loop and then what we will do is we are going to create here a number so we are going to call this integer number is equals to i multiplied by digit per digits per value received plus one so this is basically our number so the plus one is basically because we have our first digit as dollar so we don't want to use that so we are writing plus one here then we are multiplying the digits per value with i so for example if we have uh the digit one then it will multiply this with one if we have digit zero it will multiply it with zero it will become zero so in our first case when it should be 0 this value should be 0 this value should be 1 right so what can we do so here we can simply write this number number and then we can write digits per value we can write here number plus digits per value okay so this might be a little bit confusing uh let me explain it let me zoom out a little bit okay let me explain what what's happening so if i want my first if i want my very first string i will write here one and two if i want the second one i will write here two and three because the first one is the the very first one is the dollar so actually i pasted here as well so the very first one is the dollar so we don't need that so we are starting from one instead of zero so we are saying one two two so this is basically the idea so if we look at this equation this will satisfy these conditions so here you can see that we have number so number here if it is 0 then zero multiplied by digits per value it will be zero plus one it will become one so this is one then the second one is it's already one plus digits per value is one so it will become two then if we go to the second iteration this will become 1 so here this will be 1 multiplied by 1 which is 1 plus 1 so this will become 2 so here we are putting 2 and then 2 plus one which is digits per value becomes three now it is very easy to do plus one each time because in this particular case it is just digits per value is one but if we had more values for digits per value then this will become very easy to use so this is why we have written it like this i hope that was easy to understand and then we can simply replace this with i so hopefully we have not made any mistakes so far so this will store all these values in our values received and then we can use this value received for um sending the commands to the server but then we need to refresh or reset our values for the second time it sends the string so we need to write here that our received string is equals to empty and then we are going to say counter is equals to zero and then we will say counter what was it start counter start is equals to false so this will basically reset all the settings so that next time we can do all of this again okay let's actually compile this because probably there will be some mistakes okay so c is equals to dollar you have to put it like this what else okay so that was the only mistake hopefully okay so what else can we do now what we will do is once we get this value we have to uh what do you call send it to the server so we will go up and we are going to write here that we want to include include servo dot h and then we are going to define or do we need anything else okay we don't need anything else here then we are going to create objects so basically we have our servo class and from this class we are creating objects so each server will have its own object so we are going to write servo thumb then let's copy this because we are lazy what happened there too many spaces okay then here we are going to write thumb and then index and then middle and then what is it ring then pinky okay so this is basically our initialization then what do we need next um we need to define all of these in our void setup so we need to define which pin number is connected to which motor so what i have done i have connected it in a very weird way i just randomly connected it and then i tested each of these servos uh which one was connected right based on trial and error so for the thumb we will write attach and it is attached to pin number seven then we have let's paste that again it is pasting very weird let's remove that there you go actually we can go to tools and auto formats that will auto format everything actually it's not looking bad okay then we have 9 11 and 8 and 10. okay and then we have index index middle then we have ring and we have pinky okay that is good so let's save that and let's move on now the the last part is very very easy so all we have to do if we if we did not make any mistakes all we have to do is we have to first receive the data so we will write here we will call our function receive data and after we receive the data we are going to send the commands to our motors so we will write if the values received at number 0 is equals to 1 which means open it up then we will write servo thumb dot write we are going to send the command as 180 so we will rotate in one direction else else we are going to write servo thumb dot write zero i'm writing it in one line because it's easier to read that way you will see now we can just simply copy this and we can paste it multiple times and yeah so that makes it very easy to read what is happening you could write it in separate lines as well so here i can just replace this with one two three and four and then we need to replace these with our other values so we will write here index i should be capital then we are going to write here index and then we will write here middle servo middle right here middle ring ring and then pinky okay so that actually looks good to me let's compile and see if we made any mistakes of course t h umb what else oh it's working that's great so hopefully it will work now so before we go into our what do you call python script we are going to test it here so how can you test it here basically you can open the serial monitor up and as you can see uh you can hear the voice already what you can do is you can write here the signal and then you can send this command and see if it works so what i'm going to do is i'm going to record the hand and record this at the same time so you can see how exactly does it behave so let's try this out i'm going to add 0 for all of them and see if anything happens so nothing so far then we are going to write dollar then we will put all of them on so we'll put five ones and there you go so all of them open up and then we are going to try one of them to close so what we will do is we will put zero one one one one so there you go then we can put we can put let's say one zero one one one there you go so now you can see that when i send the command it actually behaves as we expect it to so there you go so all of them off and then we can send all of them on so this is basically the idea now what we can do is we can go on to our python script and we can send this string from python instead of writing it manually so here we are in our pycharm project and what we will do is we will create the code to actually send the command to our arduino board now the commands will be sent based on the gesture part so we will also look at hand gestures so we will look at how many fingers are up at any given point and based on that we will send the commands to the arduino board so what we can do is we will be using media pipe liability to actually track the gestures but the thing is that we have created a module out of this and we have packaged it together in one simple what you call package and you can install that by going to the project and you can add and here you can write cv zone and you can simply install this so this includes the serial communication part as well and it also includes the hand tracking part so all we have to do is we have to call these methods and then we can make use of them and this will install the opencv and the media pipe library by default but it will not install the serial part so what we have to do is we have to search for serial now the serial by default did not work for me so i tried serial device and that worked fine so now both the packages have been installed and you can see that we are in a python file called gesture control arm so we are going to import here cv zone and then we will import cv2 then we will write cap is equals to cv2 dot video capture and we are going to say that our id number is one so you will probably use zero and then we will write cap dot sets and we are going to set it or do we need we can skip this if we don't want to give it any size by default the camera i'm using is 640 by 480 so that should be fine then we need to write the while loop so while true we are going to say the success and the image is equals to cap dot read so this will read the frame and then we will simply write cv2 dot i am show and we are going to show the image a mage and then we will say img and what else cv2 dot weight key weight key and we will put one so this should run smoothly let's try it out there you go so as you can see uh here is my hand and you can see that it is working fine so now we will add the hand tracking part so this functionality is provided by mediapipe library uh now mediapipe library is already installed because we installed cv zone and within cv zone we have a module that makes it easier for us to get the values from the media pipe library so it's like a wrapper so here we can write detector is equals to cv zone dots hand detector and if you want you can give in some parameters like the maximum number of hands so actually we should give the maximum number of hands as one and the detection value you can decide the detection confidence can be anything by default it's 0.5 so then we are going to find our hands and we can simply use this object that we just created we can write here detector dot find hands and we will give in our image and it will return us with the image with the output drawn so we can grab that then we are going to find the position of our hand by position we mean the landmarks so we will write here detector dot find positions and then we are going to give in our image and we will also take in return we will take the lm list and we will also take the bounding box so that should give us some detail and then again we can print this out so let's try it so here is the image and if i put my hand in you can see that we have some tracking and we can see that we also have the bounding box next we are going to check how many fingers are up and again this is very easy with our package so we can simply write if lm list so if there is something available then we are going to say that detector dot fingers up this will tell us how many fingers are up and we can simply write here fingers so that we store it in the list fingers so let's print this out we will write here fingers there you go so let's run that so right now you can see all of the fingers are up and if i put all of them down you get zero then one two three four and five so all of them are working fine so and this the best part here is that this is the exact same format that we used in arduino so all we have to do is we have to send this fingers list to our arduino and that should work by default seamlessly so how can we do that to do this we have the serial module within the cv zone package so it allows us to communicate with arduino so in order to use this the first thing we will do is to create a serial object we will call it my cdl and we will write here that cv zone dot serial object and now we have to given the configurations so as we saw earlier we are using com3 the port number and then we have to given the baud rate we select it as 9600 and then we have to decide how many digits are we sending per value so in our case we are only sending one and zero so it needs only one digit to tell whether it's one or zero so we will write here one so if you're doing something else you have another project uh definitely you will have to check how many values you are sending and based on that you can change this value so once we declare this now all we have to do is we have to send the data so we will write here my serial dot send data and here we are going to send our data now this is the easiest part where we just write fingers and it should work by default so all we are doing is we are getting the values from our detector and we are simply inputting it to our send data for our serial object and let's see if it works so we are going to remove this and what happens is that if it connects properly it will first tell us whether it connected or not so here i have com3 so if i run this now it should not connect because i did not connect it yet so it says here device not connected so now i'm going to plug in the robot arm and let's see if it detects or not okay so now the device is connected and let's run it again so now as you can hear there is some movement so let's try it out here you can see i'm recording so let's see how it works there you go so now the command is sent and you can see here it says device connected so it means it is connected properly so when i bring my hand it goes up now if i put one of them down it goes down two three four and five and then one two three four and five and this is pretty amazing this spider-man let's try victory this is the victory sign so as you saw that it is quite complicated task but you can see that it is very easy to do with these two libraries that we have imported and of course at the back end we are using the media pipe library that is provided by google so all of the heavy lifting is done by the media pipe library so this is it for today's video i hope you have learned something new if you like the video give it a thumbs up don't forget to share and subscribe and do check out our new platform the cv zone and i will see you in the next one

Channel: Murtaza's Workshop - Robotics and AI

Views: 473,136

Rating: 4.9521661 out of 5

Keywords:

Id: 7KV5489rL3c

Length: 46min 37sec (2797 seconds)

Published: Tue May 11 2021