AI Virtual Painter | OpenCV Python | Computer Vision

hey everyone welcome to my channel in this video we are going to create a virtual painter using ai we will first track our hand and get its landmarks and then use the points to draw on the screen we will use two fingers for selection and one finger for drawing and the best part is that all of this will be done in real time if you would like to create real world computer vision apps do check out my premium course in which we learn how to create apps such as object detection augmented reality document scanner and a lot more the link is in the description below so without further ado let's get started [Applause] so the first thing we will do here is to go to canva.com and if you're not familiar with canva it is basically a design tool it's an online website that helps you create all these different designs for brochures business cards flyers whatever you want you can basically design here so this is a free website so you can go and start you can start by just signing up so here we have a canvas of 1280x720 so we are expecting our webcam to be of this size that's why we are using this size so the idea here is that we are going to create a design to actually make it look more appealing and make it look more like a software so this is up to you if you want to skip this step you can do that too there will be files to download from my website the ones that i already did so if you want to skip you can go ahead and do that too but here we are going to start off with a rectangle and what we will do is we can we can put it on the side but the issue with the side is that it will be hard to select different elements so what we will do is we will put the menu on the top so our menu will be here at the top now for the width let's keep it at 100 100 is too small maybe 120 125 yeah that looks good so then we can lock this or let's change the color first so i like this blue or maybe this one yeah this one looks better so let's lock this so it doesn't move around then we are going to add let's add the logo first so i'm going to go to photos and right here logo no it's in upload my bad so i have my logo here i'm going to place it on the side and yeah that looks about right then we are going to add some brushes so here we will go to the elements and we will search for brushes so let's write here brush uh okay so these are strokes but we are not getting actual brushes here we have let's write paint brush maybe yeah i think now we have better results uh i want to look for something that is free this is free okay so maybe something else that is free this is free as well so i'm looking for something free so that everyone can use it not just the pro users um [Music] maybe this one no it's pro this one is free too do we have a color up i think this one is the best okay let's keep it this way and if we want to we can change it later on so let's take this part right about here and we will zoom in and maybe a little bit smaller there you go so i think that looks good now the good thing is that most probably this is an svg file so we can change the color for it so let's say we want pink what is that purple let's try pink and what we can do is we can have another one that is a little bit bigger than this at its back so that will indicate that this has been clicked so we can put it okay let's change the color first we can put it as white or gray or dark gray let's put it at the back um i think white will be better okay i'm no designer i'm just eyeballing this so i'm gonna put it as whites yeah okay so then what we can do is we can copy both of these and we can paste them here so somewhere in the middle and we can paste another one and then we need an eraser so we don't need the last one actually the one at the back or do we okay let's keep it let's keep it for now i will tell you why later on okay so now we need an eraser so let's write here eraser okay this one is pro pro pro this one is free no not really all of these are paid oh this one is free yeah i think that could work so let's put that a little bit bigger i think i'm putting it too big anyways you can change the design later on uh we are learning the concept here or let's just make it a little bit smaller so i think that will be better so we can grab these two and we can place them here this one in the middle this one a little bit further okay i think that is better and then we can make this a little bit smaller and there you go so i think that looks good now what we can do okay these are not at the top to put it at the top here okay so now what we will do is we will change the color for these so the first one is let's say pink the second one let's say is blue or dark blue in between maybe this looks weird okay let's keep this as blue and then the last one let's keep it as green so these are the let's say three colors that we want and then what we will do is we will this doesn't seem right i think at the back needs to go higher maybe yeah maybe like this okay i'm going to copy this and we will paste it here so we'll put it at back and the same thing to the last one okay so you can move it around with your keys your arrow keys so maybe a little bit higher again you can spend a lot of time on this but we are going to skip that we will not put a lot of effort here just for a demonstration purpose but you can of course go ahead and try a lot of different things so um now what we will do is we will copy all of these we will duplicate them so here you can see or let's change the size first so we already know that our size is 1280 by 125 so we will go to resize and we will change this to 125 and we will copy and resize or you can resize is up to you i'm going to press resize so it will do it on the same one then i can unlock this and i can grab all of this and i will scale it up so it should fit perfectly because it is the same size there you go so this is the image that we need so we are going to copy this four times because we will have a selection for each one of these so we can delete the selection for this and the selection for this and this is the first one so first one this is the one selected the second one this is selected and for the third one we will have this one selected and the fourth one we will have the rubber selected so here we are going to it is whites let's change this to let's change this one to the same color as this and let's make it a little darker like let's make all of them darker so yeah and then let's make this it's a black hmm that's too much okay i think that is enough indication so we just need an indication that that has been pressed so that should be fine so now what we can do is we can download all these four images and the idea here is that whenever we detect a click we are going to uh change these images so we will see which one do we need so if we have clicked the first one then we will change the image to this if the second one is clicked we will change the image to this then the third one and the fourth one which is the eraser so at any given point only one of them can be selected so that's the idea so we can go and we can write here that this is our virtual painter let's say and we can download all of these in jpg or png whatever format you want let's do it jpg and we can download all of these so here i am in the pycharm project now this is the same project that we have been using for the hand tracking for the finger counter for the hand uh what you call volume control so we have done quite a bit of projects earlier than this so this is the exact same uh what do you call project and you can go to file settings and you can see that we have already installed our media pipe and we have already installed opencv but if you are new you can go to the add button you can write here opencv dash python and you can install this and then you can go to mediapipe mediapipe is the google library that we will be using for hand tracking so you can download and install this so this is the main idea now what we will do is first of all we will right click this and we will open it up in the explorer now once we are in our project we are going to create a new folder and we are going to call it header so we will have our header images in here so these are our images that we downloaded so i will copy all of this and i will paste them here and now you can see that these are our header files so we have a total of four header files so when we go back to our pycharm project you will see that we have a new folder by the name header and we are going to use that for our images so then we are going to right click and we will create a new python file and we will call this virtual painter so now we are going to import our libraries so we will write here import cv2 import numpy as np and then we will need imports time if we want to show the frame rate and then we will also need imports os because we need to access these files so we will need that what else we also need our hand module so we will write here import hand tracking module as htm so this will be our hand tracking module now if you are not familiar with this if you haven't watched the previous uh tutorial then make sure you go ahead and look at that because this is the tutorial in which we went step by step and we created this hand tracking module so this is very important this basically tracks the hand so if i run this you will see if i right click here this should open up my webcam and if i bring in my hand you can see it tracks the hand so this is the idea of the hand tracking module and based on these values we are going to do some painting okay so now first of all we are going to import our images so that is the first thing we are going to do we will write here that our folder path is equals to header so these are the header images and we are going to say that my list is equals to os dot list directory list directory and then we will get the name of all our folder path files so this is the idea if we print this out if we print my list you will see that we get all these uh images name so here you can see one dot jpg 2 3 and 4. so what we have to do is we have to import them so that we can use them later on to overlay on the top so here we are going to write for image path in my list my list we are going to loop and we are going to import so we will see that our image is equals to cv2.im read and we are going to read from a string and that will be the folder path slash the folder the image path so this will be our complete path that we need to read from and then we are going to store it in a list so let's call this list overlay list so this will have all the images that we want to overlay so we will write here overlay dot append and we want to append our image okay so this will overlay our images no it will import all our images now what we can do is we can print the length of our overlay list so we can see that whether we have imported all of them correctly or not so let's run that so we have imported indeed four so that is good so far and next what can we do next next we can run our webcam and then once we have our webcam we can simply overlay one of these images by default so we can call our image let's say header and we are going to give it an initial value we will say that over list at zero so now this is our image so whenever we get our original image we are going to overlay this on top of that okay so this is good for now what we can do next is we can create our loop and we can run our webcam so here we are going to write cap is equals to cv2 dot video capture video capture and we will say that device number one now you should write device number zero because i have multiple cameras i'm using one then i will write cap dot set now this is important because we want the exact same size we want one two eight zero by seven twenty so we have to make sure that the width and the height are exactly the same so we are going to write one two eight zero by seven twenty then we are going to write here while true we want to run our webcam and we want to get our images so we will write here that the success and the image is equals to cap dot read and then we are going to um okay let's just display it first so cb2 dot weight key is 1 and we have cv2 dot i am show i will say image and image so that should be enough so let's run it we have an error oh i wrote i am read no no no i am show what happened there okay so there you go so this is our image and you can see my hand so what we will do we are going to overlay our image now so how can we do that so we can simply write that our image so you might think it's a little bit difficult but it's not it's very easy because our image is a matrix we just need to define where is the location of this new image so we will slice it so here we are going to say that our height is from 0 to 125 because we know that the image size is 125 and then the width we are going to say is from 0 to 1280 so at this region we are going to say our image is equals to header and that's how simple it is so if we run this there we go so now you can see that we have our image and on top of that we have overlaid our first image now this is good because by default we want our first paint brush selected which is the pink one so we are already up to a good start so here we can write that we are setting the header image setting the header image okay so this is good and now we can just separate this because we are going to write our code over here so what else can we do now the next step is in before we go into the details of the project let's split it up into pieces so first of all we will import the image okay so that's the first step we have pretty much already done that there is another thing we need to do i will tell you later this is the first step let's write it here one then the second step is that we want to find the landmarks so find hand landmarks so this can be easily done with our hand tracking module so that should not be an issue then number three is basically checking which fingers are up so check which fingers are up now this is because we want to draw when one finger is up which is the index finger and we want to select only when two fingers are up this will allow us to easily move around the canvas without painting so when our two fingers are up it will not draw anything if we want to draw we have to put one finger up the index finger so this will allow us to easily navigate through the canvas then we have the selection mode so we will check here at the fourth stage we will say if selection mode which is when two fingers are up to fingers are up then we have to uh what do you call we have to select not draw then in the fifth one we have to check if we have the drawing mode mode when index finger is up so this is the idea now let's go on to the next parts actually do we have a next part no so this is pretty much it we have five different steps we are going to go step by step and see how we can achieve each one of these so the first thing is that for importing image it is pretty much done the only thing we have to do is we have to flip the image so we are going to flip horizontally this is because when you are drawing if you want to draw on the right side then when you move to the left it will draw on the right so to flip the image we are going to right here flip and we will write image and we want to flip in the first selection now this will allow us to solve that problem so let's try that so now the image is flipped so if i go on the right side it's going to the right if i go on the left it's going to the left so it will be easier to draw because it will be more intuitive now the second step here is to find the landmarks now the landmarks we can find but we need to import first so we are going to write here that our detector is equals to hand tracking module dots hand detector and then we are going to give in uh detection confidence so this is up to us what value do we want to use but we are going to keep a high confidence because we want it to be uh good in painting we don't want a lot of mistakes here and there so by default it's 0.5 we have changed it to 0.85 then over here we are going to say that our image is equals to detector dot find hands and we are going to send in our image so this will draw on our image and it will detect the hand so let's try that there you go so now we can see it's detecting properly so that is good the next step is to get all the landmark positions so we are going to write here landmark list is equals to detector dot find position and we will write image and we do not want to draw so draw is equals to false and then we can check if the length of our landmark list is not equals to zero not equals to zero then we are going to do something let's print the lm list so let's try that so let's see if it prints there you go so it's printing the landmarks if i move around if i go out of the image it does not print anything if i come back in it prints the what you call landmarks so that is good so now that we have this we need to know the the tip points of our index finger and the middle finger so what we will do is we will call the index finger tip as x1 and y1 so we will say x1 and y1 is equals to lm list at point number eight so this is the landmark eight so here you can see this is the landmark eight and we have the value of 729 and 396 for example in this case so this is the tip of our index finger so here i can write tip tip of index and middle finger okay so then uh as you can see it is basically the first and the second element not the zero one so here we have to write from 1 till the end so we will write it like this so it will grab only these two so we are unpacking them here so x1 and y1 so we will do the same thing here for the middle finger and for the middle finger we will write x2 and y2 and the number for middle finger is 12 so that should be good so now that we have our landmarks and everything is good now we need to check which finger is up so this thing we did in our one of our previous videos and that was i believe in the finger counting project so if you remember here if we go back here so here is the code that when it is thumb it will check if it is up or down then it will check the rest of the four fingers if they are up or down so and then it will give us the total finger count and it will also give us the list where is the list yeah the fingers is basically the list which will tell us which of these are up and which of these are down so what we can do is we can pretty much copy paste this part and we can create a method in our hand tracking module so earlier we did not do it as a method in our class but this time around because we are using it again and again it is better to create a class rather than putting it in uh different projects every time so this is our hand tracking module we are going to open that up and you can see right now we have only two methods find hands and we have find position so next we are going to put another method and that will be called fingers up so we will write here def fingers up so that will tell us which of the fingers are up so we will go to our finger counter project and from here we can copy so we have from fingers till this part i think that should be enough so we will copy that and here we are going to paste it so what we are doing is we are first creating a list called fingers and then we are basically checking if the tip of our thumb is on the right or on the left so that will tell us if it's open or closed and then for the fingers we are checking if the tip of the finger is above the other landmark which is two steps below it or it is not so if it is below that then it means it is closed if it's above that it means it is open so it is storing the value of 1 when it is open it is storing the value of 0 when it is closed but why are these errors over here these errors are here because it does not recognize tip ids and tip ids was something that we declared over here so this is the list so we can copy that and in our module because we know that it will not going to change the tip numbers the tip ids they are not going to change we are going to write it in our initialization so we will write here self dot tip ids is equals to this so now if we go back and instead of just using tips we use self.tips it should work so we will do that and there you go so that works but now the lm list it doesn't uh recognize it because we did not define it as a self as an instance so we need to write here self dot lm list and i can copy that self again and we can paste it here we can paste it here so we can paste it here we can paste it here and here and here so what happened now was that not only we are returning this lm list when we find the positions we are also storing it so that if we want it in our other functions or our other methods we can use it as well so we do not need to send it out and then receive it back again internally it is available for us to use okay so that should be good and if we just return these fingers up it should give us if the fingers are up or not which of them are up or not so we can just return the fingers and if we go back to our let's close the finger counter if we go to our virtual painter here we are going to ask we will say that detector dot fingers up fingers up and do we need to send something in no not really so that is good we don't need to send anything and we can simply receive the values in the variable or the list fingers so let's print that out so fingers and let's remove the previous prints so let's run this and see what happens so something is out oh it needs to be inside this my bad okay so this is our image i will bring in my hand oh nothing is happening why is that okay so it takes a little bit to detect now you can see uh how many fingers are there uh okay the issue is i will tell you what the issue is for the thumb it's showing one when it is open uh when it is closed and zero when it's open because we are not checking the left and right so this is an issue that needs to be solved more precisely but for now we'll just change this what was it before it was greater than less than so by writing less than it will solve the issue for now but later on we do have to fix this so i'm not going to fix it now for this one but in this case you can only use your right hand to draw so if you use your left hand it will be an issue actually it won't be an issue because we are not using the thumb at all but it will give you wrong values so here you can see the thumb uh you can see two of them then three then four so you can see all of these fingers they are being detected properly so now if i do this i should be able to select any of these and then if i do if i do this i should be able to draw so this is the main idea so we need to check uh now we know that which fingers are up now we need to check if it's selection mode or it is drawing mode so how can we do that that is very simple let me bring that in because i will forget again okay so now we need to check if uh the second oh no the first elements so we have fingers at first and fingers at second if both of them are up this means if both of them are true then it means it is selection mode so we will print here say selection mode and if that is not the case we will write here then we will write here that this is equals to false we only have the index finger up then we will say drawing mode so this is the idea now what we do after that is a different story but for now we just want to check if it is able to understand this or not so let's try that we should remove the previous print let's remove that let's run this so this is selection mode because two fingers are up and now it's drawing mode so now it's working very well we can see when it is so when it is nothing it doesn't do anything when for detection you have to make sure a lot of the image or a lot of the hand is available in the image so yeah so this is drawing mode we can draw this is the selection mode and everything is good so far okay so the next step what we can do is we can uh change or we can draw some circle around it so here if it is selection mode let's draw a rectangle so we are going to write here cv2 dot rectangle and we are going to write their image and then we are going to write that we want from x1 till y1 minus let's say 15 we want to go above because we are creating uh let's say a rectangle using two points so we don't want to just give those points we want to make it a little bit higher and a little bit lower so then we are going to use x2 and y2 and this time we will make it plus 15. so then we will give it a color so let's say two five five zero two five five this is purple and then we will say cb2 dot filled now we are drawing a rectangle here because we are going to draw a circle when it is time to draw so for the circle we are simply going to write here cb2 dot circle and we are going to give in our image and we are going to draw from x1 and y1 so we will write here x1 and y1 and then we are going to say 15 and the draw color again 2 5 5 0 2 5 5 and then we are going to write cb2 dot filled so this will be a visual indication of when we are in selection mode and when we are in drawing mode so for selection it will be rectangle for drawing it will be circle so let's try that out so here it is rectangle and then if we do this it becomes circle so here the detection is not that good but still you can see the rectangle is very small 15 is not a good value let's make it 25 and 25. yeah now it is better so here i can say the selection mode and here i can say it is drawing mode so it will be easy for us to detect so my hand should be really back and then i should be able to draw easily but because i'm near to the computer so it's a little bit difficult to do it this way but it should work fine so far we have done well okay so we have the selection mode now and we have the drawing mode so let's work with the selection mode first what we have to do now is we have to check if we are at the top of the image now if we are at the top of the image we are going to change our uh image or selection mode based on the location so first of all we will check if y1 is less than 125 so this was the value of our header so if we are in the header then we are going to do something so we will say that if our value is between 250 if our x1 is between 250 and 450 then it means it is clicking the first one so here let me just write it here checking for the click so that is what we are doing here so we will write here that if this is the case then our header is equals to overlay list at zero so that's the first one so by default it is the first one because we are using it here uh the value of header is already where is it yeah it's already the first one then we are going to copy this and we are going to paste it down here and we will write that else if else if the value is between 550 and 750 so these values i have checked before so if it's not the same we can change a little bit so then it is 750 then it should be overlay list one and then we are going to copy this we will paste it here and then one more time so we have a total of four so here it will be 800 to 950 and this will be 1200 and 1050 so this will be 2 and this will be 3. okay so let's try that out so if we move around we should be able to select so if i go up you can see it selects if i go here it selects if i go here it's an x so wherever i go it selects the correct one and the visual indicator is quite nice so it tells us which one did we select and if i go with one finger it's not going to do it because this is the drawing mode in the selection mode we can select okay so that is the difference here so now we are able to select properly so that is quite good next we are going to change the color so whenever we select something we want to indicate that the color has changed so here for example for the drawing circle or even for the rectangle we want to change the color for it so what we will do is we will declare a color we will go up here and over here we are going to say now the first image is for purple so if we go to the header and we see the first image it is for purple or you can say pink whatever color you want to anyways so we are going to say that by default our draw color color is equals to purple so whenever the value is selected another value is selected we are going to change this draw color so and we will use it to actually draw so instead of putting a random color for the rectangle and for the circle this is the color that we are going to draw and let's draw it after we have checked for the click okay we will put it here is it in the if no it should be here okay so what can we do now is that we can change this color individually so here we will say that draw color is equals to for this case it should be purple so it should be two five five zero two five five and for the second one it is blue so it is b g r so this should be on and this should be off and then we have green so b g r so this should be zero this should be two five five and this should be zero and then for the eraser we will just make it black so black will erase everything so we will make it zero zero zero two five five two five five two five five is one and zero zero zero is black so let's run that let's try this so here our curler is purple and both of them are purple we go to selection mode and now it's changed to blue you can see for both of them it's changed to blue we go to green and now you can see it's changed to green right so and if we go to the eraser it's changed to black so we can erase it for the black actually we can make an exception for the black because for erasing normally the tool is quite bigger but again we will discuss that later so now this is done so the selection mode is done we are able to select our color and it changes the color that is perfect now what we need to do is we need to draw so we already know when we are in drawing mode we now need to draw based on our points okay so now the easiest method and this is one that i have been using before as well because it is very simple whenever you are learning a new concept you should go with the simple thing rather than the most advanced or complicated one so earlier what i did was i created using simple points so whenever you have a point you just draw that single circle and then whenever you move the finger you draw the circle there as well but the issue with that is whenever you have a rapid movement it will not draw continuously as a line it will have some gaps so that is not a good way to draw so what we will do now is instead of drawing just a circle we are going to draw lines but the issue with lines is that we need a starting point and we need an ending point so here whatever current position we are in we just have that single point so we need to know the previous point as well once we know those both points then we can simply draw a line so let's draw that so here we are going to write that our cv2 dot line and we are going to draw on our image and where exactly do we want to draw we are going to draw at our x previous position and x y previous position and then the new position which is x1 and y1 then we are going to say which color do we want to draw it with we are going to say the draw color and then we are going to give in the thickness so here we can declare a variable we can say it we can say that it is brush thickness and if we want to change it we can change it from the top here so here we can create our variables and we can write here that let's say our brush thickness is 15. so we can easily change it up and now we have the x and the y previous so we will go up here and we will declare that the x previous and the y previous are 0 and 0 and then we will go down here now the issue here is that at the very first iteration at the very first frame we will not have any xp and yp we will have the value of 0 0 so it will draw a line from 0 0 to whatever the point you are at and that will look really bad so this should not be the case so how can we fix it we can fix it by writing here that if our xp our xp is equals to zero and our yb is equals to zero it means it is the very first frame that we have detected the hand or we are starting to draw then we are going to say that xp and yp is equals to x1 and y1 so instead of putting 0 0 we are saying whatever value you are at draw exactly at the same point so instead of drawing a line it will just draw a point so the very first time we see our finger it will just draw a point instead of a line after that it will keep drawing as a line so whenever we have the new point we are going to say that our xp and yp is equals to our x1 and y1 so these are our previous points so it will keep updating those so hopefully that was clear now let's try to run it and see if it draws so here if i do this it is drawing but it is removing at the same time so as you can see it is drawing something but it is removing if i go really fast then you can see it is drawing so this means that our image is updating every iteration so we cannot draw on our actual image so if we want to do that we will have to uh do something else i will tell you what that something is later on but for now we need to create a new canvas on which we can draw so what we will do is we will go up here and at the very top we are going to create a new image and we are going to call it image canvas and this will be the canvas on which we will draw so we are using numpy to draw our canvas and we will use the zeros method and the size is 720 by one two eight zero they use height um before the width so we are writing it like this and it has three channels because we want colors and it is unsigned unsigned integer of 8 bits which means it will have 0 to 255 values so that is pretty much it and now instead of drawing on our original image we are going to draw on the canvas so where did it go here so we will copy this and we will paste it here and we will say image canvas and we will show the image canvas as well canvas okay so let's try that so now we should have two images so this is the image canvas and this is the original image i will keep this at the top and if i if i bring in my hand and if i draw now you can see it draws and that is very very satisfactory okay so this is the idea but now the thing is it's not drawing here which is fine uh if you want to you can draw there as well and i will show you how to do that but before we go there let's try out different colors so that we know that it is working well so i will keep this in the front but it will be hard to see okay let's try to put it here on the side and i will try to run that okay so this is a selection mode i will select blue and now i will draw with blue then we have the uh green mode the selection mode and then i will draw with green now i want to show you something else if i don't put this part if i don't put the fist frame condition what it will do is it will draw a line from the very start so if i bring it here uh okay this is very annoying uh we need to fix this first okay let's say i am uh here is my hand and then i start drawing there you go so did you see it started from the zero zero point and it drew all the way to the current point so whenever you have uh the new points let's say i go to the blue one and i select that you will see that okay not now but in the previous one you saw that whenever you have the first first frame then it will create this problem so we are going to open that up so that we don't have this issue okay next we can also try the uh erasing parts so the black one should erase whatever we have drawn earlier but what we can do is we can make it a little bit bigger because if it is bigger it is easier to erase so let's first select so let's draw and now i will go to the eraser and i will come here and you can see it is erasing but this is not very good because it's very small so what we can do is we can have a special condition for the erasing so we can write here that if the draw color is equals to 0 0 0 then we will have the size different so here we will copy this part actually we will copy both of these and here we are going to write that instead of the brush thickness it is the eraser thickness so we will write here eraser and we will also write here that this is eraser thickness so we will go up and here we are going to write eraser thickness is equals to let's say 50. so it will be easier to raise and here we can simply write else we do this so that should be good so if we have the eraser tool now so let's draw and now let's go to the eraser tool and it's a little bit hard for me because i'm sitting very next to the webcam and there you go so i can erase now very easily and all of this will get very very nice and very very good looking once we do it on the original image the only thing is that we have two images now and it is very hard to see what is happening so now all we need to do is we need to put it on the original image so we need to draw on the original image so how can we do that we cannot draw on the original image because it refreshes every time so instead of doing that we are going to add our two images so what we can do is we can go down here so here what we can do is we can write here that image is equals to cv2 dot add weighted and then we have to give in our image the first image and then we give in the value 0.5 let's say and then we give in the second image which is image canvas and then we given the value let's say 0.5 so this will add these two images and it will blend them so let's try that out uh there is a argument missing there you go so if i draw you can see now it is drawing again i need to go back a little bit okay so now you can see it is drawing when the hand is really at the back like this it will draw well but because i'm very close to the pc and the camera it is hard for me to do this i can go back a little bit and there you go but the issue here is that this does not look very good i can change these values but still it will be a blend it will not be an actual merging of the images it the colors will not be that bright there will be a transparency on it so if you don't want that what can you do now it is a little bit complicated but if you break it down it is very simple so what we will do is we will write here uh let's where is the statement here okay so we need to go here and we will write here first of all we are going to create a gray image so we will write here image gray is equals to cv2 dot cvt color and we are going to say that image canvas and cv2 dot color underscore uh bgr to gray you are too gray so we are converting it into gray image you might be wondering wait why are we converting it to gray image i will explain in a bit then we are going to convert it uh into a binary image then we will write here image inverse is equals to now we are creating we are converting this into a binary image and we are also inversing it what does that mean i will explain as well cv2 dot threshold and image gray and we are going to write 50 and 255 and then cv2 dot threshold binary inverse so the idea basically is that we want let me actually run it so you can see better so let's say i draw something let's say i drew this and now i have this image so what i want to do i want to convert this image into black and white so wherever i have black i want it to be white and wherever i have this colored image i want it to be black so what this will do is it will create a mask with all this white and only this region as black and then i will go to this image and in this image i will make all of this black and then i will merge this image with this previous image with the black area so it will overlay these two so i know it sounds a little bit complicated but you will see how it works so we first of all we are creating that inverse image so that all that region where we drew it is black then we are going to write here that our image inverse is equal to cv2 dot cvt color and we are going to write that our image inverse is basically cv2 dot uh color color underscore gray to bgr now we are converting it back because we want to add it to our original image so we cannot add it if they are not the same dimension you cannot add a gray image to a colored image so we need to make sure both of them have three dimensions then we are going to do image is equals to cv2 dot bitwise and and we are going to add our original image with the inverse image image inverse and then we are going to add our image is equals to cv2 dot bitwise or and then we are going to write image and image canvas so again this might be a little bit confusing but let's see the results and i will go step by step and explain it as well so now here you can see i can draw oh it went down here you can see i can draw easily uh it's not updating this value it seems yeah okay it's not updating the value okay so this is drawing on the canvas now which is good but uh actually on the original image but the issue is that it's drawing this straight line whenever it is detecting again so what is happening is that whenever the hand is detected we should put this x p and y p as zero so we will copy this part and whenever the hand is detected then we will make it zero actually no whenever the hand is detected no no no whenever we start drawing again so yeah whenever we start drawing again or whenever we have a selection whenever we have a selection then we are going to do that okay let's try that yeah now it's starting from the right position instead of a random position there you go so that is good so let me explain what is happening at the back so first of all we have our gray image and then we are converting it into an image inverse so let me display that image inverse so this is our inverse image so wherever i draw something it is going to draw but with black area so let's try that so as you can see it will draw with that black area right so that is the idea then the next step is that we add these images so we are adding with and we are adding the image inverse and the image so let me show you how that looks like so i will remove this part and we will run it we will see here that when we draw you will see that now it is showing us black region wherever we drew so all we have to do now is we have to add this image to this image so when we add this because here we have colored part here it's black here we have colored part and here it's black so if we do an or operation between these two it will give us our final image so here we were doing an and operation with the original image and the image inverse here we will do an or operation to add these two up and there you go so now it's moving around and the flickering you can improve if you have a better detection if you don't have a lot of noise so again it cannot be completely uncontrolled environment the environment should be a little bit controlled to have some good results so if i go back you can see i can draw a few things and then if i go to the eraser if i go to the eraser and i select the eraser it will rub quite a bit again my conditions are not very well but you can get the idea and if i want to increase the size of the brush or the paint or whatever i can change it from here i can make this 100 for the eraser thickness and it will erase better so here if i draw something and then i can do this and i can go to the eraser and then i can erase you can see how that's how simple that is so this is basically the idea we didn't use time if you want you can show the frame rate and the frame rate is pretty much the same it is around 30 frames per second so that should be good so this is how we can actually create a virtual painter using the latest technique of hand gesture and hand tracking so this is it for today's video i hope you have learned something new if you like the video give it a thumbs up and don't forget to subscribe and do share it with your friends if you like this video and i will see you in the next one

Channel: Murtaza's Workshop - Robotics and AI

Views: 85,445

Rating: 4.9682984 out of 5

Keywords:

Id: ZiwZaAVbXQo

Length: 69min 28sec (4168 seconds)

Published: Sat Apr 24 2021