Computer Vision Game using OpenCV Python | Velcro Dart Board

hey everyone welcome to my channel in this video we are going to create a computer vision game using opencv in python we will learn how to detect the balls and find out which region did it hit on the board we will go step by step so it is easy to follow if you would like to level up your computer vision skills do check out our premium courses the link is in the description so without further ado let's get started so here we are in our pycharm project and you can see that we have called it velcro darth game and we already have the videos folder and one dot png this is the image that we will be using but why do we have these the reason is that some of you might not want to buy the actual velcro dart board so if you want to learn the technique then this should be enough to follow along and learn the principle of computer vision behind this project so we will have some videos you can see you have three videos here and then we have the one dot png this is the dart board and behind that i have also attached a board behind it so so what we will do is that instead of using the actual webcam which i have already set up next to me we are going to use the videos instead so that the ones who are at home and don't have the equipment they can follow along as well so this image and these videos they will be available to download from the website the link will be in the description below and all you have to do is you have to sign in and then you can download these files okay so the basic idea behind this project is that we want to detect the hits of the ball and we want to see where did it land and based on the location of it we are going to give it the certain points uh that are allocated with it so for example if it hits here we are going to say five if it hits here we are going to say 60. but the thing is that these are not uh square shapes or they are not circular shaped so each one of these has a different shape so for example you can see here this is a polygon with four points and again we have most of these are four points but they have different um what you call locations and they don't really fit in a rectangular shape so what we need to do is we need to check for each one of these whether the ball has hit in this space and if it hit for example in this space then we are going to say give it 60 points if it hits for example at this point we will say give it five points and so on so we need to find all the locations of these and then we will save them and then we will check where did the ball hit the second thing is that this image is actually a little bit tilted so let me exaggerate a little bit so you can see better so for example uh if you see the camera this is the dart board on the side and then the camera is below the dart board and it is facing upwards so this is the angle the angle of view of the camera so we are looking from the side so therefore the image that was supposed to be a rectangle will now be something like this so it will be a little bit bigger at the bottom so here you can see this line will be bigger than this line so i'm exaggerating it a little bit here so you can see what is happening so instead of a perfect circle like this we are going to get something like this so this is what we are getting so what we will do is we are going to collect these points we will collect these four points and we are going to work it into this so we will convert it into uh a rectangle and then from that we are going to find all the locations of our different points uh our different polygons and based on these polygons we are going to see where did the ball hit so this is quite a long project so it might take a while to actually complete i'm not sure how long this will take so let's go ahead and get right into it and let's see how it goes now the thing is we will only need one what do you call a package so we will go to file settings and we will go to the projects and the python interpreter and i'm using python 3.7 uh feel free to use uh any other version that you have but if it doesn't work then it's always best to come back to the stable version for for me it is 3.7.6 so if you want to try that you can try it with that okay so let's add our library so we are going to add cv zone and this is the only package that we will need because it will install the opencv and the uh numpy package that will be required at the back end that will do the actual computing cv zone is just a wrapper to make uh to add a little more functionality to these libraries okay so now you can see python has been installed opencv python and numpy has been installed as well so that is good now we are going to create a main file so let's call it main main dot pi and the first thing we will do is to import the computer vision library cv2 cv stands for computer vision then we are going to import cv zone and then we will import numpy as np now i will later on tell you why we need all of these um and then we will import from cv zone dots dot color module we are going to import color finder so when we are detecting the ball we are going to detect its color so you can use other methods as well you can even use a machine learning model that you can train to detect these different balls and then you can detect it from that and then the rest of the project will be the same what we do but here we are using the color method but if you are using the color method you have to make sure you don't have that color in the dark board so for example we are using green dots so they are not the green color is not present in this dark board so we can do that if you have the same color then it might be conflicting and it will cause a problem so then you might want to refer to contours method to actually get the edges or the circle or you might want to use a half transform to find the circle and there are many other ways you can do that but for this for this project we are going to do the color finder and the idea behind this project is not that you copy exactly what i have done it's uh the idea is that you understand the basics and from there you can create your own project your i'm pretty sure you are more creative than me so you can embed this into your own projects and create much more better projects so hopefully this will help you out so we will also import pickle now why do we need pickle well the thing is that we are going to define all of these points and all of these polygons and their regions and we will define their points as well so to do this we are going to create a new script in which we will just label uh this dart board so for labeling we will create a new script and we are going to save a file from that script and we are going to import it here for that information now this might seem a little bit confusing at this point but don't worry i will explain when we come to that point okay so now the first thing is that we are not using the webcam but we still need to import the video so we will write here cap.cv2 and the webcam and the video is pretty much the same so video capture if you are captured so if you are using uh a webcam you will just write zero or one whatever so you can use that but in this case we are going to use a video we will get it from the videos folder and this will be video number one so yeah we need to write the extension as well dot mp4 then we are going to write while true and we will write success and image is equals to cap dot read we are going to read each frame and we will write cb2 dot weight key weight key and we are going to write uh not zero we will write one and we will also do the i am show cb2 dot im show and we are going to write here image and img img there we go okay it seems that there is no configuration here so what you can do is you can simply right click here and run main and that will automatically solve that issue for you okay there you go so now we are getting our uh video so this is actually a static video you can say the camera the webcam is still and there's nothing showing in it video number two is actually with the with the balls hitting so here i missed one and then there's two and three so they're moving really fast so we will have to uh slow them down as well so to make sure that they are 30 frames per second so we can slow it down from here for example we can write 50 and it will slow down about 20 frames per second there you go so that is another hit and the good thing is we have different variations where it hits where it falls down when it hits in the wrong place so all that so if we run this uh it will run and then it will stop so it will give us an error so why did we get this error the reason is that the video ended so what we need to do is we need to let's just put it as one so we need to keep this loop running so that when we are testing uh it doesn't just quit every time we want it to keep running so here we are going to write frame counter plus equals one so we are counting how many frames have passed by and we will check if our frame counter actually let's just define it otherwise it will keep giving another so let's just write your frame counter is equals to zero so initially the frame count is zero we are adding to the frame counter and we will check if the cap dot gets cb2 dot cap underscore prop underscore frame underscore counts so if it is equal to that then we are going to write here frame counter is equals to zero and then we will write cap dot set uh cv2 dot uh cap underscore prop underscore position underscore frames is zero so basically what we are doing we are saying that keep counting the frames and we will check how many frames are actually in our video so if we have reached the maximum frames then we are going to set the frame back to zero so it will reset the video so right now it is at a very high speed so if we run it it will hit a few times and there you go it starts again so this is what we want we don't want it to uh stop and give us an error so that is good now the second step is to actually work the image so if you remember we need to take these points and we need to work so that we get a completely centered image and there is no tilting of the dartboard so how can we do that we can go to our image we will open it up in the explorer and here we are going to right click and we are going to click on edit so this will open up in paint and what we are going to do is we are going to note down the values of our corner points now you could do this with uh dynamically as well but uh we are not going to do that because it will just add a little more complexity and in this case it is not required because the camera is not moving the camera is stationary so we don't need to do that so let's just look at the points so if you look here when i point it to the corner you can see at the bottom left corner it will give you the pixel value right now it is saying 377 and 52 so i'm going to write that down and i will say 377 and 52 that is my first point then we are going to look at our second point our second point has to be this over here which has width and the height is zero so nine four four and seventy one so we will write here nine four four and seventy one if you are following along with the image and the videos then this should be exactly the same what i'm doing so then if we go down here it will be 261 and 624 261 and 624 then we will do the last one and the last one is one zero five eight and six one two one zero five eight and six one two so these are the four points that we need we are going to comment them right now and we will actually let's just put them in points so we will write here points uh actually let's give it a better name instead of just saying points let's say corner points corner points so our corner points are basically uh these values 3 7 7 and 52 then we have another one then we have another one and then we have another one so we have a list of values and within that list there is another list that has two values x and y so this is the basic idea and the order has to be exactly the same otherwise it will not work properly when we work the image so if this was dynamic we would have to sort these points and actually find which one is zero zero which one is width height which one is zero height and which one is zero width zero so but in this case we don't have to do that okay now to warp it what we can do is instead of doing it all in the while loop we can actually create a function and uh we can for example call this um get start board or get bored so let's say get bored b-o-a-r-d and we are going to send in our image to get the board and in return we need the output image so we will write here image or um yeah let's call it image out but is equals to this or should we call it image board i think it's better if you call it image board yeah so get bored we are we are going to define this we will write here get bored and then we will take in our image and then we need to add our warping part so for warping first of all we need two sets of points the first point we already have so the points one is equals to numpy dot floats now we are going to use the word perspective function and the get work perspective transform now these functions they require these values or these lists in a certain type this type is float 32 so that's why we are using numpy to convert this list into a float32 list otherwise it's not really required but because it's a requirement of the function we are about to use so we do have to do that so here we will write float32 oh float 32 and then we will simply given our points so these are our corner points my bad so we are going to write corner points and this will be our first point then the second points will be what exactly do you want this to be so this point here if you look at our image this point here what is the size of this okay so this point here should be zero zero this point here should be width zero this one is height uh zero height and this one is width height so we need to define each one of these so we are going to write here points 2 is equals to again numpy dot float 32 and then inside that we are going to create our matrix like we did before and this time we will say the first one is 0 0 then we have the second one the third one and the fourth one the second one is width and zero the third one is height zero and height and the fourth one is width and height so this is the basic idea but what is width what is height where do we get that from we can get it from our original image so width and height is equals to we are going to write here integer and we are going to write in our values so here we have to give in the size of the actual board so the size of the actual board is basically 400 by 380 but this is quite small so what we will do is we will multiply it with a factor so this is the actual size in mm so what we will do is we will multiply it by 1.5 and we will multiply it by 1.5 so we are keeping the aspect ratio same we are just scaling it up then once we have this information we are going to get the transformation matrix that will allow us to work the image now if you are not familiar with all of this i highly recommend that you check out my word perspective video and it will give you a little more detail about this so here we are going to write matrix is equals to cv2 dot gets perspective transform uh perspective transform and then we are going to write points uh one and then points two then we are going to write here that our image is equals to cv2 dot uh warp perspective uh perspective is basically image and then we have our matrix matrix and then we have our width and the height there you go so this will give us the output image let's let's write here output this is the output image and then what we can do is we can send it out to our actual image so we can basically uh out of the function we can write here image output so that will be returned so what is happening here is that we are creating the metrics based on this information and uh using this matrix we are transform transforming this image uh that we are getting and we are storing the output image using the width and the height that we have provided here and if you want you can actually draw the points these four points are not these these four points that will allow you to see if you are getting the correct points or not so here you can write for x in range uh let's say four we are going to write cv2 dot circle and then we are going to put it on our original image we are not putting it on this we are putting it on our original image and then we are going to write our points so we will write here corner points um we are going to get the x value and then we are going to get zero this will be the so for example this first value will be this so this is the zeroth element and this is the first of it uh the first element of it so then we are going to write again the same thing but we will get the second element which is one and then we will give it a thickness uh or this the radius will give it 15 we want it to be big and we will give it a color let's give it green two five five and zero and then we will say cb2 dot filled so we want to fill it up okay so let's go ahead and run this and see if it works so let's run that we are getting image is not defined so what is not defined where is it here image is not defined why is that okay well we need to do it after we read the image not before that okay so let's run that and there you go so here you can see these are the correct points and if you look at does it give no we didn't actually get the output so we need to write here cb2 dot i am show i am show and we will write here image let's say board and we will write here image board so let's run that and there you go so this is our image board now it is straight even though this is a little bit curved that's fine but our image actually is straight at this point so we have converted it from this to right where it looks like it is dead center it is dead straight so this is good now we have worked our image now what we need to do is we need to save one of these images so that we can use it for color detection and we can also use this to find the points where exactly uh are all of these different polygons so we need to save two images so we can write here cv2 dots i am read and we are going to um give it a name so we are going to write here image uh it should be in quotations image and then we will give it an extension.png and then we will write here image board so let's run that flag is required to be an integer what mistake did i do so i am oh sorry not i am read i am right my bad what am i doing okay so when it has all three i'm going to stop one two three stop so now i know it's a weird way to do this but here you can see now we have the image with all three balls at the same time so now we can go ahead and do the color detection for this image and for the second image we will call its uh image board v o a r d so we want it when the board is completely empty so we can stop it right now and the board is completely empty at this point so we can use this image to find these all these polygon locations because this is exactly what we are going to get every single time so these are the two images that we have that we will be using on later so let's go ahead and first of all detect the color now so now that we have our board and we have our image we are going to detect the color so for the color we don't need to uh create or we don't need to use the video instead we are going to use an image and what we can do is instead of writing this line we are going to comment this out and we are going to write uh image is equals to cv2 dots i am read and we are going to read in our img dot png so we will write here image dot png and let's see if that works and there you go it works so we are getting our main image we can forget about the image board uh because right now we are not using the actual uh size so the next step would be to actually find the color so for the color we have already imported our color module and we are going to use the color finder so when we are using the color finder what we have to do we have to write here color finder is equals to color finder and we need to give in the debug so by default the debug is false so we are going to put it true because we first of all need to find the color and then we can apply it so to find the color we will put debug as true and then over here once we get our image we can remove this part for now to get the board uh oh actually let's keep it because it will be annoying let's uh or let's keep this as well i don't want to delete it from the code maybe you want to use it later on but let's just not use it for now because we are done with doing the images i hope it did not overwrite oh yeah it did override ah that's bad so we we can fix that later i should have removed it before anyways so what we will do now is we are going to find the color so what we need to simply do is we will call our object that we have created and to call it we will simply call the update method so here we can write that our image color and our mask is equals to color finder dot update and we will give in our image to update that's it so right now it is in debug mode that's why we don't need to tell it which color to find and which color to update because we are about to find out so here you will see that we have a new track bar and what we can do is we can move this track bar to change the color here but right now we are not viewing the image color we are just uh outputting the image so we need to output cv2 dot i am show and we need to write here image color and we will write image color here so let's run that so this is our image color and if i move this you will see the color changes so we need to find the color of this green so we will move it until only this green is left so we we want to remove everything except the screen so we need to move it a little bit more there you go now we are getting somewhere so this is quite fine uh yeah i think this should be fine so we can take this as our output color so let's stop this and this here will print the color for us so what we have to do is we just have to copy it and we have to paste so what we can do is let's just run it one more time to be 100 sure because this is a tricky part so let's just move it a little bit let's say we bring it down to let's say 40 and this one we will bring it further to 20 something yeah there you go this we need to move it a little bit not too much and yeah i think that's that's good enough so we'll stop here and we will copy this and now what we have to do is we have to go up here and we have to define this as our hs v values is equals to this so here we will put it as false now we have finished the debugging we know which color we want to update which color we want to find so here we will simply give in the hsv values hsv values and now it will only detect this color so if we run this you can see it is giving this by default there is no track bar detected and this is the the image that we get so to test it out what we can do is we can remove our imbreed from here and we can go back to our video so let's run that and there you go so let's see there you go we got one hit that there you go another hit and then another hit so that is pretty cool and what we can do is we can create the mask we can show the mask out of this so let's copy this part and we are going to write image mask and we will write here mask so if we run that this is our mask so you will see there you go this is a clear hit this is a clear hit but the issue here is now that there are black lines in between so we are not really getting a perfect circle and this is a problem because when we go to detecting the contours and detecting the center point of this it will not detect it as one ball it will detect it as different small triangular or rectangular whatever the shape is uh four pieces three pieces something like that but we need to join them together and we need to remove these noise as well this noise otherwise it will not work properly so what we will do is uh instead of just writing it down here we are going to create another function and we are going to call it what should we call it let's call it um detects or yeah detect color darts let's see so and we are going to given our image again and here we are here we will do all the processing whatever is required we need to reduce the noise we need to increase the dilation erosion whatever is required we will do here so that it doesn't interfere with the rest of the code and here we can simply call in the function so what we actually need is the mask we don't care about the rest what we care about is the mask so here we are going to write mask is equals to the text color dots and we are going to send in our image board so this is the idea and image color should we keep it should we remove it let's put it inside here okay so this is yeah this is correct now and we will return the mask so here oh what did i do here we are going to write return mask so that we don't forget later on and now we need to process this a little bit better so that we can get better results in actually the mask as well or yeah let's just keep it here let's keep the mask here okay so the first thing we are going to do we are going to add a little bit of blur before we send the image this will allow it uh to not be very sharp and it will allow it to be less noisy so here we will write image blur blur is equals to cv2 dot gaussian blur and we are going to given our image and then we will give the kernel size let's say seven by seven it has to be odd numbers and then we will give the delta value so this should be fine and let's just put this as our input instead of the actual image and let's see if we see any difference so where is the image color there is the image there you go so now you are seeing the blurriness so that will help us with the detection with the noise as well okay so now if we look at this again you can see that we have some noise here and we need to remove this so one of the methods of removing noise is basically uh opening and this is basically an erosion followed by uh dilation so in opencv we have this function called open and we are going to first of all define a kernel kernel is equals to np dot once and we are going to define it by the size of 5 by 5 and then we are going to give it numpy dot unsigned integers of eight bits so this kernel we are going to use so we will write here mask is equals to cv2 dot uh more for logy x and we are going to write here mask and then we will write cv2 dot morph underscore open and then we will give in our kernel so here we will write mask so this will remove that noise so here we can write different values again this this has to be odd so if we run this now there you go so this is the image color and this is our mask without that noise now you can see we are only getting the balls so you can play around with these values to increase or decrease uh the noise then even then you might have a little bit of noise here and there if you want to remove that as well well you cannot remove all noise but you can try your best here you can add a median blur so here you can write mask is equals to median cv2 dot median blur and you can write in your mask mask and you can give in the intensity so here we are giving it a high intensity so let's see so here now you can see the noise is reduced a lot and now you can see only pretty much the balls the noise is very very good so now we have reached a good point it is uh it's not noisy anymore but the issue is now that the balls they are um they are not looking like balls that that's the problem they are far apart and they are different objects here you can see they are not joined together so we need dilation function here to join them together so we will write here mask is equals to cv2 dots dilates uh now the spellings are wrong dilates and then we are going to write mask again mask and then we will give in our kernel the kernel we are using the same if you want to you can change the kernel and then we are going to given the iteration inter iteration what are the spellings iterations is equals to 2. so we are going to write it like that and let's go ahead and try it there you go so there you go now you can see it's much better almost an object we can increase the iteration again but there is another function which is uh the opposite of this it is open the other one is close so we can copy the kernel and this function and instead of opening up we are going to write here close and we are going to give it a little bit harsher values so that you can see still there are gaps we want to close it aggressively so let's see if that works there you go so now it is almost closed and we are getting these blobs uh that we can use as the location of the ball height so again it is not perfect but it is a good start and you can keep improving with different parameters i did not spend a lot of time on this so if you want you can definitely get better results than what i have received here or what i've got here so you can definitely try to change them okay so now that we have our mask what we can do is uh we can find its location so we need to find the contour and we need to draw the contour around it so let's let's write it where is it so this is the dark color and then after that we are going to find the contour so we write here cv zone dot find contours and we will find from the image board and we want to draw on the image board and we want to find the contour from the mask and then we have a minimum value of three five zero zero so minimum area you can play around with this area as well and then we want to draw it so we will keep it like that and let's see how it works out so we are going to return it to image contours and then we will write here contour found so it will this will have all the contours and the information of their location their size their bounding box and whatnot and this is the image with the contours drawn so we are going to output that as well so we will write here image contours and we will write here image contours so let's run that and this is image contours so as soon as we get something there you go you saw that blue there you go perfect perfect oh this is flickering but still not bad so maybe we can play around with the values a little bit to get better results so this is the basic idea here you can see now we are getting the center points and we are pretty much uh getting the bounding box so you can see the bounding box is quite accurate even though we are getting weird blobs but still the bounding box is pretty much around the circle so that's a good sign so that is good let's try to change the value here a little bit so what should we change maybe the iteration here let's dilate it a little bit more yeah yeah yeah that worked so we just had to have to dilate a little bit more what if we do it for is it going to get better let's try that yeah seems to have gotten better perfect so the overall area and the center point seems quite accurate so that's good we can try it with the second video as well so video one no video one does not have anything video three and video two have the ball hits there you go there you go and there you go perfect so again uh try to play with these values uh here you can see you are getting some noise so maybe to reduce the noise we need to increase the kernel for this for the open let's make it seven and seven so now it's much better the noise is gone and the center region is i would say it's quite good okay good so this is basically the idea of how you can find these contours now what we need to do is we need to first of all save when it hits and secondly we need to check where did it hit and based on that location we are going to uh either increase the points or not increase the points and we are going to color that area in which we have found the hit so that is our next step so how exactly can we do that now the first thing is that we need to find the hits and we need to save those hits so what we will do is whenever there is a hit whenever there is a contour detected we are going to wait for five iterations or five frames before we count it as a hit because sometimes it is in the air it is reaching the board and we don't want to detect that as a hit we want it to be on the board for a little bit for example for a few milliseconds before we detect it as a hit because sometimes when we throw the ball it will hit and it will fall down so we don't want to accept that as a hit as well so this will uh actually fix that problem so what we will do is we will say that if contour found if it has anything inside what we will do is we will add a counter so so we will write here count hit is equals to zero and then here we are going to write count hit uh what did i write counts hits plus equals one so we are going to count whenever there is a contour found we are going to keep counting it and we will say if count hit if count hits is equals to five let's say so if for five consecutive frames we are getting the confound then we are going to detect this as a hit and how do we detect it as a hit we are going to save this image and its location in a list so that we can display it continuously so what we will do is we will save this in a list so we will call it uh what should we call it image list balls detected i know it's long but you know at least it we can tell what it means so here what we will do we will append so dot append and we will append the mask so whatever mask we got we are going to append it here so what this will do is this will give us a black and white image of where did the ball hit now what we are about to do is something weird and we will save the contour information as well later on but for now what we are going to do is uh we are going to print here uh let's say hit detected okay so whenever it saves we are going to write here hit detected and i will tell you why did we store these images so hit detected one it didn't say hit detected again why did that happen so one hit second hit and third hit uh okay i know why because we did not put it as zero counter count hit is equals to zero so if we run that again then hit detected it detected it detected so it's doing more than that okay so that's not good um when the count hit is five then it should be zero um oh yeah uh yeah i forgot so it will keep saying hit detected hit detected because if you look at this image now it already has the previous hit and the new hit so whenever it looks at the images it is finding this contour this and this so whenever we detect a hit whenever we detect the hit let me pause this okay now it's paused so whenever we detect the hit what we need to do is we need to remove the the contours from our previous hit so that if we detect any new contour it just finds the new one and not the old ones so if this is my first hit i will save this image and then when the new hit comes i will subtract the new image from the old one so that i don't get the previous contour detected so this might be a little bit difficult to understand so what i will do is i will write it down here to explain so for example this is my first hit and this is where it hits and then for my second hits this is where it hits this point here but my first hit is already present at that point so when i am sending it to find the contours it will find both the contours this one and this one but we don't want that we just want to find the new one so how do we do that we subtract these two images so if you subtract these images the resultant image will be this so this point here will cancel out so this is what we will do each time so for example our first image let's say we find the hit here so we will store it in a list we will say our first hit is here then we get the second hit the second hits is at this point and now our two contours are at this point so we will subtract it with this and then our resultant image will be this now we will save this image in our list so this will be our second hit and then we get a third hit the third hit is basically at this point so we will have the other two hits as well in the webcam image because they are already on the dart board so what we will do now is we are going to subtract this image from this and from this so when we subtract it from both of them then we will find this as our new contour and then we will check our new location and we will detect it as a new hit i hope that is clear so that is what we are going to do so let's go ahead and try that out so right now what we are doing we are storing that image in this image list balls detected so now what we need to do is we need to remove the previous detections so here we are going to write remove previous detections so here we are going to write for x uh no for image in image list balls detected we are going to loop through all the images that we have detected and then we are simply going to do mask mask is equals to mask minus our original image so uh this is the new mask minus our old mask our image that we are getting from this which was which was the old mask so this is the basic idea and what we can do is we can display each one of these in a separate image if you want to see we can write here x and then we can write here enumerate and we can output each of these so we can write here cb2 dot i am show and we will write string and x the reason we are enumerating because we need a different name for each image so we are putting a dynamic name for each image and then we can simply write mask why am i writing the spelling of mask wrong every single time i don't know okay so let's see so this is oh my god so it is doing a lot of times that's not good so we need to do it one time and then we need to stop so i will hold it there you go oh oh i can't that's way too much how can we stop it maybe if i just remove this frame counter maybe it will stop no it didn't stop why didn't it stop or let's just put a big delay between each of the frames so that it is easier to see what is happening so let's put a 500 millisecond delay so this will take a long time but we will be able to see now that's too much let's put it at hundred so when we get the first hits this is our first image second image third image oh my god so that's not good so we are getting a lot of images right away which is not good okay so uh how can we make sure that it is working properly or not uh what we can do is we can uh we can put this back to one first of all then we will put this back to continuous and then by the way this should be after the contour find should be after removing the previous detections okay let's run this now there you go one this zero one this is another one and then there is a third one there you go so each of these is at a different location so right now we need to check how many hits did we take so the number of hits are lower than before so let's see the first hit the second hit the turret perfect so now our hits are proper now it's repeating the video again and again that's why the hits are repeating but now we are able to detect each hit individually because we are removing the previous hit and then we are only looking at the new hit so now we can perfectly tell how many hits that we detected so this is all good but now is the tricky part where we need to see where exactly did it hit and how many points do we give to the user okay so now to find the hits we are going to create a new file we are going to right click and we will create a new file and we are going to call it path picker so we are going to pick the path from this so we are going to import cv2 and then we will import numpy as np and then we will import pickle so what exactly are we trying to do we will take in this image that we saved of the image board oh that is bad so first of all let's just go back and take the image of the board so we will write here cb2 and we will run that and we will stop so if we look at the image board now it's fine so that's good so i will remove that from here so that i don't accidentally overwrite it again so we need to bring in this image of the image board and we need to get these points now you can open up it in paints and then you can find the x y location of each one of these and then you can put it in a text file and then you can import it into the main file which is a long process to be honest so what we will do instead we are going to write a script which will be it's not complicated but uh it is a little bit technical we will write that down and based on that we will just click the points and we are going to save uh the location of the polygons even if we add like 10 points to that polygon it is fine you can add three points four points 10 points 20 points whatever amount you want you can add and then we are also going to store the number of points so here the score should be 5 here it should be 60 here it should be 10 here it should be nothing so we need to give each of these categories a score the nothings we can ignore but the rest of them we have to give points so we will create a new file with this and that will allow us to easily annotates all of this so it will be quite easy to work with so what we will write down here first of all we are going to create uh two uh lists we are going to first of all call it polygons this will be our first list in this we will have all the polygons poly gonz so we will have all the polygons and not just the polygons we will also have the points the all the polygons and their points so five points ten points whatever it is and uh each polygon can have one point uh can have two points three points five points uh whatever number of points we have so i know each one should have at least three points so then we have the path the path is basically one polygon so the path of one polygon so we will write it down one by one and then we are going to put it all together um as one polygon so here first of all we will write image is equals to cv2 dot i am read and then we are going to write image uh image board b o a r d dot png and then we are going to write uh do we need to write anything else no we don't need to write anything else okay so once uh that is done what we will do is we will create a while loop because we are clicking on points even if we have just one image we need to loop so that we can add different points so what we will do here is we will write cv2 dot im show and we want to show the image and we will call it as img and then we are going to write here that um cb2 dot weight key is one so it will keep running so let's go ahead and run this this should give us a window there you go so this is our image and we can click on it and it is running even if we close it it will not close because it is running a while loop so that is good so we can close this and now what we will do is we are going to click on the points and we are going to save their location so to do that what we have to do we have to create should we do that first or let's write it down here so we are going to write cv2 dot set mouse call back and then we are going to write image this is the this has to be exactly the same uh this is the name of the window and what do we do when we detect a click we call a function we will call it mouse points and this mouse points we are going to define here and what we will do is we will get the events and the x and y and the flags flags and the parameters so we don't need to use all of these but this is what is the requirement so we will write here if our events is equals to cv2 dot event underscore left button down so if we detect a left button click then we are going to append to our path so path dot append and we are going to append x and y so this is basically the x and y position of the mouse click append x and y so that is good but then we don't know if it actually clicked or not actually we can print it out and we can see so here if i write path so we can write print path and if i run this and if i click you can see it saves so path was one point and then second point then third point fourth point and so on but visually it's not appealing because we don't know where we clicked so what we can do is we can loop through all the points and we can draw them so we'll write here four points in paths we will write cv2 dot circle and we will put it on our original image we will put in the points we will give it a thickness sorry a radius and then we will give it a color let's say red and we will give its cv cv2 dot fill we want it filled so let's run that and now wherever we click it is going to fill up with that red color so that is perfect and it will uh what do you call add to our path but it still doesn't look like a complete path so what we can do is we can take this these points and we can create a polygon from it so to use this we we have the function uh called cb2 dot poly poly lines and uh we need to send in our image and then we need to send in the points uh for all the points of all our polygon so that is basically our path at this point so we are simply sending in the path and then we are saying that it is closed so we will write true and then we will give in the color so let's say green 2 5 5 or let's say blue 0 and then we will given the thickness which is 2. but the issue here is that it does not accept it as a list so this here is a simple list a python list it does not accept this function does not accept python list what is it accepts is a numpy array that's why we imported this numpy so why do we do that because it is the requirement of the function otherwise we would ignore so let's say we write here our points is equals to numpy dot array and then we will given our path which we want to convert and what we want to convert it into is numpy dot integer 32 and not only this but it also has to be reshaped it means it has to have some brackets here and there so we can write here dot reshape again this is just a requirement we will write here minus 1 1 and two so whenever you're drawing polygons from a list you have to do this end of story so not a pretty story but still uh that's the idea i think i press the wrong button we need to format this there you go okay so now we will be able to see so we are basically putting the polygon and we are putting it back on our image so if i click on this oh what happened oh yeah one more thing you have to add another bracket here another bracket here and you have to write bts the points in here here this is the requirement so if i click now and if i click on this there you go now you can see a line if i click on that and then i click on it blue might not be the best color so let's just change it to green so i click on this and then this and this and this don't worry about the line in between that's fine but that's the idea so now what we need to do is whenever we uh have our points so you can have more points than this so for example if i'm doing the one above five so i can have multiple points like this and that's fine so i can do it like that that's fine as well but the thing is that we need to save this as a polygon and then we need to add another polygon so basically whenever we are done with the polygon we should press a button and it should save that polygon and then we go to the next one and we draw that and then the next and the next so how can we do that it's very simple it's not complicated at all uh what we will do is instead of writing cb2.weightkey1 we are simply going it we are giving it to a variable we will write here key is equals to this and we will say that if key is equals to ord and we have pressed the q button then we are going to save this path into one big polygon okay so here what we will do first of all we will ask the user to give in the points so points is equals to or let's call it score score is equals to uh integer and we will ask for an input and in the input we will write enter [Music] enter score so yeah that is our score then we will append it to our main list which was polygons this will have all polygons this is a single polygon so we can write here single polygon okay so this is a single polygon and this is when we are done with this we put this inside polygons so here we are going to write polygons dot append but we don't want to append only uh this path we want to append the path and path and the score we want to do both of them so that later on we can import it and know which polygon has how much score and to see if we are in the right or heading in the right direction we can write here prints we can print total polygons and we will write here the length of the length of uh polygons ah not the platform polygons there you go so that will give us the length of the polygons and then we need to delete the path because we need to restart for a new polygon so whenever we do that we need to reset the path again so that is good so every time we do this let's run it so if we create one of them and if i press the q button it will ask me enter the score i will write five let's say and it will tell me total polygons is one then i can do another one i can press q it will ask me the score i will say 10 and now total polygons is two but then we need to save it uh in a file so that we can import it to our main file here so what we will do here is that we will write if our key is equals to ord and we are going to write that if it is we will give it a key that is far away so that we don't mix these buttons you can keep them close like q and w but i like to keep it far away so that they don't mix up so i will write the furthest which is p and once we do that then we need to do with open we are going to save our file by the name polygons polygons and we will give it the permission for um writing and we will write wb and then we are going to why where are the quotations okay we write as f as a file and then we are going to print the polygons before we send it polygons and then here we are going to write pickle dot dump and we are going to write polygons and we will write our file name so this is our file and that is it so here then we can simply break so we need to finish our loop so i believe we have written pretty much everything we require so let's go ahead and try for one or two and then we will do it for the whole we need to import that there and test it before we can do for the whole thing so let's say we will do it for this one actually uh which of these are we hitting let's go here so we are hitting this one and this one here but that one is not detected so uh let's change the video let's go here and use video number two so we are hitting uh no this one didn't detect this one and this one and this one so let's just do these two on the sides and then we will do the uh the rest of them later on so let's run that and we will do this one now i will press q and it will ask me for the score i will say 60 and i will press enter total polygons is 1 then i will do this one i will press q it will again ask score is 60 i will press enter and then i will go back i will click on the top here and then i will press p so now it is showing me before it saves it is showing me this list so this is one element and this is the second element so now what we need to do is we need to import this uh in our main file and if one of them or two of them right now are importing properly then it means the rest of them will import properly so we can come back and do for the whole thing so for testing we can just do one or two and then we will do the rest of it so here what we will do is we will write with open and by the way it saved the file here called polygons you can see here this is the one that has this information okay so let's go here and we will write with open and we will write polygons and we will write rb for the axis and then we are going to write as f and then we will write polygons paulie gonz with score is equals to pickle dot load and we will load in f and here we can write the length of our polygons polygons with score just to make sure that we are heading in the right direction we should have two elements here so let's run this let's run the main and there you go we have two elements so this means it is important fine we can even impor we can even display this actually we should not delete that let's keep the length uh the print yeah so we can print it out to see if we are getting the same thing yes so here you can see we are pretty much getting the same thing uh from our file polygons if you open it up you will not understand anything uh so here you can see this is the data that we are getting from this file polygons and it is exported properly from here and it is imported properly over here so that is good news so now what we can do is we can check where exactly are these points and we need to check if our ball detected is hit within these polygons so it might seem confusing but it is not it is very simple to do so how can we do that so after we detect the contours and we have found a hits once we find the hits we are going to check if this hit is in one of the polygons or not so we will write here for poly poly in polygons with points with score we are going to get our poly and what we will do is uh we are going to check whether it is inside our polygon or not to check this first of all we need the center point the center point of the contour that was found so the contour found is here so we will write here contour found and we will get the first element there might be multiple contours which in case is not very relevant at this point but uh it will get the biggest contour in the first location this is by default what find contour function in cv zone does not in opencv in cv zone it sorts out the biggest one at first so we will put it at 0 here and then we will write here center we want the center information of our contour and we will store it in center so that is good we have the points which we want to check now we need the polygon in which we want to check if this point is inside it or not so to do that again we have the polygon but we need to convert it so we will write here that our polygon let's call it p is equals to numpy dot array and our polygon is basically coming from here so let's say poly score okay let's say polyscore and then we can write here that this is our poly this is our polygon and we are getting it uh from our [Music] poly poly uh score at 0 which means this is the first element so if you remember when we are dumping it we are putting the path which is the polygon and then we are putting the score so the first element is the path the second element is the score we need the first element now which is basically our path so that's why we are writing 0 here and once we write 0 here then again as you know we have to convert it and we need to make sure that it is numpy dot int 32 so that is basically why is it giving an error oh i removed one of them that's bad okay so now our polygon is ready and now our center is ready now we need to check whether this point is inside this polygon so how can we do that we can write here cv2 dots points polygon test and we will given our poly polygon and then we will give in the center points and we will write here false so this will check uh if the this center point is within this polygon and we will call it inside so it will give us yes or no true or false so what we can do is we can print uh in inside okay so let's run that let's see what happens here which video is this there you go one means uh inside minus one means outside hit detected one means inside minus one outside so it's doing that again and again so that's good it means we are detecting the hits and we know that this polygon is inside this this ball is inside this polygon so once that is done what we need to do is if it is inside we need to do a few things first of all we are going to save the location of the ball hit then we are going to save the location of the center point and then we are going to save the location of the polygon when we have all these three stored in a list later on we will draw them we should not draw them only when we hit when we detect the hit we should draw them all the time even after the hit we need to draw them so we cannot draw them inside this if statement we need to draw it outside this if statement so we need to store all these values inside here so that we can later on draw it continuously i hope that is understood so here we will write if inside inside we can remove this 1 means true so we don't need to write equals to 1 or something like that so we will call we will create a new list and what should we call it we will call it draw ball or let's say hits draw ball info list so once the ball is hit uh the drawing information will be inside here so we can draw again again so inside that inside this list we are going to append uh at three things we are going to append the contour found at zero we want to append the bounding box information so that we can draw the bounding box then we need to append where is that yeah this one here okay so then we need to append the center point so we can draw the center point as well so we can write here center or we can just write this center here it's the same thing and then we need to add our poly so we can draw the polygon itself later on so these are the three things that we have now and now what we will do is we will draw it uh on our polygon on our image now before we go out of this loop the last thing we will do is we will increase the points so whenever we detect the hit we need to increase the points whenever it is inside so what we will do is we will say that our total points or let's say total score plus equals plus equals poly score at one so if you remember this is uh element number one zero this is element number one so we are getting this element from the list polyscore this is the poly score where is it this is the polyscore from polygons with score so total score we did not define i think we named it something else did we name it no we didn't so we write here total score is equals to zero so we will go down here and now what we can do is we can print the total score total score score prince total score so let's run that the total score is zero when it gets a hit let's wait for the hit there you go one so there were two hits and three hits four hits so every time it is adding 60. so that is the basic idea so let's go ahead and what else can we do yeah we need to draw now so pretty much everything is working at the back end but what we haven't done is we haven't drawn so what we can do is we can directly draw on our image so what we will say is for let's say um what was the first one pounding box so bounding box then the second one was center and the third one was poly in what was the name hit draw ball info list these are the three things we are getting from this list and then we are going to draw them so cv2 dot rectangle rectangle and what do we need to input we will put it on the image contours and then we will given our location which is the bounding box and then we will give in the color let's make it purple 2 5 5 0 2 5 5 and then we will put it at 2 the the thickness and then we will draw the circle cb2 dot circle circle and we will give it to image contours uh image contours and we will write in the center is the center position the radius let's put it as five and the color let's put it as green two five five zero and we want it filled so cb2 dot filled the idea and then we are going to write cv2 dot draw contours contours and then we are going to draw it on our image contours and then we will give in our polygon which is poly and then we will draw all the polygons so we will write minus 1 and then we will write color is equals to zero two five five zero green and then the thickness will be uh thickness will be cv2 dot filled so this is how it should work so let's run it and we will open up image contours this is the main image so let's wait for the hits this is the first oh this is the first hit oh this is the second hit and this is the third hit the third hit it didn't go in here it is on five that's why it didn't draw otherwise it would have drawn here why are there so many windows we need to go we need to do damage control let's um even even the original image we don't want we don't want the mask image we just want image contours at this point and we were yeah we were outputting an image here and then we were we were input outputting in a loop somewhere yeah this one let's remove that okay yeah now it is one clean image so hits didn't work uh let's do it to the other one so let's go and do it to video number three let's try that there you go it hits and it is drawing here why is it drawing here so something is wrong because it's it's not in that area then why is it showing that okay so let's go and check out what the issue is so if it is inside then this is are we printing out the score no okay if it is inside then we are appending the bounding box info the center and the poly yeah that's fine and then based on that we are drawing each of these every iteration yeah that seems fine to me do we need to reset something not really hit 0 is already resetting so we are already resetting the hits maybe it's not going inside we need to print this out uh or let's print out the score if it if it's actually detected print score what is uh it's called total score yeah so total score is 120 how is that 120 this is detecting wrong because it's inside here and it's not okay let's just let's just remove this part and let's see okay so it is hitting but the issue is that it it is thinking that it's inside when it's not so here we need to print or before that we need to print inside print inside we are printing so many things we need to remove all the prints so that we can see what is happening there's another print somewhere print with this let's remove all of them this is the one that we want yeah let's run that minus one minus one so it is minus one is it reaching inside this print print yes how it is reaching even though it is -1 so we need to put here equals to one let's see if that works yeah now it works so even if it is it is hitting it's not saving that location so that's good so which one were we using video 2 video 2 let's run that yeah so now if it hits it detects and it colors that part it didn't hit here so it didn't detect and now i believe the the numbering the scoring will be correct as well so if we go and print the total score then we can see what do we get so right now it's zero so you can see even if it uh falls down it doesn't detect it as a hit okay so this numbering is wrong just detecting 60's 120 why maybe it's detecting as a double hit so what we need to do for detecting as a double so it's going fine from 120 to 180 but 60 to 120 was too fast so maybe we need to increase this to 10 so yeah 60 and then 120 perfect that was the issue it was uh detecting multiple times okay so what we can do now is uh we can actually now the thing is it doesn't look good when it hits it it's fine but covering the whole thing up it doesn't look that nice so what we can do is we can keep it transparent so that people can see the back as well and you might think that it is quite difficult but it's actually fairly simple all we have to do we have to create a blank image image blank is equals to numpy dot zeros and we will give in the shape of our image contours image contours dot shape at 0 and then image contours dot shape shape at 1. so we are basically creating a matrix of zeros of the same size of our image contours and it is three channels rgb so we have to give in three and it is unsigned eight bit which means zero to two five five so we have to mention that as well so unsigned integer eight bits so this blank image we are going to draw on this blank image so we will copy this and instead of drawing other places we are going to draw on this so uh no no actually not this the draw contours the polygons we want to draw on that and once we do that we are going to do an added overweight uh not overweight added weighted add weighted so we will write here cv2 dot add weighted and we are going to wait in our image contours or is it image contours yeah image contours or let's use image board image board and then we will use 0.7 and image blank this will be 0.5 and then we have 0 the gamma if you did not understand i will explain now image contours is equals to this or should it be image board yeah it should be image board let's let's make it image board and we can output image board as well image board and we can actually stack them together as well let's just run this now and there you go there you go so now it doesn't actually it overlays on it it doesn't actually remove it and here on the side oh we need to we need to open this again the frame counter so on the side you can see that it shows the hits and the ball detected and the center here there you go and then here it shows which which place did it detect so that's pretty cool and the points we can display the points up somewhere here so we can actually stack these images as well at the end we can stack them here so what we can do is we can write here image stack is equals to cb zone cv is on dot stack images and we give in image contours and we give in image board and we want two columns and we want the scale to be one so we can just show image stack instead of showing two different images so if you run that now it is single image looks better and it's easier to handle so yeah there you go that is looking pretty good and yeah so it didn't detect the one above because it is not in the correct region because we have only done for this 60 this 60 and this 60. now we can go back and do for all of them and then we can return to this and see if it works or not so we will go to path picker and here we are going to run it now this will take some time so you can do it by yourself or you can watch me do this here so let's go ahead and label these so first of all we will start with five so you don't have to be very precise you can just press q and it will ask for the score that is five then we will go to the next one so this is going to take a while but not as long as if you had to do it manually through what you call paint and then write down all the values that will be a really painful process so well this is painful as well but not as painful as that so but make sure you don't make any major mistakes here otherwise you have to repeat the whole thing so that is 10 and then here is 5. uh q and five yeah so don't don't add this this is empty so we can put it here if you add it by mistake just write zero as the value this one is 5 then this one is 10. so this can be used in other projects as well if you have a different kind of a dart board or if you have a different game this can be used for many other projects for marking so hopefully this will help you in other projects so we have done one of them so we are not going to stop we are going to continue with the same and we will write it down here 60. so we will keep going in and in but right now we are doing 60s again i'm not trying to be very accurate then we will do 60 here again so don't forget to press enter after you write in the value 60 so 60s are done now we are heading to 80s 18. 80. this is the last 80 we have and the last one is hundred so we have a few of uh the edges so we will select those and there you go we'll put that in and we'll write hundred and now is the main parts we click on the top and then we click on p to save and there you go so we stored 20 polygons and each of them has their scores okay so now we can go to our main file and we can simply run and hopefully everything is already there and it will work perfectly let's hope for that so the first hit 60 that's good second hit 60 third hit five excellent i love it so uh what we need to do now is actually to write down the score because the score is not visible so that part is actually fairly simple uh where should we write the score should we write it on the contours or the image board i think on the contours because that is the place where we are doing the highlight as well so let's put it on image contours so here we are going to write so now we are going to use the cv zone put text rect so a normal text will just have a text and nothing at the background but with a cv zone put text rect there will be a rectangle at the back of the text so it will be easier to read so we will write here cb zone dot put text rect we will give in our image contours we want to draw on that and then we will write an f string so we will write total points and we will given our point values so total actually let's call it score i'm always mixing those up score total score and then we are going to give in the position let's give it 10 and 40 and then we will give it a scale the scale is equals to 2 and the offsets let's keep it as 20. you don't have to give in all these values uh it's just that this will look a little bit better you can also change their colors but i'm not going to do it right now but if you wanted to you could and again you have to store it back into image uh contours not contours we are doing it on the board so image uh port yeah wait which one is the yeah we are doing it on image board p o a r d is equals to this and this should be image board as well let's push this down so we can see uh yeah so let's go ahead and run it and hopefully this time oh there's an error yep there is an error bad argument what happened there did i miss a bracket or something here image board and image board let's try it again yeah still the same problem uh so if i remove this line oh if i remove this line does it work yeah it works so that line is called oh yeah i remember now yeah yeah yeah i forgot so basically uh this just does not uh output one image it outputs an image and the location of the bounding box so we don't need the location of the bounding box so we can ignore that by writing underscore and there you go wait why is it giving it on that it should be on the board oh yeah that is the board yeah my bad that is the board okay so let's run that uh but let's do it after the waiting because otherwise the colors they can tell they get done so we don't want that let's do it after that yeah there you go total score zero and one two three go 60 excellent 120 excellent 125 excellent i love it so this is basically and now it's repeating so it will keep adding the score which well it would not make any sense but that's the idea so here you can see the center points where they are drawn and this is the location of the hit so let's let's try out the other video where is video number three let's try that five it didn't detect the five oh that didn't detect so the reason it's not detecting these two is because if you look at the center of this it is outside it is on the black line this black line so that's why it is not detecting that so it is outside that so it's not detecting this part this part is not detected because it is in the red zone which has no points this part is detected because it lies the center if you see the screen point lies within this 60 region so that's why it is detecting fine so i would say that is a win-win situation uh i really like this video the second one it shows that when it hits and drops it doesn't detect it and then when you hit it on one side it detects it the other one detects it and the upper one detects it as well so this was basically our project for today it was a fun game that we created using basic image processing techniques and what we did at the end is counted the score where that where did the ball hit and what was the actual score so this is it for today's video i hope you have learned something new if you did don't forget to like and share and subscribe and i will see you in the next one

Channel: Murtaza's Workshop - Robotics and AI

Views: 78,758

Keywords:

Id: tVe3r6186OA

Length: 110min 1sec (6601 seconds)

Published: Mon Nov 08 2021