I tried building a Computer Vision Game in 12 Hours

so a lot of people think that computer vision is not a good field as it has no real world applications so let me tell those people that it is very real and it is one of the best ah come on so today i will try to build a computer vision game that will be applicable in real world application in just 12 hours so let's see how that goes the idea of the project is that it consists of a camera a projector and a computer the computer will be running a game where the balloons move up and you have to pop them to actually score the camera would use computer vision methods to find the position of the balloons and check if they are hit or not if the balloon is hit it would send the command to the game and it would pop the balloon and generate a new one so let's see how we can do this so the first step was to put all my thoughts together and this was not a difficult task because frankly i don't have a lot of thoughts to begin with i used canva for this because it is very easy to use and you can find a lot of pre-built templates there because i'm not a designer i have one simple rule when it comes to design hey iss keep it simple stupid great advice hurts my feelings every time this means we do not have to go deep and get complicated all we have to do is simplify the process and get up and running so this is what i did i created a simple layout with a white background and i added some balloons and some text for the score and for the time so the next step was the choice of balloons now i had to be a little picky here because the choice would reflect directly on the image processing techniques i decided to go with the balloons that had no edges so that it does not interfere with the image processing techniques the next step was to list down all the appropriate techniques that we can use in this project here the two main categories were artificial intelligence and plain old image processing with artificial intelligence we have an end-to-end approach so here we will collect the data of different balloons and then we will train a model to detect these balloons and we will use a classification model to classify whether an object or the balloon is hit or not but this seemed like an overkill so i decided to go with simple image processing here i split it into two parts first was to detect the balloon and the second was to detect whether the balloon has been hit or not the first approach we can use is the color filtration here we can filter out the white background and we can find the different colored balloons but this technique is not very robust because it gets affected by the change of lighting the second method is using edge detection here we use the edges to find the boundaries of the balloon and then we use this information to get the bounding box so that we know the location of each of these balloons so it was quite clear that the second option was more feasible so i decided to go with that to find the hit within the balloon i decided to go with simple circle detection at this point i started to daydream and i started to think about all these different ideas i could implement within this project like a point system based on where you hit so if you hit in the middle of the balloon you will get more points if you hit on the side you will get less points but all of this was way too soon so i simply shut my brain and i continued for testing i created a few images that had a few balloons with the hits and a few balloons without the hit so this would allow us to test without actually implementing the setup of the projector and the camera this whole process took about one hour to complete so now it was time to put on the nerdy glasses and start typing the code so here the idea was to write a main script that will handle the pipeline of the complete project and then we will have separate modules for detection of the balloon and the detection of the hit within the balloon the modular structure makes it easier to write the code and also it makes it easy to maintain it later on starting with the balloon detection the goal was to find the location of the balloons so first i applied some pre-processing techniques to make it easier for the balloon to be detected and then i cropped the upper region to avoid detecting the time and the score text so now the image was ready and all i had to do was apply the edge detection here the boundary of the balloon is detected and a bounding box is drawn around the edges now that the location of the balloon was known it was time to find the hit within the balloon so i created a new hit module that handles the logic of detecting the hits within these balloons first we crop the balloon regions based on the location provided by the balloon detection module now using this information we get these images and we find the circles within these images to detect the hit here i added a size filter and a restriction to detect only one ball at a time this makes the process a little bit smoother and a little more accurate now this process requires a lot of tuning and i did not spend a lot of time here because this is just a dummy image and later on we are going to use the actual webcam feed coming off from the projector so tuning this would not make much sense now the concept was tested and we had to go ahead and test it on the actual camera with the projector when using the camera the image is usually at an angle because the camera cannot be pointing 100 straight to compensate for this tilt i created a calibration module that allows us to select the edges of the projected screen and save the information in a calibration file then this calibration file can be used to run the actual game now the image would be quite straight now this process could be automated as well but because of the time limit i did not want to get into that much detail now came the hardware side with the projector i did not have a lot of choice because i already had a projector installed and it was ready to go now the projector i'm using is the absent home cinema 2150 now this is an entry level projector and it is not very expensive i got it about four years ago for around eight hundred dollars and the good thing about this projector is that it can provide 2500 lumens which makes it very bright and usable in a daylight situation so you cannot have it in a very bright light but with a decent amount of light you will be able to see what is happening on the screen now i'm not going to lie but till this point things were going really well actually more than what i had expected but then as soon as i got excited i started facing some problems so now was the choice of the camera my first choice was the logitech c920 webcam now this is an hd webcam that allows us to get a very good quality image but the problem was that for some reason i was not able to get it running with pie chart it was running with its own software but as soon as i ran it in pycharm it would not work next in line was the wide-angle genius webcam it was also 1080p but this did not have a nut to actually mount on a tripod so i had to get a little bit creative to get it working now this camera worked but boya boy the image quality was bad it was really really bad so now i had to bring in the big guns i brought in my dslr camera that connected to my computer and i got finally a very good quality image using the testing image i could see that the balloons were being detected very accurately and the hit was being detected as well so at this point i did not go for tuning i just went ahead to the game part since i wanted the game to be within the same python environment the obvious choice was using pygame now pygame is a very popular library for creating games in python but the issue was that i had no prior experience with pygame apart from creating small python scripts here and there after some searching i found a blog written by harrison kinsley aka sendex this was a great starting point as it showed how to import an image along with all the basic requirements so i built upon this code and after a few minutes i had my balloon imported and it was moving around at this point i had the balloon going up at a particular speed but there was no logic of popping the balloon and spawning it now in games we usually create objects and if we want to remove the object we usually destroy it but this type of approach seemed a bit hard to achieve in that limited time frame so instead i moved the position of the balloon once it popped or moved out of the screen now at this point i had spent almost 11 hours building this project and it was about time that i tested it out with the game ready and the image processing working i took my phone and recorded the testing footage and here it is okay so now i have finished with the coding and i haven't tested yet so here is the projector the projector is projecting right on the wall and we have our code over here so the first thing is to calibrate so we are going to run this and you can see that we have let's run that again okay so we need to calibrate we will select the points so now it's calibrated and now all we have to do is we have to run the balloon game so let's try that out okay so now the game is over here so i'm going to shift it to the projector over here because this is just a static image so i'm going to take this and it's here and now it's there there you go so now we can see the game has been played but uh let's check it here if it's detecting properly so here it should show us the balloon so um so i can see that if it's not completely on the screen it will not show us the balloon so we need to fix that but when it's completely on the screen then it will show us the balloon so we can fix that later but let's test now i have a ball right here so i'm going to test with this so whenever it comes up and let's try it and there you go so on the first try that that is amazing so let's check this okay so we can see here that it wasn't exactly there were two of them this was the shadow and this was the actual ball and this is where it hits so it seems that it is good now we need to fix some bugs and let's see how they work out okay so after some testing i have found a few bugs so you can see that if i hit on the side it will pop as well and the reason is because i have a reflection so if you look closely there will be a shadow there you go so the shadow actually comes on the balloon and that is detected as a hit as well so if i hit just close enough it will give the shadow on the balloon and why that happens is because my projector is not in the center of the room so it is on the side so if i'm hitting from an angle it will cast the shadow and it is also because of the lighting so therefore it is very hard to distinguish between that at least at this point and the second thing is the score the score is just going crazy and this has something to do with the shadow as well we can make it a little bit harder to score but because this is for kids i don't want to add too much difficulty so even if you are heating very close and the shadow accepts that i'm fine with that i don't have an issue but later on if you place it in the middle then the projector will not have that issue so the result was amazing even though not perfect but the logic and the game worked together to build an amazing experience the next day i came back with a fresh perspective and added some background music along with the popping image and the popping sound i also fixed the scoring system and tuned it a little bit more to get some better results this took about two more hours and then it was all done and now for the real testing i called in the big boss [Music] okay so she needed some help i mean who doesn't as you can see the project turned out quite well and even though it took two more hours than the limits i still think it was worth it to spend all this time to create something that is useful and that actually is quite fun so i hope this inspires you to create your own computer vision projects and do let me know in the comments what did you think about this project and this is it for today's video and i will see you in the next one

Channel: Murtaza's Workshop - Robotics and AI

Views: 267,347

Rating: 4.9655914 out of 5

Keywords:

Id: i_UiTIr4jTk

Length: 15min 33sec (933 seconds)

Published: Sun May 16 2021