Dominating an Online Game with Object Detection Using OpenCV - Template Matching.

what's going on clarity coders sorry about the way today we're going to be jumping back into opencv and taking a look at template matching template matching is a technique we can use to detect images inside of another image without training a neural network i'm going to walk you through a simple example line by line in a jupiter notebook to explain and show you the technique then we're going to apply that technique to a simple game to see if we can get clarity coders on the leaderboard and kick your chop can we top that leaderboard let's jump right in and find out [Music] now in this part i'm going to walk you through a simple example of template matching in opencv so for this we're going to use opencv obviously and numpy to get our example done so if you haven't already you want to copy in this line which is just a pip install opencv now if you've been following along with the tutorial series you should already have this done so you'll probably see something like this where it says requirements already met then we can pull in the two libraries that we're going to use for this project the two libraries that i'm going to use for this project are cv2 and numpy now in the later applied template example we use some other libraries to detect keystrokes and things like that but in this example we're just going to use these two libraries and you can hit shift enter to run that as well now inside of the github i'm going to have the images that we're going to use now typically what you're going to see when you do template matching is you're going to see a source image now you can imagine that as a live camera feed or some type of original image that you're looking for another object inside of so you've probably seen ones where you detect a mask or you detect an eyeball or whatever it is out of the image we're going to call these our farm image and our wheat image so if you look at these images it's just a simple game like a farming game and we're going to try and pick out the different instances of wheat inside that image now this is a little tricky because some of the wheat images are covered up by other things and sometimes there may be different shades maybe the game turns from day to night or something like that and you can imagine this in a real world setting as well if you want this notebook this code or these images they're all on the github in the lesson three folder so you can grab them from the description we're gonna run shift enter and pull in our two images now these are in the same directory as the script so you'll notice i'm just referencing the image the file name directly and then to show these images i'm going to create a couple more blocks of code here this is going to be to show our farm image you can see that we do cv2 weight key now that's going to allow it to stay up until we hit a key and then destroy all windows this is something that i have to do in jupyter notebooks notebooks so it doesn't freeze the notebook so if we do shift enter here you can see it appears just to hang but if you look down in your taskbar you should now have an image pulled up and you can see that this is going to be our original image and we're going to try and detect wheat images out of it so now what i've done is i've created a needle image what a play on needle in the haystack so we're going to look for an image inside of this image now what i did was straight out of this image inside paint i just cut out the wheat image that we're going to look for now my hope is that we detect multiple instances of wheat and then we can do something based on that information maybe if it's like a security camera or something like that we throw an alert if it's something like you're creating an automated game or whatever it is maybe you're sending you're clicking in that location to harvest the wheat next let's show our needle so the needle in this case is this tiny image here you can see it is literally something that i cut out of our original farm image so that's what we're going to be looking for we'll close that as well and we'll we'll reference these back a couple times when we're going through this demonstration now for template matching what we're going to do is we're going to use different algorithms that i'll show here to try and see a match in our image so we're going to try and pull out features in a match now i'll note the docs here and you can read up on each different type of method and there's a lot of mathematics that goes along with it but essentially what we're going to do is slide our needle image across our original image and try to find a match now for each of those locations that we test we're going to get a value back that's how close of a match it is so what you're going to see is when we originally do this you'll see almost like a heat map that shows where the best matches are so let's go ahead and take a look at what that looks like here so what we can do is we can choose one of the six template matching tech algorithms to use and then we can run it and get our result so what i'm doing here is i'm doing cv2.match template i'm doing the image that we're looking for the needle inside of which is our farm underscore image and then our needle image itself so this is what i'm looking for which is the wheat image we can run this and i've also picked one of the algorithms to use you can use whatever one you would like if you're doing this in a real world project i would suggest testing them all and seeing what works best in your case now this is pretty cool we can actually grab that result and show it to the screen just like our other items so what i'm going to do here is i'm going to do i am show again with our result the same exact weight key destroy all windows and if we look at this you'll see at first it doesn't look very useful but if you look a little bit closer at this image you'll notice it's a black and white image and there's some white markings along it you can tell where it's lighter and where it's darker so these light images are where there's a closer match now if you look closely and overlay our original image here you can see that it picks out five pretty distinct white points now those white points are where the wheat is so we know we have something going here this is a pretty good it's it's finding some pretty good results right now so if we want to get our best matches back we can use cv2 dot min max location on our result so what this is going to do it's going to return four things it's going to return the minimum value so that's the worst match of all the matches the maximum value which is the absolute best value of the best match it's going to return the min location so that's the location of the worst match and the max location which is the location of the best match so if we run this we're going to get some values back and one of those values is our max location now this is very useful this is the x y coordinate of our best match so with that we can draw a rectangle around our original image to highlight where our best match was so let's go ahead and take a look at that and actually before we take a look at that let's also look at our max this is our max location let's also take a look at our max value now you'll notice that our max value is a 0.999 percent match which is really good right which makes perfect sense because i literally copied and pasted the image from it so it should have a pretty good match so we're going to use that later so we're going to later we're going to say a threshold of what is a good enough match so obviously in real world examples you're not going to get a 999 match very often so what is a good enough match that we still think it's wheat that's what we're looking for here so with our best match we're not quite to the multiple matches yet let's try and draw a rectangle around our best match now if you remember when you're drawing rectangles we need to define a top left point and then a bottom right point well we have our top left point here in our max location we need to define a bottom right so one thing i know is how big my wheat needle image was so i'm going to get the width of my needle image the height of my needle image and then i can add those values to my max location to get the bottom right point on the image so now that we have the the width and the height let's go ahead and create our rectangle so to create our rectangle you might remember we're going to do cv2 dot rectangle we're going to pass in the image that we want to draw the rectangle on we're going to pass in our top left point which was defined up here in max location and then we needed to define our bottom right point now this is a little different so we're going to take the x value out of max location which was in the zero index and we're going to add the width to it then we're going to take the max location that was in the 1 index which is the y-axis and add the height to it then we're going to draw a box around that and this is just color and line thickness if you remember from our last video now we can do shift enter you'll get some output here and if we go back we can display our farm image again so i'm going to go up to the cell that had our farm image and you can see that our best match was right here which is probably where i cut the image out of now you're saying great we found the exact image that doesn't do us a lot of good right we want to find similar images if we're creating a farming game or if we're saying do something when you see a mask it's not going to be the same mask every time or something like that so what we're going to do is we're going to now define a threshold so we can hopefully highlight all the ones that are close enough matches that we think it's wheat i'm going to close this now remember every time we do something different we want to reload in our images so i'm going to run this cell and i'm doing that so we no longer have our yellow marking on that and now we can go ahead and create our threshold now i've done this ahead of time so i'm just using a threshold that i know works pretty good at finding the different images you're gonna have to play around with that in your example it might be 85 it might be 60 maybe it's 40. so what you know is if you have too many false positives you want to try and increase your threshold make it harder to find a match if you don't find all the things you want to find you might want to lower your threshold to find more matches so what we're going to do is we're just going to use a threshold of 60. now we can do some numpy arithmetic here comparison here and we're going to say where our result is greater than or equal to our threshold we want to keep those locations now remember that's going to turn return two different results it's going to return a y location and x location and then they're in the opposite order you would think so what we're going to do is we're going to receive in some y locations and we're going to receive in some x locations here now if we look at one of these we can see about how many matches we have so you can see that we got 13 matches here we have 13 x locations and presumably hopefully 13 y locations so now to draw our 13 matches which you're probably thinking that's too many matches right he said this worked pretty good hang with me for just a second so we're going to draw our 13 matches what we're going to do is we're going to do a little for loop here we're going to do 4 x and y you'll notice i put them in the correct order this time in zip x location comma y location so again i'm putting them in the correct order x y coordinate and i'm zipping them together so i'm pairing each one together so the first item in the x location is paired with the first item in the y location to create our points then each time through this i'm drawing a rectangle on the x y coordinate i'm doing just like what we did before from for on the x coordinate i'm adding the width on the y-coordinate i'm adding the height and then i'm drawing the square now if i run this i can go back up and i can show our image again and you'll see that we got all five of our wheat images and you may ask yourself well we did get all five but it said we had 13 matches you'll notice some of these have a thicker square around them than other ones and that's because it's finding multiple matches now maybe this isn't an issue in your program or maybe it is so what we want to do is show a way that we can limit it to just the one match so essentially we want each five to show up but we don't want multiple highlights over the same patch of wheat and that sort of thing so we can use another algorithm built into opencv that allows us to group these together so what we want to do is we want to create a group of rectangles and then we're going to use a function that's built in to combine those rectangles if they overlap so what is a rectangle well in opencv in this algorithm it wants an x-coordinate a y-coordinate a width and a height for each rectangle which is pretty easy that's basically what we have anyways so we're going to create a blank list to hold our rectangles and remember here we're trying to look at another way to do this so i'm going to reload my images so you want to reload that image so we're back to a clean slate here and this time each time through our for loop instead of drawing the rectangle right away we're going to append the rectangle to our rectangles list so you'll notice we're doing rectangles.append each time and inside of this we're using square brackets so it's a list inside of a list and we're defining it just like opencv once x y width and then height and each one i'm converting to an integer to make sure we're not going to blow up this function down the road now one other thing you'll notice is in our case not every rectangle was duplicated one had just a single square over it that's going to be an issue when the group function runs it's going to want at least two rectangles to group them together and that's not the effect we want we wanted to find all five even if a single rectangle is over it now i've looked at the code and i've looked at a couple other forms and there's basically one workaround to do this and that's to duplicate the rectangles so we're gonna make sure that it has at least two on it by putting every rectangle in there twice so we're literally gonna copy this line paste it below and that will duplicate the rectangles as it puts them into our rectangle list so if we run this then we can run length of rectangles and you can see that we got 26 which makes sense because we did duplicate those now we can pass it into our grouping function so i'm going to grab a couple values out here and you can read up in the documentation on these actual parameters and play around with it to meet your match so basically what we're saying here is how many rectangles do we want in a group we want just one so it's going to pair those down and then there's a coefficient here that you can play around with it'll basically say how close it has to be to consider it a group so how much of an overlap is okay to group them together this is the parameter that i found was useful go to the docs right here and you can play around with these values as well in your own problems so we're gonna go ahead and run this and now we can take a look at the length of our rectangles again and you can see that we're down to our five which is very promising now let's copy our function from above our for loop that drew the rectangles and instead of iterating over our zip this time we're going to iterate over rectangles and we're going to unpack from rectangles each time an x a y a width and a height now we can run this let's slide up and look at our farm image and you can see perfect we got five exact matches that look perfect for our image so that's exactly what we wanted now this is a very simplistic example but it gets you started it shows you exactly how to do template matching now i wanted to do a fun little program just to highlight this even more so what we're going to do here is we're gonna take a look at another example of template matching really quick here so i don't know about you guys but in my childhood i like to play a lot of these silly flash games now they're mostly html5 but we're going to use that to show another example of template matching here this is called kick you chop so the concept of this game is really simple if you click on the left side you break a branch off of the left side and your guy stays on the left side if you click on the right side he moves to the right side and you need to avoid the tree branches that come down i'm going to show you my best run here or one of my best runs on my way to getting the highest score and kick your chat the program's really simple it's just gonna test using template matching to see if there's a branch above our man's head and if it is we're gonna click the opposite side [Music] you can see here that it's testing above the player's head to see if there's a branch there and when there is it's clicking on the other side so i'm doing this using screenshots with mss and then pi auto gui to actually click the screen either on the left or the right you'll see the mouse moves on its own [Music] now i simply took a section above the player to test so i needed to use a stick from the left side and a stick from the right side as my needle images [Music] if i'm on the left side of the screen if my player is on the left side of the screen i use the left image if my player is on the right side of the screen i use the right image and i only test whichever side i'm on because that's the only time i'm worried about a branch hitting me in the head [Music] now if you run out of time with the green bar on the top you lose and also if you're on the same side as a branch comes down like here you will also lose that's it for this time everyone thanks for watching if you haven't please subscribe it helps the channel let me know in the comments if you want to see anything special or in particular and until next time keep coding you

Channel: ClarityCoders

Views: 308,972

Rating: 4.9306951 out of 5

Keywords:

Id: vXqKniVe6P8

Length: 20min 25sec (1225 seconds)

Published: Mon Apr 19 2021

Reddit Comments

[u/Abalado]

Do we even have a canonical definition of object detection? Because if it's not, all hate towards the title is just waste of time.

[u/Dwigt-Snooot]

For those mad at wording, it is a form of object detection per OpenCV. Which is the most important source sense it's an OpenCV tutorial. Of course, there are better and way more complex versions of object detection that's the point of the video. To show an example of a simple use case for simple needs!

https://docs.opencv.org/3.4/d4/dc6/tutorial_py_template_matching.html

[u/[deleted]]

Subbed for sure gonna watch later.

[u/NLonSec]

man i should really brush up on my python, there's so much cool shit you can do with it

[u/wlcosta]

Template matching is not object detection. l

[u/soulwarp]

I thought it was a great video. I'll be going over the tutorial again when I can get on a computer. Like/subbed