Thresholding with Match Template - OpenCV Object Detection in Games #2

with thresholding we can detect multiple objects using opencv's match template function in the official tutorial they discussed a little bit how to do this but it's very brief so if you have any questions about how thresholding works with template matching i'll be going more in depth with it in this video i'm also going to be using the debugger in vs code so that should be interesting if you've never seen how to use that before hey i'm ben and this is part 2 of my opencv series in part 1 we used the min max location function to get the best matching position for our needle image but match template actually returns a result matrix for all of the positions that it searched so the idea behind thresholding is instead of just checking the best match location to see if it's above a certain threshold we want to try to get all the locations that are above the threshold that we set you can also think of it like this instead of just trying to get the position of the one brightest pixel we'll try to get the positions of all of the white pixels above a certain threshold of whiteness so in the case of our forum image where we're trying to match all these cabbages we're hopefully going to be able to get the position of all of these white spots these bright spots that correspond to the cabbages in our original image so starting with the code that we left off with last time i'm actually going to delete most of this i'm just going to keep the match template and the image reading parts and let's take a closer look at the result that's returned by match template so i'm just going to print that out and when you run this you'll see that match template actually returns a multi-dimensional array it's got a bunch of numbers in it and each one of these numbers represents a confidence score for how closely the needle image matches when placed at a certain position on the haystack image because this result matrix is so large numpy will actually truncate it for you when you print it out like this it'll add these three dots to indicate that there's a lot of missing data so if we wanted to see all of our data we have two options one thing we could do is we could change the numpy settings by calling set print options and by importing sys we can tell it to print basically everything that's in our data to the console but you want to be careful with this if you have a lot of data then doing this will actually lock up your process and in our case the data that's in our result array is really too large for the console to handle so the other option we have is we can try using the debugger so i'll go ahead and delete that code then here after we call match template and the line below it i'll put a breakpoint and so clicking there indicates that we have set a breakpoint for our debugger and then i'm going to go over to this run tab with a little bug on it and here there'll be a button to run and debug so when we run this i'll choose to run it as a python file and as our program is executing when it gets down to this line where i've set the breakpoint it'll stop and it highlights this line for us and it also shows us all the variables over here in this window on the left so i'm going to move these windows around a little bit and we want to take a look at the result variable so down here is the result it's still truncated here so we can expand this and then we can keep expanding these items further to get into seeing you know in the first row of our data what appears there and we can actually step through all those individually as well but of course there's almost 2000 items here so it would take you a long time to go through all of it but if you wanted to know the confidence value at a specific position say 0 for the y and 22 for the x this would be the confidence value there so we got 0.1311 for that one so the way that match template returns its data the first dimension in that matrix is the y position and the second one is the x position and when you're done with your debugger at this step you can press play to continue the execution or you could step over lines individually as well and just to make this extra clear in case you don't have a debugger to work with i'm going to swap in a much smaller image for our haystack in our needle so that we can print and see the entire result from match template so i'm using this super tiny picture of copper in the game albion and out of it i cut an even smaller needle so that should be plenty small enough for us to actually see the result if we print it for match template so when we run this now we can see that we do get the full matrix printed out and so this first value here this 0.65 that's the confidence that the needle image matches the haystack image at that position and that position is where y is 0 and x is 0. and so this first list is the entire column where y is zero the second one is the column where y is one so you can imagine match template is taking this needle image and it's moving it across every spot that it can place it inside this haystack image and it's giving us a score for it and that score indicates how closely the needle image matches up to the haystack image that's beneath it if you imagine them kind of like overlaid on top of each other and you'll notice that the size of this result array isn't exactly the same dimensions as your haystack image that's because if you were to take your needle image and place it towards the bottom side or the right side of the image kind of overlaid it part of the neat language would kind of hang off the edge so it's kind of ambiguous what a meaningful match value would be when the needle image isn't fully on top of the haystack image so to handle that edge case opencv decided to just not include those values in the result i'm sure it's not even doing that comparison and if you look at the match template documentation that's what it's trying to say here when it's talking about the size of the result matrix that you're going to get for example our haystack image here is 27 by 16 and our needle image is 7x7 so you expect the result matrix to be 27 minus 7 plus 1 so it'll be 21 in the x dimension so each one of these inner lists should have 21 items and i just counted them up and they do and in the y dimension we should have 16 minus 7 which is 9 plus 1 which is 10 so we should have 10 for the outer dimension as well so now that we understand our result matrix a little bit better what we want to do is we want to get all of the positions where the confidence value the scores here are above a certain threshold that we set so if i set my threshold to 0.8 i would want to get all of these 0.8 values all these values that are greater than 0.8 but none of the positions for the values that are below 0.8 and to do that the documentation suggests that we should use the numpy where function so i'm going to set my initial threshold to 0.85 and then using np where we're going to get a result for all of the locations above that threshold now let's go ahead and print out that location's result to see what we get so when i run this you can see that we got two locations above that 0.85 threshold the first array here are your y positions and the second one are the x positions so the first one is the first dimension of that matrix followed by the second dimension of that matrix and in this case we found it at two positions where x is one and y is one and then also where x is fifteen and y is five so that's great that we have the data for our positions now but the format that's returned by np-ware isn't super convenient so let's convert this into a list of xy tuples so to do that i'm going to use this magic looking line of code right here so i guess first let me show you what this line of code does and then i'll explain to you how it works so you can see when we run it first we had these two different y and x arrays that was returned by npware and then what we end up with is a list of x y coordinates so let me explain this from the inside out the first part's the easiest to explain these brackets colon colon negative one that just reverses the list i'll bring up a python terminal here to show you so if we start with a matrix that looks like this it's just a simple two-dimensional matrix using that reverse syntax it'll just bring this second list front and put this first list in the back so you can see it just reversed the first dimension of this matrix and the second part of this line of code is the zip star so star unpacks a list and zip merges lists into new lists of each item that's at the same index so star by unpacking a list is essentially just going to remove the outer dimension in our matrix so instead of having a single two-dimensional array we'll have two one-dimensional arrays and then zip is going to make new lists for us where it takes all of the items at a single index and combines them together so the first list will be 7 10 the second list will be 8 20 third list will be 9 30. so if we did zip star res and let's keep our reversal in there this will return a generator and so to see the result of that generator we can just wrap it in list so there you can see the result we get it's a list of tuples where we took our initial list we reversed it and then we combined the items from each list that share a common index so that's how in our code we were able to go from these two arrays and combine them into xy tuples okay so in our code i've gone ahead and switched back to our initial images i'm just going to play around with this threshold until i get a reasonable number of results back so at 0.85 i'm just getting back one result so let's lower it a little bit it's just .80 i'm getting back a few results they all appear to be very close to each other though so i'm going to lower a little more okay and so when i drop the threshold down to just point five i get back a whole bunch of results uh but not an overwhelming amount i'm curious to see what it's matching here and what this looks like so let's go ahead and use our knowledge from the last video and draw rectangles for all these locations that we found okay so this code should look very similar to before this time i'm checking if locations so as if we got location results from np where above our threshold and then we're going to need to loop over all of those locations to draw a box for each one and there are certain things that aren't going to change through each iteration of that loop so i went ahead and pulled those out before we do the loop so that is the size of the needle image the line color we're drawing and the type of line we're drawing and then for each location that we found the top left corner is just going to be that location and then we can calculate the bottom right corner again just using the size of the needle image and then i go ahead and use rectangle to draw the box and once all those have been drawn i am using i am show to display the matches so let's go ahead and see what we get okay so here it looks like we got five real matches and they're all cabbages so that's a good sign but of course in our results that we printed out we're seeing a lot more than just five position coordinates and in the result image some of these green lines are really thick so what's happening is is that we're finding a lot of different position results that are very close to each other and when we draw all those rectangles on top of each other it results in these thick looking lines i wonder what sort of results we can get if we drop the threshold even lower i'll try 0.4 okay so at 0.4 we're getting a ton more results and we're getting almost all the cabbages which is really good but we're also getting several things that aren't cabbages like this haystack over here uh this rock that this pig is looking at just the middle of the grass here so if you're writing code to find all the cabbages this may or may not be what you're looking for because this does eventually find all the cabbages but also finds a lot of false positives so i'll just keep adjusting the threshold until i get as many cabbages as possible without actually getting anything that's not a cabbage all right so 0.48 is pretty good i get a lot more cabbages but i'm still getting this one false positive over here and at .49 i've eliminated all the false positives and i'm just getting cabbages but i'm not getting all of the cabbages so depending on your use case this may or may not be the result you're looking for so i decided to play around with the comparison method that i'm using and i found that this square diff norm is actually giving me the best results for this particular image and with square diff normed that was one of those comparison methods that actually inverts the results so in this case the black pixels are actually the best matches so in our code i actually changed it so that locations we're searching for are actually below a given threshold and the threshold that i have set is 0.17 you can see this actually gave me a lot more cabbage matches without any false positives compared to that co-f norm that i was using originally so play around with the comparison method play around with the threshold that you're setting to get the best results but even using this other comparison method i'm still getting a lot of overlap between my rectangles which visually might not be a big deal but of course that indicates that we're getting a lot of results in our result list that are very close together so that's actually not going to be as useful if you're trying to identify good locations to click on but i think this is a good place to stop for this lesson and in the next video i'll talk about how we can group together these overlapping rectangles into single detection results and not long after that i'll make a video showing you how to apply these object detection techniques in real time so let me know what you're using opencv for in the comments and i'll see you next time you

Channel: Learn Code By Gaming

Views: 34,128

Keywords: opencv, python, opencv object detection, detect multiple objects in image opencv, opencv python tutorial, numpy, numpy where function, python numpy where, opencv game bot, how to use opencv, using opencv to detect objects, python find image in image, threshold opencv python, template matching, object detection tutorial, feature matching, how to detect objects in images using opencv, opencv projects, python zip function, matchtemplate, opencv threshold, opencv thresholding

Id: ffRYijPR8pk

Length: 14min 4sec (844 seconds)

Published: Sun May 03 2020