OpenCV Object Detection in Games Python Tutorial #1

OpenCV is an open-source computer vision library with hundreds of functions for processing and understanding images in this video I'm going to show you how to get started with open CV in Python by using it to find an image inside another image this simple form of object detection will be a good starting point before we move on to more advanced image recognition techniques hey I'm Ben and this is low in Kodambakkam in concepts by applying them to video games the quickest way to get started with open CV is by pip installing the pre-built binaries to do that you simply pip install OpenCV - python once installed you can use the library by importing cv2 and throughout my videos you'll see me importing CV - as CV which is somewhat common but you're free to import it either way so it's up to you whether you want a reference open CV as CV or CV - and when you install OpenCV numpy will also be installed and an umpire is used extensively when working with open CV data so the top of your files - normally look like import CV to s CV and import numpy as M V and that's all there is for setup so now let's grab an image we want to process I'm gonna be using this screen shot from Albion online which is a free-to-play MMO and I actually just grabbed this image from the steam page for the game but if you're following along feel free to use a screen shot from whatever game you want and what we're gonna do is we're gonna crop out a small section from our screen shot and I'm gonna save that as a separate image file and then we're gonna use open CV to find the position of that smaller image inside the entire screen shot and this is the image that I cropped out earlier from my screen shot just using Photoshop I just picked one of the cabbages so if you follow my channel you're probably already familiar with a auto GUI and if you don't know what Pi auto GUI is it's just a Python library for automating keyboard and mouse inputs and it also has some basic computer vision features like allowing you to take a screenshot doing simple pixel color matching and it also has this locate on-screen function which does essentially exactly what I'm gonna teach you in this video how to do with open CV located on screen just takes a smaller craft image and it returns the position of that image if it finds it somewhere on your screen I bring this up because if you look at the PI auto GUI documentation you'll see that to use this confidence parameter you actually need to have open CV installed in order for that to work and the confidence parameter is really useful it allows you to do fuzzy matching so you can still find an image on the screen even if it's not a pixel perfect match so if peyote GUI is using open CV under the hood you can reason that using open CV directly it's gonna be a lot more flexible and a lot more powerful so if you're following along for my previous tutorials the architecture you should have in mind is this continue to use peyote GUI or PI direct input to automate your inputs but now we can use open CV to get much better vision into what's going on on the screen the open CV function that we'll be focusing on is called match template and the opencv documentation is a little confusing so let me show you how to navigate it starting from Doc's OpenCV org go over to the doxygen HTML section and then click on the link for your version of OpenCV I have four point 2.0 installed and here you'll see the different modules that OpenCV is organized into and because of the way that we installed OpenCV we only have access to these main modules right now but that's plenty to get us started and we're going to be diving into the image processing module and then go down to object detection and in here you'll find just one function it's the match template function that we're looking for and as a Python programmer this documentation should look a little weird to you that's because OpenCV is written in C++ and most of this is C++ documentation sometimes they'll give us a quick note down here about the Python method signature but it makes sense that an image processing library like OpenCV would be written in C++ because C++ is so much faster at those sorts of algorithms and because of the Python wrapper we get the best of both worlds we get the speed of C++ and the joy of working with Python so looking at the match template documentation we can see that we're going to give it an image to search over and then an image to search for then we're going to give it some method for doing the comparison and you'll end up with some sort of result array there's a little more info on what that we saw the way will look like down here in the parameters section and if you scroll up you'll see this scary-looking math stuff and these are just all the different comparison method types you have to choose from maybe you can visualize these like Mia from the matrix but I can so let me show you another part of the documentation that will show you what these look like if you go back to the main page of the documentation I'll open it in a new tab and then go to OpenCV Python tutorials and then to image processing in OpenCV and finally down to template matching and here you'll find a nice quick tutorial for how to use match template so in the example code they're searching for instances of this cropped image of a soccer player and in the results they show you the output for all the different comparison methods so in the images on the left the bright white pixels those are the positions that matched most closely we the smaller cropped image that they were searching for until you scroll down to the last two methods these squared if methods they're inverted so the black pixels are the closest matches and it's good to note that these closest matches the white pixels corresponds with the top left corner of the image that you're searching for so when we write our code you should experiment with all of these different methods to find out what works best for your use case and you can refer back to these examples to give you some idea about what the differences are between the different methods all right so let's write some code first thing we want to do is load in the image files in a format that Koken cv understands so to do that we're gonna use the I am read functions so I'm gonna call my first one haystack age that's a screenshot that I'll be searching through CVE I'm read and the first parameter is the file path of the image that we're loading so in my case because the image is in my Python view I can just use the file name I'll be on and the second parameter is a flag that allows us to do some pre-processing when we load in that image I'm just gonna load in my image unchanged though so that'll be CV that I am read unchanged and then similar code for the next image I'll call this the needle image so we'll be searching for the needle inside of our haystack this file is called lb and cabbage and I'll also read it in unchanged let's take a quick peek at the different read options that we have we can read it in unchanged where we could also immediately convert the grayscale we could also immediately convert it to a BG our color format which is the format that OpenCV prefers you can also reduce the size of the image you write when you're loading it in so now that we have our image files loaded we can go ahead and call match template and for the comparison method I chose this CCO BFF normed and I just find that this one gives me the best results most of the time and now that we've called match template let's go ahead and take a peek at the results so to do that I'm going to use imshow which is just going to show an image in a window for us the first parameter is just the name of that window and then the second parameter is the image of the data that you want to display if you were to run this code you'd see that it runs and it closes immediately we don't get a chance to look at the image so to get our script to pause and to give us a chance to review the image we can call CVA key and wait key we'll just pause our script at this position until we press any key on keyboard so let's go ahead and run this and see what we get all right so this is the result of match template remember the white pixels will be the best national occasions whereas the black pixels will be the worst locations and we can see that it is about what we expect the brightest match is right here at the location of the cabbage that I cut out but then also some of these other cabbages around it are also pretty good matches because remember we're starting with this screen shot and we're looking for this cabbage inside of it so now that we know that match template is working how we expect what we want to do is you want to get the pixel position of this brightest white pixel and that will be the position for where our needle image appears in our haystack image so to do that first I'm just going to get rid of this debug code and then we're going to want to use a function called min Max location and min Max location looks like this you simply call C V dot min Max LOC and you give it the result from match template and it's gonna return four values it's going to give you the minimum value that is the blackest pixel it's also going to give you the maximum value that is the whitest pixel how wide is it and these mini max values will be on a scale from zero to one where one is the brightest white and zero is the darkest black and then it will give you the location for that minimum value and it'll also give you the location for the maximum value so this will get the best match position and for the comparison method that we used were interested in the maximum value location so let's go ahead and just print out that location and the value see what we got all right so I put in some print statements here and this max location will just be an XY tuple so let's run this and see what we get so looking at our output here the best match location was here and the caffeine's of that is 0.98 which is not quite a perfect match but it is really really close I'm not sure why it's not considered a perfect match since I did literally cut this out of that original image or must have to do with how the algorithm works for the comparison method of each of the way that template matching works is it's always going to give you back some sort of result so even if we were to take an image that doesn't appear anywhere in our screenshot and we run it through this algorithm min/max location is still going to return a max location and a max value it's just that that max value will be much lower than 0.98 it'll be something like 0.4 to indicate that that is the best match location but it's actually not that close of a match so to verify that you did actually find the image that you were looking for you want some code like this you want to set some sort of threshold where any value that you find above that threshold it'll be considered a match but if it's below that threshold here I'm using 0.8 we'll just consider that as the needle wasn't found in that screenshot so in our case 0.98 is greater than 0.8 so it's going to tell us needle found and again this max value is just a measure of the confidence that it did find a matching location so now that we have the location of the needle image inside of the haystack image what I'd like to do is I'd like to take that original screenshot and just draw a rectangle around where we think that image is so to do that OpenCV as this rectangle function and to use this function you simply give it an image to draw on and I'm going to give it the haystack image which is our original screen shot and then you need to give it the coordinates for the top left corner of the vector angle and also the coordinates for the bottom right corner of the rectangle and the fourth parameter is the line color for the rectangle and this is in BGR format and all these values go from 0 to 255 so here we got blue and the red at 0 and the green at 255 so we expect this to be a green line and then you set the thickness of the rectangle boarder in pixels and then you set the line type for the vector angle and here line 4 is going to be the outline of a rectangle that we're looking for and now that we've drawn on our screenshot we can simply use I am show again to upload that image and see what we got but before we can run it we of course need to define top left and bottom right so top left we already have that's just going to be the max location and for bottom right we can calculate bottom right just using the top left location and then adding on the width the height of the needle image and when you're using OpenCV you can grab the width in the height of an image just by using the shape property so we'll grab those right there and now we can calculate the bottom right corner of our rectangle just by taking the x-coordinate of the top left and adding on the width of the image and then the y-coordinate of the top left and adding on the height of the image and when I run this you can see that we've now have our original screen shot except we've drawn that green rectangle around the image that we cut down so we've successfully found our needle image at this location and that was off-screen for you a little bit so remember when we're reading in our image files we can downscale them right there so I'll cut them in half you can see a little bit better so here we have the half size images and it's still finding that cabbage successfully another thing you can do I'll back out of that have my images be unchanged again you can also save this image to a file instead of trying to show it using item show so I'll comment that out and that function is just I am right so the first parameter is the file name of the image that you want to save and the second parameter is just the image data and it will smartly format your image in the appropriate format for the file extension that you give it so now when I run that I now have a result image in my folder you can see it's that same image as before with our handy green rectangle from here there's tons of directions you could go with OpenCV I want to keep exploring it and I want to keep making videos about what I find along the way so if something specific you want to see let me know in the comments I'd like to keep these videos as short as possible because OpenCV is such a massive library to make it approachable I think I really needs to break things down into small chunks so that's my plan subscribe if you want to see those videos and I'll see you next time you

Channel: Learn Code By Gaming

Views: 96,662

Keywords: opencv, python, opencv object detection, opencv python tutorial, python image processing, python find image on screen and click, opencv find image in image, opencv object tracking, opencv python install, python find image in image, find objects in pictures, computer vision projects, using opencv to detect objects, opencv projects, opencv matchtemplate, opencv match image, opencv template matching, computer vision tutorial, image recognition, detect objects

Id: KecMlLUuiE4

Length: 14min 29sec (869 seconds)

Published: Sat Apr 25 2020