Real-time Object Detection - OpenCV Object Detection in Games #5

what's up guys today we're finally going to bring together everything we've learned so far in this series and detect objects with opencv in real time hey i'm ben thanks for following along with this tutorial series if we haven't met before and you're confused about anything we're doing here i've got links in the description to the source code and to the full series playlist so in the first three parts of this tutorial series we learned how to use opencv's mesh template to detect objects inside of images and then in part 4 we learned how to quickly capture screenshots and display them as a video stream inside of an opencv window and now we're ready to combine everything together and detect objects in real time so let's jump into the code so let's start by talking about our window capture class from last time there aren't any changes that we need to make here but there are a couple small improvements that i'd like to make first the first change i want to make is down here at the list window names function and i want to turn this into a static function because it's not using the self parameter at all inside of the function that is it's not accessing any of the properties inside of this class and by making this a static function we'll be able to call it without needing to have an instance of this class so it'll just be a little bit more convenient to use since it is more of a utility function so to add a static method inside python you just add a decorator above the definition line and that decorator is static method and then we'll delete the self parameter and then let me show you in maine what this allows us to do so previously we would need to create an instance of the window capture class and then we would call the method on that object but now that this is a static method we no longer use the object in order to call it we just use the class name itself so this will be window capture dot list window names and this can just be a little bit more convenient because we can access this helper function without needing to create an instance of our class first so for instance i could call this before we ever declare this object then we'll test this out i'll just call exit so that our script ends right after we call this function you can see when i run this i do get the list of all the window names just like before all right so the next change i want to make is in the constructor of the window capture class i've noticed that for certain applications when you try to capture the window you either get a blank screen or just captures the initial screenshot and then doesn't continue to update and this is just a limitation of how we're calling the windows api so to account for this when we come across this situation we can revert to just capturing the entire desktop it's done so when we know that find window isn't going to work for a certain application let's go ahead and just pass in none for window name instead and then when window name is none we'll just go ahead and capture the entire screen so i'll default this parameter to none and then if it is none we'll just go ahead and set self hwnd to the entire desktop so that's win32 gui there's a get desktop window function but if we do have a window name we'll just do exactly how we did before [Music] so i noticed this issue specifically with chrome and firefox with chrome when i tried to capture that window it would just give me a black screen and with firefox it would just take the initial screen capture and then every additional screen capture that i tried to do it just never updated so now let's copy over our find click positions function from video 3 and i'm going to go ahead and put this in a new file and i'm going to call it vision.poi and when you copy this in remember to import cv and numpy and then there are a number of changes we need to make to this function in order to make it work with the window capture class that we have so the first change is we're no longer going to be passing in an image file path for our haystack image instead we're just going to have an image coming from window capture that's already an opencv image and so that image just exists in memory it doesn't exist as a file anywhere so that means that we don't need to do this i am read step let's go ahead and delete that and then what we're passing in is actually going to be the haystack image itself not the path so i'll change that parameter name and so down here in match template this haystack image is going to be passed in directly from the window capture class and then scrolling down to the bottom in this debug mode section we no longer want to call weight key here so i'll comment that out remember we're going to be doing our weight key stuff in the main file instead it'll be down in our main while loop and then one more change inside of our find click positions function this i am show here currently it's only going to be called if we actually find some results so if match template actually gets result and then group rectangles actually return some rectangles so if this length of rectangles is you know greater than zero that's when we're currently going to draw the i am show but we want this i am show to be called every time through our main while loop that way we get that video stream effect even if we didn't find any results so to do that i'll just take this block of code and tab it over one so now it's outside of this if rectangle statement alright so the window capture and the vision we're going to combine those all together inside of the main file and we've already imported our window capture class let's go ahead and import our find click positions function that'll be from vision import find import find click positions and then inside the main loop we'll go ahead and disable this i am show call this item show is just showing us the raw screenshot that we got but now of course we want to be showing our processed image instead so we're going to replace this with a call to find click positions and then when we call fine click positions let's remind ourselves what uh those parameters are again let me just go ahead and copy these over so the needle image path that's going to be a screenshot that we take then we crop down to just the thing that we're looking for i haven't done that yet so for now i'm just going to turn this into an empty string the haystack image this is the image that we're processing and we know this is going to be the screenshot that's returned by the window capture git screenshot the threshold we don't know what that's going to be yet so we'll just leave that at 0.5 and then for the debug mode i'm going to go ahead and draw those green rectangles around what we're looking for all right so now i've got albion open and we need to decide what we're going to look for and i think i'll start by looking for this this limestone deposit over here so i'll just manually take a screenshot of that and then crop it out all right so now we got the image that match template will use to find the limestone that we're trying to detect and in the fine click positions first parameter we'll just give it the file name so that was lbn limestone jpg now if i haven't messed anything up we should be able to detect that limestone in real time so i'll go ahead and run our code all right so here's the game i'll go ahead and move that over to my other screen and then here's the output from opencv and we are indeed finding that limestone deposit that we were searching for and as i move around in the game our detection is still working and is still able to find that limestone deposit and as you play around with this remember to adjust the threshold setting i find that 0.6 works best for me in this case and if you've got this working as you experiment with it you're going to find all kinds of limitations and shortcomings and i definitely want to talk all about those but before we do i first want to go back into our vision code and make some improvements i want to go ahead and convert this into a class just like our window capture class just so we stay more organized as we start improving upon this so i'm going to call this new class vision and on a new class i like to start with the constructor so let's think about when we create an object from this class what do we want to pass in as a starting point i think i'd like to pass in the needle image path and then i was thinking also the method that match template uses to do the comparison that would be a good optional parameter to have so that becomes an optional parameter because we are setting a default here so when we create a new instance of our vision class we can go ahead and read in the needle image right there so i'll go ahead and borrow this code and then let's save this to a property on the vision class or on the objects created by the vision class so we're going to pass in the needle image path and then we're immediately going to read that and save that to a variable and then when we read that needle image in we also know that the shape of that image isn't going to change so we can go ahead and save the width and the height so i'll borrow this code down here again and then those will also be properties [Music] so add the self and then the same thing for the method we'll also save that to a property and this time we don't want to hard code our chosen method here we want to use the one that's passed in through the constructor [Music] so this is a good demonstration of why we use self to refer to properties on the class because this variable here the self.method that is a class property and that's different from this method variable which is the name of our parameter so self.method and method two separate variables here now that we know what some of our properties are going to be let's go ahead and declare those at the top of our class so here i've declared those property variables that we were using down here in the constructor and i've given them some default values so now that we have the constructor set up let's go ahead and make find click positions a method inside of the vision class [Music] so i'll copy all that i'll tab it over one so it now falls within the vision class go ahead and clear up these extra empty lines and then i'm going to use this opportunity to change the name of this function i'm just going to call it find instead of find click positions and then because it's a method it needs to have the self as the first parameter and we no longer need the needle image path here because we pass that in through the constructor and it's going to save that as a property on our class we do still need the haystack image that's going to be the screenshot that comes from the window capture and then threshold and debug mode those will also be unchanged so down in the body of this function we no longer need to read in that needle image because we've already done that same with setting the method that's already been done and then we need to update all of the variables that we just deleted to refer to the properties on the class so to do that for example this needle image that variable no longer exists it's not defined but self.needle image is and so on with method where we use the needle width and the needle height down here and i think that's it now let's head back over to our main file and we'll need to update our import because this find click positions no longer exists now we have the vision class and then outside of our main loop is where we can initialize that vision class and we can do this here because we know that this needle image that we're looking for is not going to change each time through the main loop and then where we called find click positions before instead that's just going to be find and of course that's a method on the vision class and we called that object that we created vision limestone so it'll be visionlimestone.find and we no longer need to pass in the needle image as the first parameter because we already did that when we initialized our class so all we need to do is pass in the updated screenshot and then set our threshold and the debug mode and then by calling this that's going to display the updated processed image and then remember this find function is going to actually return it's going to turn the click points for us so we can go ahead and capture those too if we want and i'm not going to be using these points in this tutorial but if you were making a bot with this those points would be useful for you and again these points are the center points and the objects that you're trying to detect and then if you run this it should work just the same as before and when i ran it i got this error named needle image it's not defined and it tells us that's in the vision code on line number 20. [Music] so if i go into the vision file look at line number 20. indeed this needle image isn't defined anywhere we forgot to add the self in front of this and then same for the line below it that also needs these self and so now when i run it it does work just the same as before but in this case it's not actually detecting the limestone that we took a screenshot of before because as you can see what this limestone looks like it's actually changed albion has a day night cycle and as it goes through that cycle all of the colors inside the game change and that makes it even more difficult to do the kind of object detection that we're trying to do here but if you were playing a more simple game that didn't have this sort of dynamic lighting you know this might be as far as you need to go so let me give you an example of what that might look like so here's an example of a game that's a little more simple as far as object detection goes all of the graphics are pretty static and that makes it easier to detect them so i've gone ahead and cut out a needle image from one of those bottles that's spinning around so let's go ahead and update our code and see if we can detect these bottles as they move around inside the game so inside of main i'm going to be capturing a different window and i'm also going to be using a different needle image so we'll need to update where we initialize these classes so i just commented out that albion section of code for now and for our vision we'll be using that bottle image and because this is a browser game and i'm running it in chrome i'll go ahead and use the full screen capture so again if we pass in none or just nothing to window capture that'll now capture our entire screen and then down in our main loop where we call find i'm going to disable this albion code for now and instead we're going to use our new vision for this bottle game i've tested this before and i think the threshold of 7 was pretty good 0.7 and because those bottles are already green let's go ahead and draw some some crosshairs on there instead so as i run this i've got the game over here on the left and i've got our detection matches over here on the right and you can see that as long as the bottles are in this vertical position you know it's having no problem detecting where they are so if you're looking to detect objects in a simpler game like this one where the graphics maybe aren't as 3d and aren't as dynamic then the window capture and the vision code that we have written so far might be all that you need but what if you are trying to detect objects in a more complicated game like albion maybe it would help if i list out all of the problems so the first problem is not every deposit looks the same you know from match templates eyes this limestone deposit down here looks a lot different than this limestone deposit over here on the left and of course we even have limestone deposits over here hiding behind the tree which might be easy for a human to see but hard for match template to find the next problem is this the shape of these resources changes as you move around it's a little bit easier to see here with this copper ore as i walk around the shape actually changes and again that's a problem for a match template and then as we talked about before albion has this day night cycle which changes all the colors and the lighting in the game over time and that makes it really hard to find a consistent pattern in some games you might be able to play with the graphics settings and turn down like the shadows and all the special lighting and just turn the graphics down below and that might help you to get more consistency with your detection but that won't always work for example on albion there's no way to turn off this day night cycle so to deal with these problems the first thing that you might think about trying is just having lots of needle images to search for and that might work for certain situations if you just have you know two or three needle images that you need to look for but the problem with searching for lots of needle images is that it slows down our processing quite a bit so you can see in our console right now we're only searching for a single needle image but i'm already down to 6 frames per second and as you call match template even more times with more needle images this is just going to slow down even more but still searching for multiple images is something worth trying if you want to do that as a project before the next video and if you're feeling a little bit defeated because we've gone this far but our image detection still doesn't work very well hold on because there's a lot more we can do in the next several parts of this tutorial series we'll discuss some strategies for improving our performance both in the accuracy of the object detections and in how quickly we can process images i hope you're still having fun and i hope you're learning a lot thanks for all the likes in the comments on these videos and i'll see you in the next one you

Channel: Learn Code By Gaming

Views: 34,678

Keywords: opencv, python, object detection, opencv tutorial, opencv object detection, opencv python tutorial, using opencv to detect objects, opencv projects, computer vision tutorial, real time object detection, real time object detection opencv python, real time object tracking, realtime, real-time, real time, opencv real time object recognition, open cv, opencv2, computer vision projects, artificial intelligence course, albion online bot, opencv projects with code, python opencv game

Id: 7k4j-uL8WSQ

Length: 18min 15sec (1095 seconds)

Published: Mon Jun 22 2020