HSV Color Range Thresholding - OpenCV Object Detection in Games #6

what's up guys in today's video i'm going to show you how to use pre-processing on images to make your object detection better in opencv specifically i'm going to be talking about hsv thresholding hey i'm ben this is learn code by gaming and we are up to part number six in the opencv for video games tutorial so to recap in the previous videos we talked about how to apply opencv's match template to video games in real time in order to detect objects that we want to find on the screen and what we have so far works perfect in some situations but it falls short with more visually complicated games so one way we can dress that is by filtering out objects we're clearly not interested in before we ever send the image to match template for example if the object we want to detect is blue we can just remove everything that isn't blue from the image another thing we can do is over saturate our image which will exaggerate the differences between different types of objects the net effect of this should be that we can get more positive identifications without also increasing the number of false positives all right so in our code we're going to pick up exactly where we left off and all of our work today is going to be inside the main file and then the vision class and then before we get into our image pre-processing the first thing i want to do is refactor this find method so right now find is a rather large function we're calling match template in it we're grouping the rectangles we're finding the center point of those rectangles we're doing a bunch of debug output and i'd rather split all these different things into their own methods and by doing this it'll set us up to potentially apply multiple different types of filters and mix match them in the future and so by refactoring this find method it'll give us greater flexibility so i think i'm going to keep this find method to do our match template call but i'll get rid of the debug mode parameter and i think i'll cut this off right where it has this list of rectangles and i'll just return those and i don't want to lose the rest of this functionality i just want to move it into other methods so one of the things this code is doing is it's determining the center position of a bunch of rectangles so let's make a method to just do exactly that we'll give it a list of rectangles and it'll return a list of center positions all right so now we've got this much more compact method and it just does one thing and the last thing this code did it was showing our original image but with the rectangles or the crosses drawn on it so let's split that out into two different methods we'll have one where we draw the rectangles on an image and another where we draw the crosshairs on an image and instead of having i am show as part of the debug code let's go ahead and make that the responsibility of the main file alright so here's what i came up with for drawing rectangles we're going to give it an image to draw on we're also going to give it a list of rectangles and then the code inside is pretty much the same as before to draw those rectangles and instead of calling i am show it's just going to return that resulting image and then draw crosshairs is the same except we give it a list of points instead of a list of rectangles and now we need to update our main file to work with these new methods so find still takes a screenshot and a threshold but it no longer has this debug parameter and it no longer returns a list of click points instead it returns a list of rectangles and this no longer displays the processed image now it's just doing the object detection and now that we have those list of rectangles drawing those onto our image is now a separate step so now we call draw rectangles we give it our screenshot image we give it the list of rectangles and it's going to return an output image and finally it is now the responsibility of the main file to call i am show to show that image so fundamentally nothing's changed we've just refactored our code and if we run it it should work just the same as before so i'll run this to make sure it's still working you can see it's still still operating the same as it was before so now our goal with the rest of the opencv series is to do some pre-processing on this screenshot image before we ever send it to the find function that will make the find function work better and the first thing i want to try is a color filtering technique that knocks out all the colors we're not interested in and opencv has an official tutorial on how to do this so if anything i say doesn't make sense you might want to check out this article too theirs is called thresholding operations using in range and they've got some images here to help you understand the difference between an hsv image and an rgb image so up to this point all the images we've been working with have been in bgr format which stands for blue green red and if we were interested only in blue objects we could just set all of the g and the r values to zero and we'd be left with only the blue pixels but any other color besides those three are going to be a mixture of those three colors and so they're a lot harder to isolate using the bgr format so what we're going to do is we're going to take our image and convert it to hsv format that way the hue or the color is just a single value we have to worry about and then it also makes it easy for us to increase the saturation and the value because those are isolated too so while it's possible to do color filtering with the bgr images it's just a lot more convenient in the hsv format and doing this filtering itself isn't terribly complicated you just convert to hsv format using convert color and then you'll call cv.inrange and apply your filter to it but it becomes really tedious to try to figure out what exactly you should use for all these low and high parameters to make your filter so in the example tutorial here they've created track bars on a gui interface what that allows you to do is you can move these sliders to adjust your low and high settings for your filters in real time and i think that's a pretty good idea so let's go ahead and use the same technique so the first thing we need to do is we need to build a gui window and this is all built into opencv in this window we'll need a name and we'll reference it by that name so let's go ahead and make that a constant on the vision class and then i'll create a new method to build the gui so here you can see the gui code i just create a window and then i resize that window and then i add a bunch of track bars to it and i've set the minimum and the maximum value on those track bars and for the maximum value track bars i've also initialized those to be all the way over to the right instead of at the beginning and the one thing that needs a little bit explaining is this nothing function right here so this create track bar function it requires you to pass in a callback function to it and then anytime a trackbar is changed it'll basically run that function for you and in our case we're not going to use this we're just going to look up the trackbar positions when we need it instead but this isn't an optional parameter we have to give it something so i've just created this nothing function to pass in so it's happy and this method won't do anything yet or it won't control anything but let's go ahead and call it in main just to make sure that we are seeing the gui that we expect so outside of the main loop i'm calling init control gui on our limestone vision object and when i run this you can see that we have this gui window over here with all these sliders now because there's so many values here with these track bars instead of keeping track of each one of them individually i think what i want to do is create a custom data object that will just contain the state of all of these sliders so i'll create a new file and i'll call it hsv filter and to create a custom data structure in python is really simple you just create a class and i've just created a constructor that will save each one of those slider values to a different property so the next thing i want to do is over in the vision class i want to add a method that will return an hsv filter object with a state of all the slider values so first we'll need to import our new class and then i've decided to call this new function get hsv filter from controls and you can see this simply initializes a new object from the hsv filter class and then it populates each property with the value from the track bars using this get trackbar position function and it will return that hsb filter object and each one of these strings that identifies a track bar needs to correspond with the strings that we used when we created the track bar so those are just the names of those track bars so now that we can get the positions of all those track bars easily let's write a function to actually apply that filter to our image so i've decided to call this method apply hsv filter and it's going to take the image that we want to apply the filter to and then the hsp filter itself i'll make that optional so if we have a predetermined filter that we want to apply every time we can go ahead and pass that in otherwise if this is set to none we'll go ahead and use the values from the get hsv filter from controls to build our filter so the first thing we need to do is convert our image into hsv format and we know how to do that from the official tutorial that we saw earlier and then if we haven't been given an hsv filter object let's go ahead and get one from the function we just wrote and then this is the code that will actually apply the minimum max values from our filter so we'll have a minimum hue a minimum saturation and a minimum value and that'll all be part of one np array and then same for our maximum values we've got the max hue saturation and value and then just like we saw in the official tutorial we go ahead and call cv.inrange to apply to our hsv image the lower and the upper arrays and then if we take another look at that official tutorial when they call in range what that actually is returning is this frame threshold you can see in the in range documentation it says it checks to see if an array element lies between the elements of the other two arrays so what it's going to return is more like a mask where the pixels that are in range are going to be a one and the pixels that outer veins are going to be a zero and what you'll end up with with the result from in range is a black and white image like this where again the white pixels are inside of the range and the black pixels are outside of the range so this last line here where i'm calling bitwise and what i'm doing is i'm taking that mask and i'm applying it to our original image and anywhere where the mask is zero our resulting image will also be turned to zero so those pixels will be made black and anywhere where our mask is one it'll basically ignore those and it'll leave those pixels alone and then the final step is to convert our image back into that bgr format and return it and if you notice in our original gui that we created i added sliders not only for the hsv min and max i also added sliders to add and subtract to the value in the saturation so with those what i'm hoping to do is mostly add saturation and value to brighten up the image and to increase the contrast between the different pixels and i think that'll help us get better results so in apply hsv filter we need to write that code to increase or decrease the saturation in the value by the value that our track bars are at and that should be pretty simple for example if we want to increase our saturation value by 10 we should just be able to add 10 to each one of the saturation channels the one thing you need to watch out for is if we already have a pixel that that's at 255 saturation and we add 10 to it we don't want that to go up to 265 we want to keep that at the maximum of 255. so basically we don't want to go above or below the maximum when we're adding or subtracting these values so here's the method that will do that for you it's called shift channel and i just found this on stack overflow because i was too lazy to try to figure out myself we can see it's not too crazy it's either just adding or subtracting the amount that we want without going above or below the limit and i want to apply those adjustments before we do the thresholding so that'll go right in here and to do it we first need to split the hsv image into its component hs and v values and then we can shift the saturation and the value corresponding to those sliders or to the hsv filter object that we were given and then once we've made those adjustments we can just merge it back in into a single image so now we're ready to go try this out back in main and i'm going to disable the object detection stuff for now let's first just see what it looks like when we apply a filter to our screenshots so i'm calling apply hsp filter sending in the raw screenshot and i'll have that be our output image so we'll run this now let's just try out all the different sliders and see what they do you want to test out every single one make sure they all work you can see by adding value we can really brighten up our image or we can subtract value to really darken it we can also add saturation to really increase the contrast between the different colors or if we subtract saturation it goes more to grayscale we can also adjust the value minimum to make all of the darker pixels go all the way to black or we can adjust the value maximum to take all of the brightest pixels and turn those black same with saturation we can use the minimum saturation to take all of the grayscale pixels and make them black or pull down the saturation max and take all those really bright saturated pixels and turn those ones black and for the hue you kind of want to think of hue as a color wheel so if we really narrow down the hue so here between 0 and 17 we can see mostly red type colors are coming through but if i move that band over to the right you can see now i'm getting mostly the green colors coming through with the hue between 90 and 130 that seems to be about the blue range and right at the top of the hue it seems like we've gotten back to maybe the purpley reds to be more certain about the colors that we're selecting maybe we add a little bit of value and a ton of saturation now when i play around with the hues you see the greens really jump out of there now same with the reds increasing the saturation makes that a lot easier to isolate now you can more easily see at the top of the hue range that is kind of the more purpley reds versus at the bottom of the hue range that's kind of the more yellowy type reds and once you have some sense of what the different filters do what i want you to do is to try to pick an object on your screen and try to move the filters to isolate that object so for example let's try to isolate this fire pit right here so i'll generally start by increasing the value a little bit to make it brighter and then increasing the saturation and then just kind of play around with the sliders until you just see that fire pit really jumping out and nothing else don't forget to use the saturation in the value sliders too so this looks pretty good you can see i've isolated the fire pretty well and from that initial more complicated image that we have you can see that we've eliminated a lot of it right off the bat and what we're left with is this image where a lot less is going on and so the idea is is if we feed this into match template and we give it a needle image that is maybe just a crop out of this fire pit right here that's going to be a lot easier for it to find versus what we initially started with it's easier because there's a lot less going on there's a lot less chance for a false positive and even though some other things on here like this button are the same color red as the fire pit you can see that the outline of it is clearly different so hopefully match template knows not to match those up alright so let's try this out in game i'm going to try to isolate this limestone over here all right so once i got some settings that i like i'm going to take a screenshot and i'm going to crop out the object that i'm looking for and you also need to remember to write down or save all the values of these sliders because we'll be making an hsb filter using those settings now in main we'll need to remember to import the hsv filter class and then we'll update the needle image to use our processed image and this is what that needle image that i cropped out looks like and then i'll go ahead and create that hsv filter object using the values from the sliders that i saved and then we can just pass that filter into apply hsv filter now when i run it you can see that that filter is automatically applied and it doesn't matter what i do with these sliders anymore because i gave it that set filter and you can see that the day night cycle is already kind of changing what it looks like when i apply these filters so i'm kind of trying to move a little faster so now i'm going to reinstate the object detection stuff i'm going to save this one as processed image that we get back from applying the hsv filter and i'm going to go ahead and give that to the find function so that way match template will be using this processed image as the haystack and then the needle image will be this one that i cropped out and then when we draw the rectangles we'll go ahead and use the original screenshot for that so even though we're getting the rectangles using the process stuff we'll still on our output draw the rectangles on the original screenshots but if we wanted to see what the processed images look like at the same time we can just create a new window for that so now when we run this we'll see both outputs so now you can see we're processing the image to get this and we're using the needle image that looks something like what i cropped out here and over here you can see that when we pass that into find match template is finding that successfully and that's already working a little bit better here it's finding uh two examples of that limestone instead of just the one and what this allows us to do is because there's less chance that we'll find false positives we can lower our match threshold a little bit and as we lower that we should be able to match more of the limestone deposits that we weren't matching before so you can see as i run around here it's doing a much better job of matching some of these limestone deposits of course we still have one false positive here matching with this wheelbarrow so it does look like it needs a little bit more fine tuning still both on our hsb filter and on our match template threshold and that's about all there is to know about hsv thresholding or color filtering and before you go there are just two more changes that i made to the find function that i want to tell you about the first is is if find finds lots of results it'll really slow down your program and i've actually had mine locked up before so i've added this optional max results parameter it'll default to 10 and then down at the bottom of this method or if you have more than that maximum number of results it'll go ahead and just truncate it so this way if you're playing with your match threshold and you get a little bit too aggressive with it it won't lock up your program the other change i made is back near the top of the find method if we don't find any locations for match template it's going to go ahead and return an empty array i was experimenting with using multiple needle images and then combining all those find results together so all those lists of rectangles i just wanted to concatenate them together and i'll put them on a single output image but when i did that when i tried to join together a list that did have rectangles and a list that didn't it was giving me an error and i found you can fix that just by reshaping that empty array before you return it so that's what this is about here so go ahead and give this method a try you'll find that it works really well in some situations and not so great in others for example on albion it works pretty good right now but we still haven't gotten over the issue with the day night cycle and that's because that cycle causes the colors to shift so i've got an idea for another processing technique to try out in the next video to try to solve that so that's about all there is to know about the hsv thresholding technique i hope you guys are able to find some success with it in the next video i've got one more pre-processing strategy that i want to show you so if you like this video that one should be interesting for you too and i'll see you then oh yeah and all of the links to this code is in the description or you can just go to the learn code by gaming github and you'll find it there you can also go to learncodebygaming.com i've got articles there for all the videos that i make i figure i write scripts for all these videos anyways so if you'd prefer to read this stuff you can find those there in written form so yeah go check out learncobargaming.com if you haven't already you

Channel: Learn Code By Gaming

Views: 18,261

Keywords: opencv, python, object detection, opencv hsv color detection, opencv hsv color range, hsv opencv python, hsv threshold, hsv thresholding, hsv filter, hsv filtering, opencv inrange, opencv gui, opencv trackbar, hue saturation value, computer vision tutorial, python custom data structure, color channels, opencv color detection, opencv color tracking, hsvfilter, python opencv game, opencv projects python, opencv image preprocessing, image processing using python, inrange, open cv

Id: 0tKzqsRtmyY

Length: 22min 48sec (1368 seconds)

Published: Tue Jul 14 2020