Making a Whack a Mole bot using OpenCV's Match Template

what's up guys today we're gonna make a bot that plays guacamole for us using opencv's match template so let's dive right into it first things first i have my workspace folder with my walkabot.pi file a subfolder named imgs where we will store our images and in our dot pi file i pasted the link to the game that we will be bought in so let's open it and manually play it once so we set a standard for our gamebot after we watch a wonderful ad that is my final score was 3250 so let's beat it let's pip install the modules we will need by going to our terminal and typing pip install opencv dash python and pip install payer gui since i already have them installed i will get the requirement already satisfied message after that let's move to our code and import them by typing import cv2 and import pi out of gui we will also type from time import sleep to have some time to prepare the game before running the script pi out of gui will get us screenshots and will click for us it has what we can call a cool down setting which causes it to wait a few seconds between each function this will actually hurt the bot so we will remove it by typing pi out of gui dot pause in caps and we're gonna set it equal to zero next we need to get our template which is the image we're looking for so we will manually take a screenshot of some moles and try to find something similar in all of them after some time i decided that their hands and noses are pretty much the same but since their hands move too much i will use their noses so let's cut it out using the almighty paint and store it within our imgs subfolder back in our code let's create a variable called template and set it equal to cv2 dot in read which will open the image the parameter for this function is a string of the name of the image which is nose.png for me but since the image is in the imgs subfolder we need to add imgs forward slash before the name of the image and so it will read imgs forward slash nose dot png we will need the dimensions of the template which are its width and height but before that let's gray our image i will create a template underscore grade variable we could just add the zero black to the ingrid function and that would automatically make it grey but there is something i want to do with the colored image so let's set our template underscore grade variable equal to cb2 dot cvt color capital c and the first parameter is the image that we want to convert which is our template image and the second parameter for this function is the code for the color conversion in our case we want to use cv2 dot color underscore rgb to gray which as you may have guessed will convert our red green and blue image to a grayscale image now let's go ahead and get the width and height of the template image for this we will create a template underscore w for width and template underscore h for height and set them equal to template underscore gray dot shape and we will need to slice it like this so it returns the values in the correct order so the logic is this when you work with images normally you get the width first and the height second but for some reason opencv returns height and then width so slicing the tuple like this will return the values reversed up next we need the game windows dimensions let's create an x y w and h variables to get their values i will take another screenshot of the game window note that i take it on full screen to make sure we don't move our coordinates when we scroll up or down the relevant area is about here if we move our cursor to the top left corner of the area we will get its x and y coordinates if we expand the selected area we get the width and height of the window now that we have our values we can set our variables equal to 5 23 for the x-coordinate 247 for the y coordinate 8.75 for the width and 6.79 for the height let's give ourselves three seconds to prepare our game window before the script starts running by calling the sleep function its parameter is an integer or float of the seconds we want it to wait in our case 3 seconds is fine for the main part of our script let's create a while loop because we want the bot to always run unless we tell it to stop when the game window is visible we will take a screenshot of it so let's call pi autoguide.screenshot the first parameter in this function is a string of how we want to save the image as so let's save it in the imgs forward slash folder and just call it image dot png the second parameter is a tuple with four values the x and y coordinates of where we want the screenshot and then the width and the height of the image so let's just type in our variables now that the image is stored let's open it in a variable called image with the cv2 dot in grid function inside this loop we will actually have another while loop because we want to process the same screenshot several times before taking another one this will save some time for the method that we are using we need to grade our image so let's create an image underscore grade variable and copy the same function with as with the great template to perform match template let's create a variable called result and set it equal to cv2 dot match template the first parameter is the image the second one is the template on both we will use the great versions the third parameter is the method you can use whichever you like but i prefer this one to extract the results let's run min max lock on the result since it returns four values we will unpack them on main underscore ball max underscore ball min underscore lock and max underscore lock remember that max ball returns the highest matching score and max lock the coordinates of where that score is on the screen to make sure we always get the best matches let's establish a threshold so if the max file is equal or greater than 0.8 which is an eighty percent match we will consider it a good match and proceed to run pi article dot click this function takes in two arguments the x and y locations of where we want it to click so let's set x equal to max underscore log 0 plus x remember that max lock returns a tuple with the x and y coordinates of the best match so if we add the best locations x coordinate plus the x of where the screenshot was taken we will get the exact x coordinate of the best match then we do the same with the y-coordinate but we access max lock index 1 and we add the y-coordinate of where the screenshot was taken the next thing we want to do is draw a rectangle over the nose of the mold we just clicked on so we can run much template again on the same image in case there is another mode to be found this is faster than taking a screenshot once again to do this we will update the image variable by setting it equal to cv2 dot rectangle this function will draw our rectangle on the image its first parameter is the image we want to draw the rectangle on so let's set it equal to our image variable the second parameter is the points one which are the x and y coordinates of where the rectangle should start so we can just set it equal to the max lock the third parameter is 0.2 which is where the rectangle should end let's set it equal to a tuple the first element is the x-coordinate which we can get by adding the max lock index 0 plus the template's width the second element is the y-coordinate which we can get by adding the max lock index one plus the template's height the next parameter is the color of the rectangle which we specify through a bgr tuple so we're gonna do blue green and red here let's make it red so we're gonna do zero zero two fifty five finally we specify the thickness of the line if we type minus instead of giving thickness to the rectangle it will fill it with the color that we specified in this case it's going to be red once we have drawn a rectangle the secondary loop will start again and look for another match but what if there are no matches so we're gonna do else break so if there are no matches the bot will exit the secondary loop into the main loop again and take another screenshot enter the secondary loop to run match template on this new image and start the process all over again until the end of time or until we decide it should stop so let's see how it works something important to know is that if your bot is running and using thai auto gui you can move your mouse to the top left corner of the screen and it should end the script automatically if you're using vs code like me just use the stop code option and if you are using a terminal you can close it it did really good after the third try it broke my personal record there seems to be some rng involved so the maximum number of points you can get per run is not always the same another problem that i could identify is that some letters pop up like when you reach a certain score and the bot can't see the molds behind it so i guess it's up to you guys to find a solution for that as a cool bonus let's show what the computer sees we need a smaller copy of the image so let's create a variable called image underscore meaning and set it equal to cb2 dot resize first parameter is the source image so let's set it equal to image and the second parameter is a tuple with the new width and height let's just hardcore some numbers to about half of the current size so let's do 450 for the width and 350 for the height something to note is that the width and height must be specified with integers not floats now we can call cv2 dot in show the name of the window will be vision and the image we want to display is the mini underscore img to keep the window up and updating every time a screenshot is taken let's call cv2 dot wait key and specify a delay time of 10 milliseconds before each update i will comment out the clicking part so we just see what the computer sees instead of clicking all right this should work perfect so let's do it but nope something went wrong okay so i type show instead of in show and this should fix it and here it is i don't know about you guys but i just love seeing this so if you like the video don't forget to like and subscribe and if you tried your go at about like this one let me know what score you got on the comments below and i'll see you guys later thank you

Channel: Noob Coding and Gaming

Views: 4,325

Keywords: opencv, opencv python, opencv tutorial, opencv python tutorial, whack a mole, opencv visual studio code, opencv projects, game bots, automating games, game automation, how to make a game bot, image recognition

Id: SZiVZM6qf6U

Length: 17min 22sec (1042 seconds)

Published: Sat Apr 10 2021