Using Image Recognition to find Mario

so recently I have been replaying one of my favorite games new Super Mario Brothers and I remember that there are mini games included if you haven't played any of these mini games they are absolute masterpieces and while I was replaying one of my favorite mini games I got an idea the mini game I am talking about is called wanted the idea of the game is simple a character who is wanted is shown and your task is to find and click on the character before the time Runs Out if you press the wrong character then you lose precious time but if you press the correct character you gain 5 Seconds of time now it starts off extremely easy but it gets difficult very quickly so my idea is to write a program to use image recognition to try to automatically find and click on The Wanted character but I have no idea how this is going to go I guess to start I should pick some tools for this project normally I use C plus plus for my projects but for this I'm going to use Python because python has everything we need I'm going to use a library called Pi Auto GUI to take screenshots of the game and to move the mouse around and for processing the screenshots I'll use pill the python image Library so first let's get some code going to take some screenshots [Music] alright so now we have some basic code to take screenshots of the game while it is running I have it split right now into two separate screenshots one for the top screen of the DS and one for the bottom I could display the screenshot by typing the name of the screenshot dot show and as you can see it works [Music] so the way this program is going to detect anything is that I will provide the program with images of all four of the characters being Mario Luigi Wario and Yoshi and depending on which one is found on the top screen AKA The Wanted character the bottom screen is then searched for the corresponding character icon to test if the code can recognize anything at all I'm going to use a function in pi Auto GUI called locate that will attempt to find the image within a base image and return the bounding box if one is located so I'll Loop through all of the characters and then pass the Imogen of the character into the function along with the screenshot and there is an argument to the function called confidence that determines how close a matched image must be to the original in this case it doesn't need to be super accurate so I'll set it to about 90 percent now let's print out the bounding box of the image and see what we get dang it well it is not working at all and that is because we have a problem if we put the screenshot of the top screen next to one of the character textures you could see that the screenshot is just too big and since they are completely different sizes there is no chance that Pi Auto GUI can match the image to the screenshot so we need to scale down the screenshot for this to work since the original DS has a resolution of 256 by 192 pixels I have to divide the size of the screenshot by the amount that the image is upscaled by in my case it is three and three quarters so I decided to add this attribute to a properties file and then load in the information once the program is run and with the screenshots now the correct size it is finally giving me an output now with this information we now know which character is wanted at any given time so the next thing to do is to search the bottom screen for the corresponding character icon this is quite similar to before except that once we get a bounding box from the locate function we find the center of it and then click on it with the mouse so let's test it out [Music] oh come on Luigi's right there so the code works good until the character icons start to overlap the problem is that if the icon isn't by itself the edges get filled up with pixels from the other icons preventing the program from finding it now you might think the solution to this is to reduce the confidence argument in the locate function but reducing it only increases the chance that it clicks on the wrong character by accident so we need a new idea instead of looking for the entire icon at once let's instead search for parts of the icon if we look at a situation like this the only way where we can tell where Luigi is is by looking at the bottom part of the icon where his chin is located so what I have done is created what I like to call sub icons each icon gets split up into an array of slices and then when the programmers run if it cannot find the entire icon it then searches for subsections of the icon instead so let's put this idea to the test [Music] [Music] [Music] Mission failed we'll go next time alright so we seem to have a new problem if I ask the program to display the sub icon used during this altercation something becomes clear or unclear depending on how you see it it seems that some of the sub icons are identical to each other causing the program to click on the wrong character by accident so I wrote a simple function to remove any duplicate sub icons if they happen to match each other close enough so now let's see how far this program can go everybody [Music] [Music] [Music] [Music] [Music] thank you [Music] [Music] levels [Music] [Music] yeah whoa yeah baby that's what I've been waiting for that's what it's all about I'm so happy I was able to make it to level 100. it actually went a little bit further and ended up dying on level 107. [Music] anyways I hope you guys enjoyed this video and if you did make sure to subscribe and I will see you in the next one bye-bye [Music]

Channel: CodeNoodles

Views: 1,442,646

Keywords: artificial intelligence, ai, super mario, programming, machine learning, ai learns, ai learns to play, coding, computer science, image recognition, python, image recognition python, mario, super mario bros, new super mario bros, ai learns mario, comedy

Id: gr6GyecpwKg

Length: 8min 46sec (526 seconds)

Published: Sun Oct 23 2022