Using Image Recognition to play Mario Party Minigames

this video is sponsored by brilliant.org but more about that towards the end of the video nearly a year ago I made two videos in which I wrote some python programs to automatically play mini games from new Super Mario Brothers and it was a really fun project to work on but there is another game from the same console that also has some really cool mini games and that game is Mario Party DS now I have a confession to make unlike new Super Mario Brothers I didn't actually play this game growing up but I have spent some time going through this game and playing the mini games and I gotta say I definitely missed out but in this video we are going to be seeing if I can use image recognition to play Mario Party mini games let's begin alright the first mini game I want to automate is literally the first mini game and it is called Goomba Wrangler in this chaotic minigame your goal is to Circle as many Goombas as you can while trying to avoid the bombs difficulty of the of this game is that it's very hard to circle the Goombas quickly and accurately enough so image recognition is either going to make this problem disappear or it will become even more of a problem but before I could start working on writing any code to automate this minigame I had to write out some boilerplate code to handle the mini game scripts I'll be writing in this video this code will allow me to easily switch between mini games and make this process a whole lot easier [Music] since we're now done with the boilerplate code let's finally get started on the first minigame script the first thing I did is that I created a function that allows me to draw circles on the screen the library I am using to control the mouse is called Pi Auto GUI and it can only move the mouse in a straight line from point A to point B so making this function was pretty important now I'm sure you're wondering how can you draw a circle with straight lines well the same way that computers render Circles by using polygons as you increase the number of points in a polygon its shape begins to approach that of a circle so all I have to do is to generate the corners of a polygon with enough points and then connect the dots connect the dots sure you could see our code in action drawing a polygon but clearly the game isn't detecting it as a circle yet but as soon as I turn up the number of points in the polygon it is actually able to pick up some Goombas now that we have a way to Circle an area on the screen we have to figure out where we need to Circle so I think it's time for some image recognition the first step of this process is to get some images of what we're trying to locate so I took a screenshot of the game and used it to make some smaller images of the things we will need to locate then I began writing out some code to locate the Goombas the library I'm using to locate things on the screen is the same library that can control the mouse Pi Auto GUI and when we try to locate an image using pi Auto GUI if it finds the image we are searching for it Returns the bounding box of its location on screen so what I did is that when a Goomba is detected the mouse will draw a circle around the center of the Goomba's bounding box and here is what it looks like in action clearly the program is correctly identifying the Goombas which is good but it doesn't seem to be able to Circle them quickly enough most functions in pi Auto GUI have an argument called duration and this is used to control how slowly or quickly a task is performed but if we look at the code that moves the mouse right now I already have the duration set to zero so how can I make it any faster well there is a solution but logically if the actual time it took to perform these Mouse movements was actually zero seconds the circle would be drawn instantaneously but it's not and I figured out that this is because in pi Auto GUI all functions have both a minimum duration and a pause in between each function call but we are able to modify these values so by lowering these values we are now able to circle the Goombas significantly quicker but now we have a new problem the bombs are being circled too so to fix this issue I wrote out some code to locate the bombs as well and to then find the Goomba that is the farthest distance from any bomb on let's see how it does [Music] [Laughter] [Music] thank you even with some mistakes it was still able to absolutely destroy our opponent oh my God God but this might have something to do with the fact that the difficulty is on easy right now so let's try this again but in the hardest difficulty expert [Music] [Applause] [Music] it's pretty impressive that with so many little mistakes this program is still able to win in the hardest difficulty now I spent a whole lot of time trying to adjust the script to be as accurate as possible but sometimes a bomb could be picked up by accident anyway either way I think this first minigame script was a success so let's now move on to the second Mini-Game we will be automating in this video Trace Cadets in this mini game you're given a grid of shapes and your task is to draw the shapes that are currently outlined in Blue on this table and this could be quite tricky because if your lines aren't very straight or well aligned with the grid the game won't recognize them so this should be the perfect mini game for automating to get started just like with Goomba Wrangler I decided it would be wise to begin by writing out some code to control the mouse but this time it's for drawing line segments on the table and this was a pretty straightforward process the only information I really needed was how far the table is from the corner of the screen as well as the length of each segment on the table I then wrote out a simple function that can draw a shape by chaining multiple segments together and to test this code out I decided to input a list to represent a square and it works just as you'd expect now that we have the simple part out of the way it's time to get into the meat of this problem how do we detect what shapes we have to draw well my first idea was to make up an image of each shape and then to have the program try and find them on the grid and that would work fine for the rectangular shapes but for all of the others it wouldn't work very well and this is because if we try and locate a shape that isn't rectangular then the areas at the edges of the shape could be different depending on where in the grid they are located and this would prevent the program from being able to accurately locate our shapes so that idea isn't gonna work but I had another idea that turned out to be a total mistake I won't go too in depth but basically I tried to use the python image library to transform the screenshot of the grid so that it would be black and white and the black area would represent the shapes that are yet to be drawn and the white area would represent everything else then I would take the mouse and trace the outline of the black shapes in order to draw the shape on the grid while this idea didn't work out so well but surprisingly enough I did actually manage to get the code to transform the screenshot to work correctly but there was so many other issues at that point that it wasn't worth continuing with that idea so what do we do now well I have an idea since there are only four different grids that this Mini-Game uses why not just detect what grid is being used when the game starts and then trace a pre-made list of shapes that are in that grid that way we don't actually have to detect what shapes need to be drawn to get started I wrote out some code to load in an image of each one of the four grids and then I wrote out some code to perform an image search for each grid and once a grid is located it will begin drawing its corresponding shapes then I had the laborious task of manually writing out each individual shape in every one of the four grits [Music] but the code is now done so let's see it in action as soon as the game starts the program will begin drawing the shapes we are in expert mode right now and in order to get the program to be fast enough to win I had to manually adjust the delays to be as small as possible while also still working [Music] thank you while this Mini-Game has made me learn that sometimes the simplest solution is the easiest and if you want to start learning some new skills then you should check out brilliant.org brilliant is a website dedicated to helping you learn new skills in a fun and intuitive way they have a wide variety of stem courses that teach you skills that are invaluable and are rapidly developing world and if you are a busy person like me brilliant has you covered because it breaks down its courses into bite-sized pieces that allow you the freedom to learn no matter where you are and when you use brilliant you're not just memorizing formulas or equations you are building real intuition so that when you see a problem you will have the tools to solve it yourself and on top of all of that brilliant is very interactable and makes for an engaging experience so you might even forget that you are learning new skills to access everything that brilliant has to offer for a full 30 days visit brilliant.org code noodles or click the link down in the description below and what's great is that the first 200 of you will get a full 20 off of Brilliance annual premium subscription thank you brilliant for sponsoring this video but now let's get back to the chaos alright there is one last Mini-Game that I would like to automate in this video and hopefully it should be fairly easy as a palette cleanser for the first two mini games and this Mini-Game is called domino effect the goal of this Mini-Game is to jump from log to log by pressing a b X and Y but if you're too slow the logs topple over like dominoes and you sit there embarrassingly as your opponent wins to get started I took a couple of screenshots of the game and made a picture of each one of the four buttons then I began work on the script this was pretty straightforward basically each one of the four buttons is searched for and then if any one of them is far enough to the left then it means that it is ready to be pressed the only issue I had with the script is that for some strange reason the DS wouldn't recognize Pi auto gui's key presses so I had to instead use another Library called Pi direct input it seems like Pi direct input is like a sister library to Pi Auto GUI since it has the same functions as Pi Auto GUI anyways our script is now finished so let's see if it can win in expert mode [Music] thank you well it looks like we are three for three when it comes to our programs winning in the hardest difficulty this was a lot of fun getting to write some Python scripts to play mini games once again and if you guys would like to see me do another video where I automate a couple more Mario Party mini games just let me know in the comments down below anyways thank you guys as always for watching and for your wonderful support and I will see you in the next video bye-bye [Music] okay

Channel: CodeNoodles

Views: 137,971

Keywords: programming, coding, computer science, noodles, comedy, artificial intelligence, ai, super mario, mario party, mario party ds, ai learns, ai learns to play, image recognition, automation, mario, image detection, python, python for beginners, learn python, pyautogui

Id: JSnMEq2pz7Y

Length: 13min 16sec (796 seconds)

Published: Fri Jul 07 2023