Image Recognition DESTROYS Mario Minigames

[Music] new Super Mario Brothers and Super Mario 64 DS are two Nintendo DS games that have some amazing mini-games and in my last video I wrote a Python program to automatically play a mini game called wanted you guys liked it and because of democracy I'm going to be automating more mini games in this video so let's get started the first Mini-Game I'm going to be automating is called sort or explode it is a simple sorting minigame with bombs and there are two colors red and black the gimmick is that the bombs explode if they aren't sorted quick enough meaning that speed is key especially later on when it seems almost impossible to keep up which is perfect for the program to play since the program I'm going to be writing needs to be able to recognize the bombs I quickly rip some assets from the game and began writing the code to load in the images and to try to recognize the bombs the way the program is able to locate anything at all is that it takes a screenshot of the game every few frames and then uses a function called locate in pi Auto GUI an image recognition library to try to find the bounding box of the first bomb it could find so let's see if it works so now that we know where the bombs are located we need to drag the bombs into the goals this is actually pretty straightforward all it involves is moving the mouse to the position of the bomb and then using a function to drag the mouse to the goal corresponding to its color so let's try it foreign but it's a little slow so let's turn down the time it takes for each drag nope here's the issue what's the duration of each drag is under 16 milliseconds then the dragging is too fast for the game to process it so we need a new idea and that's when I realized something when the mouse is dragged it isn't a continuous movement in fact it just moves the mouse a certain distance instantly so what's stopping me from just moving the mouse over the bomb grabbing it and then immediately moving it to the goal let's try it [Music] foreign [Music] [Music] it seems once the bombs start to overlap the mouse can sometimes grab the wrong Bomb by accident so I added some code to make sure the bombs actually make their way out of the spawn before getting moved so how many bombs do you think the program can now sort 100 500 maybe even a thousand well let's find out the answer [Music] [Music] [Applause] [Music] thank you [Music] [Applause] [Music] [Music] [Music] thank you [Music] foreign [Music] I was honestly hoping it would make it to 9999 but 3100 is good enough for me continuing the theme of bombs we move into the next mini game bomb squad in this Mini-Game hordes of para bombs drop from the sky and try to destroy a bed of flowers so the goal is to use this giant slingshot to shoot cannonballs at the bombs before they destroy the flowers to start like before I wrote some basic code to load in some textures but this time it loads in different angles of the bombs because as they move through the air they turn in different directions then in the code for locating the bombs I use a function called locate all that generates a list of all the visible bombs on screen and it works now that we can locate all of the bombs on the screen we have to select one of them to be shot at so I added a loop that finds whichever bomb is currently lowest on the screen to Target we need some way of calculating where the Cannonball needs to be pulled in order to line up with the target pop if we draw a vector from The Cannonball to the Target bomb we can then calculate the components by subtracting the bomb's position from the cannonball's position then using inverse tangent we could find the angle of this vector and flip its direction around by adding 180 degrees or Pi radians then at the end we recalculate the components of the new Vector using cosine and sine and then move the mouse to this new location and Bam it works the only problem that really arises is that when the bomb is really low on the screen it isn't good for the slingshot to aim towards the ground as it is unlikely to hit any other bombs as it moves by so I added a little if statement to retain the original angle of the bomb if it is lower than the Cannonball so let's see how our script can do [Music] foreign [Music] you know it didn't do perfect but I'd still call that a success the next and final Mini-Game I will be automating is called memory master this Mini-Game is a much more difficult version of Memory Match a mini game where you must pair cards together that are the same by flipping them over in this harder version however there are 16 cards at the beginning instead of eight making it nearly impossible to complete without outside interference to start the script I load in all the different cards and then locate them and add them to a list to keep track of them each card is assigned a different value and blank cards that are currently unknown have a value of zero so as the program runs the values of the cards only change if a card that is located has a higher value than the previous this allows the program to remember what each card's value is even after it is no longer visible to the viewer and all that needs to be done is to make it so that the mouse clicks on the cards that pair together and then when no match is found for it to click on the cards that still have an unknown value to complete the search so let's see how it does [Music] all right [Music] [Music] foreign [Music] [Music] [Music] thank you [Music] [Music] all right [Music] dang it it's stuck at least it got past 100 Levels but I'm kind of confused what caused this to happen thank you guys for sticking to the end of this video I had fun working on the scripts for these mini games and if you would like to check them out they're on my GitHub link down below anyways thank you guys for watching and for your support as always and I will see you in the next video bye-bye [Music]

Channel: CodeNoodles

Views: 539,053

Keywords: artificial intelligence, ai, super mario, programming, machine learning, ai learns, ai learns to play, coding, computer science, image recognition, python, image recognition python, mario, super mario bros, new super mario bros, ai learns mario, comedy

Id: sHDrOlrxjUc

Length: 13min 35sec (815 seconds)

Published: Sat Nov 19 2022