AI Plays HTML5 Game FastWords. Destroys All Competition with OPENCV and Python.

welcome clarity coders in today's video we're going to hack another video game and see if we can get to the top score this is an html5 or flash based game i don't know about you but i grew up playing these stupid little games over and over again and this one gives us a great opportunity to show some new computer vision tasks but first a huge shout out to wyatt he beat my score in kikichop and posted the code in our discord as you know if you beat my high score post the code i'll give you a shout out in the next video i also had some comments that said that i should stop putting my face on the videos well i'm gonna put my face on the videos even harder now but that's enough of that let's talk about the game a little bit it's called fast words it's kind of a boring game but what happens is it shows you on the screen a word you have to memorize it and then click the tiles as they fall down the screen it's gonna bring up some interesting opportunities let's not waste any more time and jump right in [Music] so anytime you're working with a program like this the first thing you want to do is really break down what you're attempting to do here so the first thing that i think we need to do the small chunk of this problem is be able to read the text into our program so then our program knows what it should be typing for the word the word momentarily shows up in the yellow box we're going to take a screenshot of that yellow box using this line of code and that'll give us the image itself so to extract the words from the image we can use a python library the library that we're going to use is pi tesseract in just a couple lines of code we can pass in an image and it'll give us its best guess about getting the text out of that image itself so what i'm going to do is i'm going to take several screenshots about 25 different screen shots of that yellow box and we'll see what different words we get appearing there once we start our game so if i push run here you can see we're starting a new game and you can see it found hat so that's perfect it actually was hat and it found it let's go ahead and break this and we'll try again let's run the program again starting a new game it'll click on there it found hunger and here you'll see we have a little bit of an issue the first time it looked for the word it found hunger with a slash then it found uh two tildes and hunger and then finally it found hunger so that's why i keep checking multiple times to try and get the last instance of it showing up that's usually the most accurate now you'll notice on this next one we found diamond and the very last iteration found diamond with a tilde at the end now that's going to cause some issues right our program is going to try and type in that tilde down the road or if it was a slash or whatever and it's not going to be successful so we need to do some stuff to clean up this word after we find it in case it has some other characters now you may have noticed already that the word always shows up in capitals and it also doesn't have any special characters if you play the game for a while you'll also notice that there's never more than one word so let's add in some code to clean up our word after we find it you can pull in a library or regex using this line and now we can start to clean up our text the first thing i want to do is strip out so only alpha characters show up and no spaces and we only want capitals now we can go ahead and break this program and we'll try again and now i'm printing out the target word after we clean it as well so you'll see we're starting a new game we click on the try button which is just template matching we've done that before you can see that it caught a couple of images here it has the first time the first word it found was f space colors then it found tilde tilde colors and it found tilde chili colors pollen now this would have messed our program up but because of the cleaning we put in here you can see our target before cleaning oops that's wrong that's our target after cleaning actually so you can see our target after we did our cleaning was actually colors which is the word that we're looking for so that's perfect so now we have our program pulling in the characters off of the text and now we can worry about actually clicking on the letters now you might remember and have some ideas from our previous programs how we could do this we can use a for loop to iterate over the string and grab out each letter by letter and then we can use template matching to decide what we're looking for and then if we're looking for a c for example we need to have an image that looks like the c so our program knows what it's looking for so what i did is i used my handy friend paint and i took screenshots of each letter in the alphabet so you can see here that if we extract c is going to be our first because we're looking for colors we can go into the letters folder and it will have a c in there.png and then our program will use template matching to watch the letters flow by in this window here and when it sees that c it's going to click on it once it's clicked on it then it's going to move to the next letter which is going to be an o it's going to look through the letters again find the o dot png and then click on that letter so let's add this back into our program so now we're cleaning up the text and one thing i noticed while playing the game is occasionally it would misspell a word but it looks like most of the time in this game they're actually valid dictionary words so what i did to combat this is say for example it's spelled colors with two l's i used an outside library called spell checker that i just pip installed and then be after we clean up our text i'm also gonna run it through a spell checker and that's really it so we're extracting the text then we're using template matching again to find the locations and when i go to click on the letters i'm just adding on 20 pixels to whatever i found because they're kind of moving down so on the y-axis i'm moving 20 pixels down from where i originally saw the image and then clicking there now let's take a look at our best performance and see how it stacks up now this is our best run with the bot i'm going to speed it up a little bit for you guys obviously the only real mistakes it makes is occasionally it'll read in like two l's instead of one and that ended a lot of the runs early but other than that runs pretty smooth it finds all the letters if you're doing this on your pc you might have to adjust the screenshot values and maybe how far down you're clicking the letters depending on your processing speed i hope you guys all enjoyed this video if you did please leave a comment below the subscribes help the channel if you didn't like it leave a comment as well those help too if you want to talk to me directly operating the discord i'm there 24 7. i respond to all my messages [Music] if you taught me on the high score let me know and i'll shout you out in the next video and until next time keep coding you

Channel: ClarityCoders

Views: 19,121

Keywords: OpenCV, Python, codebullet

Id: 4QwVHgZEbNw

Length: 8min 2sec (482 seconds)

Published: Tue Jul 06 2021