AI Destroys a Mobile Game in Real-Time with OpenCV (Object Detection)

welcome clarity coders glad to be back today i'm going to show you how you can automate mobile apps or games using python the technique i'm going to use uses a powerful computer vision library called opencv to create an ai to play the game of zigzag now this game is extremely easy you just tap the screen to change the ball's direction and try to stay on the track with my human skills i'm able to score around 50 points it's actually really hard however there are a bunch of trophies that i'd like to accomplish one which is an ultra rare trophy that only point five percent of people have accomplished and that's a thousand point game now following this video i'm going to release a detailed series on how to use all the functions in opencv to create your own ai it's all going to be free and posted on youtube so please subscribe so you don't miss out on that let's not waste any more time and jump right in [Music] so first things first like always all the code is below for you to download for this completed project go ahead and grab that if you haven't already i'm going to walk you through a little bit of setup because i do it a little differently than some other people that automate android apps and games and things of that nature now one of the things you'll notice right away with other people doing this like engineer man is the code that they're doing is too slow to actually play real-time games now the game that we're gonna be playing has to be fast enough to work in real time so the first thing we're gonna do setup wise is we're going to use an emulator i'm doing that for a couple reasons the main reason is you could hook up your phone over usb but some of you don't have an android phone the emulator that i'm going to use is ld player i'm not tied to that in any way you can use whatever emulator you would like to use you'll notice it comes with some apps pre-installed on it don't judge me for those i don't recommend them in any way they're just what comes installed on the player from here we can open up system apps go to the play store and download whatever we would like now you're gonna have to but that's all download so you can go through that we're just gonna search for zig zag and install that game now that we've got the game installed we can go ahead and open it up you can see it gives us a nice little player shaped like a phone i'm gonna resize this just a bit good lord that ad scared me so watch your volume i'm gonna go ahead and kill my volume so i don't my pants here okay that had an ad and then kind of died out on us let's try and open it again cool you can see we got our zig zag set up now now the thing that i did to set this up is i kind of put it in the top right corner it just has to be sorted in the same location each time when we're grabbing screen grabs and stuff like that it doesn't have to be exact when we take a look at this we can see that the game's exactly what we expected so we're going to start off moving right when we click we change to left and we get our best score each time we hit retry and we can play again you also notice that there's a store where you can buy things with your rewards that you get in games when you collect the diamonds so we're gonna unlock all those as well during this video and then of course the trophies if you click on the trophy you can see all the achievements that we have to unlock and we're going to try and accomplish all that in this video so how do we start the process of adding ai to this program the most important thing or the thing we have to worry about first is our ball on the track how do we know where the ball is and how close it is to the edge of the screen opencv makes this really easy it has a module called hue circles so we're going to pull in our screenshot into our program it's going to turn it into an array of numbers and then we're going to pass it into the hue circle function you can see here that it's found the ball on the screen now that's going to pass us back some information it's gonna pass us back the ball's x location the y location and the radius of the ball itself so now we can track where it's going on the screen now obviously we're not making any decisions yet because we have no idea where the edges are or anything like that but you can see here as it plays through it can see where the ball is at all times now our next step is how do we find the edges of the track itself you haven't seen it yet but you'll notice a couple things one when you hit the diamonds they explode in a cloud that can kind of cause a little distraction there in different colors and also once we get so far into the game the game colors begin to change as well so we can't simply test for a pixel value and then click when we hit a pixel value or anything like that because we don't know what the next color will be so we can bring in another opencv function this is called hue lines all these functions have a lot of different parameters and to find the exact parameters you need you really have to dial them in and i'm going to show you a program that i use later to find these exact values that's available in the github of course let's look at what it looks like when we pull in the hue lines into this program so now what we can do is we can use our balls location and test to see how close it is to a red line now if we're moving to the right we only want to look for the right edge of the track if we're moving to the left we only want to look for the left edge of the track so as our ball progresses through this simulation we're painting these lines constantly now i'm going to slow down this video a little bit at this point just so it's not too flashy the lines kind of jump around but you can see here that it paints the lines as we go throughout the simulation now one thing you might have noticed and that i alluded to earlier is if you look at this picture it's making some mistakes right you can see that the two in the vision up in the top right corner it thinks there's a circle there so doesn't that throw off our calculations yes it does so what we wanted to do was limit the area of where it looks for the ball so what i did is i picked out the x and y coordinate that was close to being around the ball and i only look for circles in that area so therefore i'm going to ignore the two and things like that now with the edges here you'll notice it's doing actually a really good job of finding just the edges and you may be very curious why it's not finding edges on things like the diamond that was another workaround we had to do we actually had to mask that specific color out so one thing we did with our program to accomplish these things is we look at a very small window so this is all our program's actually seeing this tiny view right here now you'll see we also eliminated the diamonds so if you look at this next small picture here we put a mask based on color to mask out the diamonds color itself so that our image doesn't see it then we took just the lines of the program so we eliminated the diamond already and then turned everything into line values and then we drew the edges based on that so you'll see here these white lines are the only thing our program sees so we're really simplifying what's put into our program now of course for our view i made it a little bigger window so you can kind of visualize what's happening but for the computer's view we want to keep it as simplistic as possible now you may ask yourself again how did i come up with these values i'm throwing around a lot of numbers and different things like that i created a little program called line detection in the github that you'll find where we can adjust values with these sliders so if you notice the mask for example i have a lower hue a lower saturation and a lower vibrance something or brightness what's the v stand for an hsv anyways you can adjust those so if we want to increase them we can lower the hue lower the saturation and lower the vibrance or whatever it is and you'll see that our mask starts to increase in size and if i keep pulling these down eventually we're going to mask out everything now you can see now our program doesn't see anything at all so it's finding that balance and if we go the other way and we don't mask out anything so if we turn up these all the way you'll notice that our mask is nothing now what you see here is you'll notice that diamond is back in there and you'll see on our image there's a red line on the diamond now if we hit that it's going to throw us off the track so that's how we had to dial in these values you'll notice there's also some a hue threshold a minimum line size a min a maximum gap and different things like that that will cause different things to happen when i play with those you'll see we get more or less lines but the program is going to allow you to dial in those values pretty easily by just playing around now the line detection program you can actually open multiple images at one time so you can look at a lot of different things at once we also had to pick a value of how far to the right or left we wanted to test for an edge i settled at around 38 pixels that was something that i played around with as well now once we have this all put together we can actually test our program now as you can see we're feeding data through our program constantly it's finding the edges now you'll notice a lot of times some of the edges go missing on the program itself so what we did to combat that is every time we we actually did a check for a white value as well you can see that there's pure white sometimes on the side of the track we check for that and we also click if we hit that white value as it keeps progressing you can see that the colors continue to change on the map itself but that isn't really a problem because we're not worried about the colors at all the hue lines are performed with a grayscale image so i actually use this chunk of code to grayscale the image before i pass it through that hue lines function you'll notice here that it simply tears through the map now of course this didn't happen right away this had i had to do a lot of testing to find the exact values that worked well with this track but i can assure you that i did not do this myself because i don't have that ability and you'll notice we gave it a look ahead value of 38 between starting at 20 pixels away and ending at 38 so it has some time to find that line if it misses it initially you'll notice sometimes the edge is missing on an image but maybe the exact next frame it's back now that a thousand point trophy was the hardest there's a couple other lower values and there's also a thousand game one if you played a thousand games i did that by simply automating this to play all night long and it actually i left it on for like two days on accident and it played over 5 000 games now obviously you can recreate this from what i've shown you here today and i hope that someone does if you thought this video was cool please let me know any other games you'd like to see me try and top the leaderboards on if you really liked this program but you don't fully understand opencv and you really want an in-depth description of how i did everything you can you can go to the github and download it but you might not understand the code exactly if you're not familiar with opencv i am going to release a full course going over in depth how i break down these opencv functions and how they work on a multitude of different projects if you want to be a part of that course it's absolutely free all you have to do is subscribe and hit that bell so you don't miss out and i will post that within a couple weeks the first iterations of that if you have any questions or you want to get started or you want to talk to me or the people that i hang around with the other coders i hang around join the discord the link will be in the description below we have people on 24 7 all over the world so you can get the help you need whether that's downloading this code how to get started how to install python whatever it is we've got your back in that group join ahead like i said if you haven't and until next time keep coding

Channel: ClarityCoders

Views: 393,344

Rating: 4.9375515 out of 5

Keywords: ai beats game, AI beats android game, AI Unlocks achievements, bot beats game, bot beats android game, bot unlocks trophies, python beats games, ai beats gae, ai beats gale

Id: OzWU18AwS9k

Length: 12min 18sec (738 seconds)

Published: Mon Dec 28 2020

Reddit Comments

[u/solarpanzer]

Nicely done with classical image processing. I wouldn't call it AI.

Edit: People have been using the term AI synonymous with ML lately. But I guess in the classical sense of the term, this does indeed fall into the domain of AI.

[u/bkgn]

I've always much preferred memory-based approaches to bots over CV, but I've never tried writing a bot for Android so I have no idea how difficult reading memory on Android/Android emulators is.

[u/haroldjaap]

Nice! I think you'd like code bullet as well