Automating Android Games with Python: Simon Says (Top Score)

what's going on engineers we're back with another game automation and this time it's gonna be Simon Says on Android and to accomplish this we'll be using Python and the Python ADB client library so let's just jump in and see what kind of high score we can get so for those of you who don't know what Simon Says is it's a memory type game whereby you're given a sequence of colors then you have to repeat back the same colors each time you succeed it'll give you one additional color so we'll start by clicking play here you'll see that it picks red so then I'll have to click red now it will be red and then blue and I have to do red and blue and it just goes on from there so next time it'll be three red blue blue and I'd have to click red blue blue of course if you get the sequence wrong then it's just game over as you can imagine there's tens if not hundreds of these games on the Google Play Store so I just download the one that had the most downloads as I was brainstorming the approach for this game I had to overcome the challenge of needing to read the screen basically in real time in the stick hero video I also read the screen but I did so just by taking a screen cap with ADB the problem of that is taking a screen cap is very slow and you learn real women I build to make this work as if I can read it fast because you can see that it just flashes really briefly so I need to capture all of that I need to capture exactly when it changes or no matter what I would do is I would just capture the raw frame buffer with ADB but for that I would need a rooted phone which I do not have so I'm gonna cheat a little bit and I'm gonna read it directly from my monitor as I said earlier and like the stick here of video we'll be using the Python ADB client so I'll simply put this in here and this will give us access to the device through ADB and it'll allow us to run commands on the device as if we were touching it physically the first thing we need to do is capture a total of four pixels and it's going to be the pixel right in the middle the green square the red square the L Square and the blue square those four pixels should be the only thing we need and that will make sure that we're able to sample the pixels very very often to get the coordinates all need to look at I'm going to use a nifty tool called xzo tool and get mouse location - - shell I'll put my mouse over the green run it put over the yellow over the red and over the blue we can then do is take this data get over into our editor and get it ready to put into a dictionary I'll start by removing the excess stuff we don't need and we need to represent these values is a left top width and height so we can start by doing the left value which would be this the top value which will be this then we could specify a width of 1 and a height of 1 at our curly braces and then we'll need to give it a name so the first one's gonna be green and then yellow and then red and then blue put some equal signs in there and that will be good we'll be using MSS to create the screenshots will not remove screenshots they're just single pixels but we'll be using MSS to do that so we got a cranium instance of that game the actual pixels pretty easy it's simply SCT grab and then we specify the coordinate so we can do is wrap that in numpy array which will give us an array of the actual pixel the RGB values we'll set that to call it green pixel and then we'll make 4 more of these for each of the colors now that we have all these ready it's probably worth printing these out just to make sure everything's ok so we'll do each of these here I'll come over to our terminal and we'll run it and see what happens so this appeared to be the right colors we'll bring over our color picker here and we'll just look at the green the green is 51 84 44 that they are backwards but that's fine all we're gonna need is a couple the values anyways of course if we check out the yellow that looks correct the red is also correct and the bully is also correct so this is showing us what we needed so the only value we actually need is actually the red value and I'll explain why that is we don't actually have to detect is the color value there all we have to detect is is the color value not there and to do that we actually have to detect the blue background behind the four colors so the reason the red value is really helpful is because if we go over the green we can see we have a pretty high red 44 you know for red we have 180 for yellow we have 179 and for blue we have 43 but if we go on the background we have a red of 3 and that's just because it's a really dark blue so for the purpose of our program we're gonna say the button is pressed if the red value is below like 10 so because this 44 is the only value we need which is going to be the third and a list which is within a list which is within a list the way we're gonna have to subscript this is we'll call it like green red value it's going to be green pixel it's going to be the 0 and then 0 and then 2 and then as usual we'll duplicate this 3 more times for each of the colors the next thing we need to do is actually detect whether or not a square is pressed and as we said if a square is press it'll have a red value of below 10 now during my testing I did find that because I'm capturing data so fast there's a chance that the actual pixel will come back all zeroed so rather than try to throttle the speed down I'm simply gonna just check to make sure it's not a zero so instead of just less than 10 we're gonna do on a range of 1 to 10 so that's pretty easy to represent we can just do if 1 is less than or equal to green R which is less than or equal to 10 and then if it does find it will do print detected and green better yet instead of having it print something I think I'm gonna create a function called like detect next and then what this will do is this will be a function which is called which we'll just wait until a square is pressed and it'll return that value so rather than printing it I'll simply return like a G for green and I'll need to duplicate this 3 more times for each of the colors and I'll copy these into here now of course for blue it'll be B and R and Y for yellow and then because it's detecting I'll need to run it in a loop so it runs until it detects something so I'll put a well true there then I'll put a ever so slight delay just to keep the detection rate consistent so if all goes well we should be able to just print out detect next and then run our program it should give us either Gy rb2 pant which when it presses so we should be able to start a new game and let it pick one it'll pick a red so that'll be fine I can do is I can run the program and then when I click read it's gonna do red again and then it does R so it actually worked really well it detected the next car that was pressed but why stop here why did detect one so let's just put it in finit loop and we'll just detect forever so we'll come back over here we'll write it again I know it was blue i'll pick blue yes see it did three B's and three r's that's a little too fast now I can do BR and then it'll do it again see but that's too many to two and two so we have to find a way to make that work the reason is duplicate is because it doesn't know when to stop detecting basically as soon as it detects blue it just loops around again and then it sees it it's still clicked this means that the game likely shows it in the click State for about two to three hundred milliseconds but we don't want to actually try to time it perfectly so I'll have to find a better way and the best way I thought to do this is to basically change the function so that it looks to make sure the button is in the unpress state before it starts watching modifying this is pretty simple where I introduce a new variable called detecting we're going to set it to false by default and it will introduce a simple conditional that just says if not detecting then just continue basically don't actually check it and don't return so checking to see if the squares are in the uncheck condition it's actually pretty easy it's just the opposite of what we've done down here so down here we're saying that it's in the pressed position if the red value is between 1 and 10 so in the unpressed it's just going to be greater than 10 so we'll end up doing here is if not detecting and green underscore R is greater than 10 and and it will do it three more times for each of the colors so if it's not currently detecting and all of them are in the unpress state then we'll simply set the detecting variable to true so let's see if it works we'll start on your game here and then we'll run our program now it did blue so I'll click blue it should do blue again you can see it does B B for blue and blue so I'll do blue blue does B B and then G for green so you can see now it's only doing it one at a time now which is exactly what we want so now that we have a function that does a really great job of just detecting the next color that's press all I really have to do is just do the logic for actually processing the moves so need two variables for this the first one we'll need is a moves variable which will tell us how many moves it should watch for because it'll do one and the o2 two and three and four so this is just going to be a simple increment and then we need a list to hold the actual moves we'll need to make so in an infinite loop which will represent each of the rounds and in iterator for I in range moves we will then detect the actual next color Betsey's once it detects a color we will append it to the colors list just for debugging will print the value of colors just to make sure everything's good now because when you play the game once it tells you what cars to press you can't immediately press the color so I'll have to introduce a short delay of about one second and it's at this point that we actually have to make the touch action on the phone to pick the actual car that we want so it'll be a simple iterator for color in colors and then in here we're going to detect what color is what so if color equals G then we'll have to do device dot shell input tap and then this is gonna be the XY value of where we want to tap so for the green we'll come over to our thing here we'll put it right in the middle of the green and it's about 265 by 470 so we'll put that here 265 by 470 and just to save some time I filled out the remaining three colors along with the remaining three XY coordinate pairs and then the very last thing we'll need to do is simply increment the moves variable by one and reset the colors list to empty and I think it's time for our test so we'll go to terminal we'll start our program and we'll click Play we'll take our hands off we'll watch it work so it sees Y and then it press Y the score went till 1 and then Y blue and it picked Y blue the scores now with two white blue blue good scores at 3 y blue blue red and it's good to go it scores it for so it's on round 10 now and everything still seems to be fine seems to be working so we'll just cut it off right here I did let this run earlier I stopped it at about 31 as you can see on the leaderboard I currently hold the number one spot at 31 and the previous person only has 17 but so sorry beautiful carp you are just no match for Python however I fully expect within a couple days of this video being released that these scores are much much higher and somebody dethrones me anyway that's all there is if you have any questions or comments about anything that's on this video please make sure to leave them below in the comments also I came up with and implemented this approach in probably a combined maybe thirty minutes so if you have any improvements or if you have a better way you think I could have done this definitely let me know in the comments below as well this was a really fun project in fact all of these Android automations have been really fun and I really hope you enjoyed watching and that hope to see on the next video take care

Channel: Engineer Man

Views: 53,236

Keywords: android automation, python, simon says, engineer man

Id: aosN_FvW2e4

Length: 11min 30sec (690 seconds)

Published: Sun May 31 2020