Automating Android Games with Python: Stick Hero

what's going on generis today we're gonna automate the game stick hero and try to get insanely high scores but besides that we're also gonna solve a fun problem with Python so win-win also this score right here I actually got it it's not fake and I'm confident that if I wanted to get a hundred times that I could so let's just jump in and see what we can do so first of all look at the mechanics of the game the game is very simple you get a little ninja looking guy here and all you do is when you hold the mouse down it draws a stick in the air and then when you think it's long enough you let go of the mouse and it drops across and hopefully you land it on the pillar if it's six too short is a 60 long then you lose so the script gotta bill is to somehow figure out the distance between the character and the pillar and then take that distance and somehow apply it to the exact amount of time to hold the mouse down so as far as automating the actual long press on the screen that's pretty straightforward we can do that with adb shell you do adb shell input touch screen swipe we could specify the start and the start coordinates and then the end coordinates and then specify in milliseconds how long to hold down so this would make like a two-second long press if we run that we can see it makes the stick that's way too long and then we lose the game of course the hard part is figuring out the actual delay it's not always going to be 2,000 so it's on we can calculate it upfront because the distance changes all the time so it's gonna be some value in between so as I was thinking up of an approach for this I thought if I could take a screenshot of the screen and then unpack that to a two-dimensional array of pixels I could take a single row of pixels and analyze those to figure out distance and the row of pixels how with a look at would be somewhere below the ninja I have the benefit of having the pillar he's standing on is solid black the pillar we want to go to a solid black and then the stuff in between is not solid black so if I could figure out the exact location of the start of the gap the end of the gap and then the width of the pillar then from there I can calculate the relevant values to figure out how long the stick has to be for this we're still going to use ADB but we're going to use Python ADB so we'll do the proper imports and it will carry ourselves a client by supplying the host which will be localhost and the port will be thank 5037 we'll need to get all its devices and select ours so we'll do devices adb dot Isis you'll do a quick check to make sure that there's actually a device there and if there is we'll set device two devices zero we're just going to assume that that's gonna be our device at this point I just want to verify we have a connection to our device so I'm going to take the same command that I ran over here and instead I'm gonna run it through Python so I can do twice shell and I can specify this command here then what we'll do is we will run that and see if we get something over on the screen and we do great so we're connected our next objective is to get a screenshot of the screen and fortunately python adb makes it really easy if we can do image equals device screencap and then we can save that to a file call it at screen dot PNG open it for writing binary as f f dot right image now what this should do is give us an image called screen dot PNG that we can then use later to analyze let's just run it again and make sure we get that image okay so I think it worked look in our directory here we got screen dot PNG so here we go we got our screenshot of the screen now that we have the image will be unpacking that two two dimensional array of pixels and for that we will be using pillow and numpy opening the image with pillow is simple where we use our image variable it will do image dot open and then screen dot PNG will then take that image and we'll load it using numpy dot array specify image and the type will be numpy dot u n Tate at this point we'll do a quick print on image just to kind of see this data make sure it's there so we run our program and we get a long list of things and this data seems to be consistent with what's on the screen we have some colored values up here which represents the top of the screenshot and then we have black values down here it's these three numbers that we're concerned this is the RGB and zero zero zero is gonna be black but we don't care about all the data we just care about one single row and my screen is going to be 1920 pixels high so we want row roughly 1400 that should place us just below where the ninjas so instead of printing everything let's print image 1400 and this should give us the 1400 and first element in the array right against it we get and once again the dad is mostly consistent the first pixel is almost black and seven five six so we'll have to deal with that in our code and then we have a number of elements that are black that's I represent the first pillar and at the very end we have a number of light elements that's going to be the area after the second pillar I'm not sure if the fourth number is I'm assuming it's an alpha value but we can just ignore that for our purposes so at this point like to simplify it a little bit by converting it from a numpy array to just a list of lists of pixels without that alpha value so for this I can use simple list comprehension I can do pixels equals I for I in image and then remember going to take the 1401 and then for each I will convert that to lists and then we'll just take the first three values in that list then let's print out pixels and just to make sure we got what we want run the program and that looks good now we have a full list of the pixel data and as I scroll through it I can see that it represents mostly what's on the screen so it starts with that weird seven five six pixel that must just be a gray line they have on the Left we'll go ahead and remove that and has followed up by a series of black which is going to be the first pillar and that's followed by a series of light colors that's gonna be the gap followed by another series of black and then more light color so we have some pretty clear boundaries in this data so there's three very specific X values that we need we need the x value where it transitions from solid black to color and then we need the x value where it transitions from color to black and then we need the x value where it transitions back to color from there some simple math should be able to give us our distance so we'll start by creating a list called transitions this is going to hold the values that we need they don't need to loop over each pixel and we're going to need to use enumerate because not only do we need the pixel we also need the index of the list because the index of the list is going to relate to our x value that we're going to need so put pixels into here now we're going to need the red green and blue from each so we'll do RGB equals and then we'll need to use list comprehension again to convert each one into an int so we'll do into I or I in pixel they were just gonna print out the RGB just to make sure that that works fine run our program and everything's good now we just have three simple integer values which we can look at to figure out what's going on here so our first problem is in that first pixel remember it's not solid black with some other colors so rather than just guess and say okay it'll be just one pixel or two pixels or maybe zero the best way I thought to do it would be to just remove or just ignore anything up to the first solid black pixel so for that we're going to use a variable called ignore called call flag and then the first thing we'll do after we get our pixel values is to check to see if the pixel is not black and if it is then skip it so for that will say if ignore + r + G + B does not equal zero then continue once a finally does hit a pixel that is actually solid black then we could just continue on with our iteration and we'll set ignore to false that way it doesn't do any more checking after that now as we're seeking through the row of pixels we're gonna be on black and they're gonna be on color and then black again and then color again but the problem of color is that all the colors are not uniformed they might be different so we can't just say if the pixel changes from black or it changes from something else back to black then store that value we can't do that directly because all the colors are different so we're going to do is have a variable something called like black equals true to store at what point in the row of pixels that were on fortunately black is always black so our transition points are just from black to something else and then from something else back to black according to this variable so the way it's going to work is we're going to say if black meaning that we're currently on black and the sum of RGB is not zero which means it's a color then we want to flip black to its opposite we want to add the current index to transitions so do dot append I and then we want to continue our iteration now fortunately the opposite of this is the exact same so we're just going to copy this down except instead of if black creative if not black and then instead of the sum being not equal to zero we're going to do equal to zero and if all works well we should be left with exactly three values in here and start at the cliff at the start of the pillar and the end pillar so let's save this we'll run our program and hope it works and it does so this says is the first transition is at exposition to 42 the second transition is at 5:41 and then the next is at 782 so we can check that over here on our program so I'm just gonna hold this down just so I can see the number so I'm showing 242 here if I move it over to the start of the target pillar I'm seeing about 541 and if I move it to the end of the pillar I'm seeing about 782 so all that looks good so rad an index into transitions will just extract the proper values and name them so we have three positions now and it's going to be start target one and target two and that'll represent the start and end of the target equal to transitions now it's kind of a little math so as far as the gap goes between the start and target one that's simply going to be gap equals target 1 minus start next we'll want the actual width of the target so we can get that by just subtracting target 2 minus target 1 now we need to calculate the actual distance now as far as a distance goes we want to make the stick long enough to drop right in the center of the target so the distance is going to need to be the full size of the gap plus half of the target so at this point let's print out the value of distance and then that way we can go back to our screenshot we can check our math so let's run this and see what it tells us for this current screen it says our distance is five hundred and eight point five so we can check this math pretty easily well drag our indicator right to the edge here and I'm showing an x value of 240 if I do 240 plus 508 then get 748 which should be the value right in between the red here so for my mouse right here I get 750 that's close enough and last but not least we actually have to send the long press command to the phone to actually hold it down for the right amount of time for this we can use device dot shell we can do input whoops input touch screen swipe start and and then we need to actually figure out how the distance equates to time now it took out a stopwatch on my phone and I checked the amount of time it took from when I started holding to when I ended and I found that for a distance of about 300 the stopwatch read about 300 milliseconds so this makes me think is that the game renders at roughly one pixel per millisecond I thought that that would be a sensible starting point which makes it really easy because that means the distance is also the millisecond delay so in this case we'll simply drop the integer value of distance into here so let's run at once this will be the first time and hopefully this works we'll run the program it's gonna get distance a 330 and it did it perfectly so run it a few more times just to see it was aiming to adjust so 627 that dropped a little far let me see if that's just a fluke right again 602 that dropped a little far it keeps dropping a little far so I think we have to apply a coefficient to the distance because it's only a little far we're just going to take a percentage of the total distance by just adjusting distance here so we'll put this in parentheses and then we will multiply it by say point nine eight it'll be 98 percent of the distance as milliseconds delay we'll come back over here we'll run it again it's it's still a little far let's write another time okay that one landed perfectly right one more time awesome that's hanging right in the center look I keep in mind as well it's not perfect science the long press might not be a hundred percent accurate but it's pretty good it's hitting it in the center almost every time of course last but not least we just have to introduce a loop now because we don't want to do this manually every time so we can simply scroll up here we'll start at the point where it takes a screenshot will throw a well true here and that will indent everything out one then we'll just have to have it sleep will do timed out sleep a sensible amount maybe 2.5 seconds nobodies need to add that import up here at this point we should have our automation completely done so we're just going to run this and just watch it for a little bit and see if it works perfect that worked the delay is probably still too long we could probably shorten that a little bit let's do that so we'll shorten this down to two point one no right again and hopefully that doesn't mess anything up okay got it yeah that's that's perfect perfect watch a little bit longer perfect again that was a little far but still seems to be working yeah it seems like the shorter it is always gonna be a close one oh no perfect so the two point one was a little fast because what it is it took the screenshot from the previous screen and applied it to the next screen and then it stopped working so I think like 2.3 2.5 somewhere around there is the sweet spot I'm pretty sure if you ran it just like this you would get a score of as long as your device stayed online automating games is not only a lot of fun but it's always an interesting challenge because all the games are different and they all have different ways that you would need to automate them I hope to do more of these in the future and if you have any questions or comments about this video or if you might have a better way of approaching this or maybe a completely different way then you should definitely let me know or better yet submit a pull request for the project and maybe I'll merge your version into the github repo or in that I expect more of these videos in the future I really enjoy doing this so I'm gonna do a few more and are in that hope to see in the next video take care what was it gonna get it yeah

Channel: Engineer Man

Views: 272,674

Rating: 4.9639134 out of 5

Keywords: android, android automation, python android, stick hero, adb, monkeyrunner, engineer man

Id: Du__JfXqsAs

Length: 15min 22sec (922 seconds)

Published: Sun May 10 2020