Automating Android Games with Python and OpenCV: Pong

what's going on engineers welcome to the final and as it turns out the most complicated game automation I've done so far and that's the Android game pong using Python and OpenCV of course mostly though a Python is for those of you don't know it OpenCV is it's a computer vision library it's going to allow us to actually watch the game that we're automating in real time and it allow us to write software to react to certain things that happen in that game in real time so let's just jump in an autumn a pond for those of you who are unfamiliar of the mechanics of the game pong it's quite simple you simply have a ball that goes between two paddles and then you just move the paddle to hit the ball back to one side eventually somebody will miss like I'm gonna miss here ok I'm gonna miss now and then that's it for the game so as far as what we need to automate that's pretty simple because as far as the input that we need to give to the game it's really just this bar that can go up and down so what we need to do is write something that can move that bar chain to position it and the bar would have to react to wherever the ball is as far as wearing some code to actually track the ball in real time that's where the computer vision comes in so I think that would be step one we'll put aside the part about controlling the paddle for now we'll focus on just getting OpenCV to work to actually track the ball in real time as it goes around the screen to start we need to import CV - that's our computer vision library as well as a few other things that we've seen in previous videos like numpy and an MSS and python pillow now to watch what's happening in the game in real time we're not actually taking a video feed necessarily what we're doing is we're taking screenshots very rapidly and we're just feeding those into open CV for processing let's do that we'll create a new instance of our MSS to do screenshots I will start an infinite loop which will basically Rio screenshot and process it infinitely to actually take the screenshot we can use the grab method on SCT and what we need to pass this is the size of the screen and then where it should take the screenshot so as far as what portion of the screen we want to watch it's pretty much the entire thing here with the exception of the buttons on the side here and the reason we want to exclude these buttons is because you'll notice that the home button on the Android looks a lot like a circle and if we don't exclude that we're gonna get some false positives so if you're using xdo tool to figure out the coordinates of everything what this works out to be is the exposition of 2560 the Y position of fourteen seventy and then nine hundred and ten pixels across and 540 pixels down now that we've done this we can convert it to a number array by doing MPA ray and provide the screen then we use CV to dot I am show give it a name like output and then provide the image this will do is this will put the image on the screen so we can actually debug what's going on and then we'll just add some code here so we can click the Q key to basically break out of the screen so it should be good to test just to make sure that we actually get the image so we'll open up the game and we'll run this code here pong that PI and you'll see what we're actually seeing here is this is what the computer vision sees and at the moment it's just exactly what's over here - the buttons on the right hand side but you can see it whatever we do over here is also repeated on this other screen the thicker the head next is we want to draw a circle around the ball that's going across the screen and once we have that it should be pretty easy to determine where the paddle should be before we actually start detecting where the ball is the first thing we have to do actually is get rid of this score right here and that's because the score might be a six it might be a ten and those are things that look a lot like zeros so we actually have to basically cover that which is fairly simple we can use CV - that rectangle to draw a rectangle on that if you actually want to see it for testing we can change this from a black square to a green square and then when we run it you'll notice that that area is now covered by a green box but we don't want it to be green want to be black so we can change this 255 back to a zero the function that we're actually using to detect the ball going across the screen is called huff circles and Huff circles expects a greyscale image and right now we have a color image so we'll have to change the color space from color to gray so we'll create a variable called gray and we will do CV to dot C BT color give it the image and then pass it CV to color BGR to gray and I might be thinking yourself hold on pong is already black and white why need to do this and you're absolutely right pong is already black and white but the thing is the screen shot is still taking in color so we still have to convert it to a a skill grayscale makes things a lot easier because rather than having to sort through like sixteen point seven million different colors you only have to sort through 256 colors to actually detect the circles we create new variable called circles we call c v2 dot puff circles we pass at the gray version of our image and cb2 tough gradient as far as the additional parameters supplied to the Hough circles function I mostly just dialed those in through trial and error until I got a consistent detection of the actual ball going across the screen the next step is going to be to actually draw a green circle around the ball as it's going across the screen so for that we'll start by testing of circles is not none next we'll convert circles into a UN sixteen numpy right and then for each point that fine so for PT in circles zero semicolon we will draw a circle around that in green with a five stroke and because the circle array is in the format of Center X Center Y and then radius we can unpack that into X Y R call C V to circle on the image pass it the X Y as the center of the circle and then the radius and then the color so I think they are ready to run see if it works so we will start our program we can see now that there's nothing on the screen after we click wall you could see that there is now a red circle around the ball as it goes across the screen so what we're doing at the moment is we're actually detecting where that circle is in real time so this seems to be working fine however the one value here it's extremely important to us is actually the Y value because that determines where the ball is up and down on the screen which is going to determine to where we need to angle our paddle so let's rerun this again but this time we're gonna output the Y value into the console over here and what this will do is this will show us where it is so you can see that the higher the ball goes the lower the number gets the lower the ball goes the higher the number gets so we're just about ready to move on to step two which will be actually moving the paddle according to where the ball is so the approach we're gonna use for this is we're actually going to start a second thread and that second thread is going to be responsible for looking at the current Y position of the ball and then trying to move paddle to that y-position to match it and the reason we have to use threading here is because it's the only way that we can keep reading screenshots really fast and move the paddle at the same time so that means we need to actually store this y value so that way the second threat to access it so we'll come back up here and we'll store it in a variable colleges ball Y it will say it's a 0 by default and then we'll come back down here and instead of printing Y we'll do ball y equals y we'll need a couple new imports make this happen we'll need a way to actually access the adb client and they'll need to import threading so we can use threads they will need just a little more boilerplate code it's the same code that I've used in all the other videos and all it does is it just connects to the adb client and allows us to send different commands to the phone so does for constant values that we'll have to establish up front to make sure that we can move the paddle in an effective manner and the first one's gonna be the center X point on the paddle the center Y Point and then the range which the pal can move to save time I've already done this ahead of time and these are the values the paddle is at an exposition of 1700 it's at a Y position of 540 and then the minimum it can move is 160 all the way down to 920 it's now it's time to construct a thread that's gonna be responsible for actually moving the paddle so we'll create a function I guess we'll just call it move paddle it'll be fine and what this will be this this will be the entry point into it with the new thread well create a thread called teeth to do new thread with you threading dot thread specify a target which would be moved paddle and that we called t dot start on that because the move paddle function will need to modify the paddle Y variable we need to use global paddle Y the intents for this thread to run pretty much until we close out the program so introducing a variable called running which will default to true and then we will run this while running and at the bottom here you'll notice that once we exit we set that rank to false and that will cause the thread to exit what we're interested in now is to determine where should the paddle move to so we'll call it variable just called 2 now we'll start that as ball y x 2 now remember ball Y is the variable that's constantly being updated as it determines the new ball position the reason I have to multiply it by two is because our phone mirror here is actually half the size of the normal phone so if the ball is actually at four hundred pixels according to my monitor that's actually at 800 pixels according to the phone if I was writing this a full-size which I'm not just because it would take up the whole screen I wouldn't build of doing anything else then I could lead out I could leave out that multiply to next we have to do is clamp the two value to somewhere between paddle Wineman and paddle ymax so this is as simple as doing if ball y is less than paddle y min then set to to paddle Y min and of course the same is true for max so I'll just copy that change this to Max and instead of less than we'll do greater than now if you recall from previous videos to actually do a swipe gesture which is what I'm gonna do we want to basically swipe the paddle down a little bit it's done in the format of input touch screen swipe x1y1 x2y2 and then the duration of milliseconds it should take to actually do the action so of course the origin word should move from is simple so x1 is simply the paddle x y1 of course is paddle y x2 is going to be paddle x again and that's because the x value never changes where the the paddle goes and then y2 is going to be equal to our new variable called 2 and of course our device that she'll call look like this which is going to be input touch screen swipe x1 y1 x2 y2 and then 10 millisecond duration to actually do the action and of course last but not least because we actually moved the paddle we actually have to set the paddle y-value to the to value and now i think it's time to tensed so we'll start by doing wall mode and then we'll turn on our code you can see that it's moving the paddle around so it sees the ball moving and then it moves the paddle now it is kind of slow and that's because we are up against the fact that ATP is not that fast and that's why it's sort of glitchy and if the ball were to go crazy it would actually probably not do so well yeah you can see that got screwed up there now in the future video I'm going to make a high-performance version of this the reason I want to put a version 1 first is because the high-performance version I'm going to have to my phone I just haven't done that yet additionally I didn't want the version one to rely on somebody having a rooted phone anybody with Android and a Windows or Linux system can make this work no problem once I rip my phone that's gonna allow me to do two things it's gonna let me stream a raw video at as fast as I can possibly take it and process it it's also gonna allow me to stream the actual events directly on the device without having to go through something like adb shell so that means instead of getting a couple movements a second i'm gonna be able to get like a hundred and that's really all there is to it although it didn't perform optimally I do believe the approach is probably the best I'm going to get and of course if you have any questions or comments or if you can think up of a better approach or something that I could have done better you should definitely let me know below in the comments and are in that definitely be on the lookout for the high performance version of this video and as always I hope to see in the next video take care

Channel: Engineer Man

Views: 47,957

Keywords: android automation, python, python opencv, python opencv android, python pong, engineer man

Id: U2dS8pu2baY

Length: 11min 55sec (715 seconds)

Published: Sun Jun 21 2020