How To Bot with OpenCV - OpenCV Object Detection in Games #9

building a bot with opencv is a matter of combining the object detection techniques that we've discussed for the last eight videos with some gui automation now that you know how to find an object using opencv you just need to use something like pi auto gui or pi direct input to click on the objects that you find but there are a few architectural decisions that can make it kind of difficult to know where to get started so in this video i'll walk you through one way to approach this that should be pretty flexible and act as a good foundation for building a more capable bot hey i'm ben and this is lincoln by gaming today we're wrapping up our exploration of opencv and how it can be used to build a video game bot if you're just now dropping in there's a link in the description to the full tutorial playlist the most straightforward way to add automated mouse clicks to our bot script is to do them inline by that i mean we can take a screenshot process it find any objects that we want and then perform any actions that we want about to take so we can do that sequentially and then just loop back around when our bot is done with its actions so let me show you what that would look like and then i'll point out some limitations with that method the code i'm starting off with today is the same as the last video about cascade classifiers except i've stripped out a lot of the stuff that we aren't using anymore since i'll just be using the cascade classifier to do my object detection i don't need any of the hsp filter candy edit detection or any of that stuff so you can still find all that code on github i just want to clean out our project a little bit before we start writing the bot code the library we'll be using to automate our mouse inputs is pi auto gui and to get that on your virtual environment is just a simple pip install payout of gui and if you find that payout gui isn't working with the video game you're trying to play check out pi direct input that's the library i wrote this solves some of the issues that priority gui has with directx games but it works the same as prior to gui does and you can install it just as simply with pip install pi direct input and i made a video about pi direct input so go check that out if you need to know more all right so here in our main script in our main loop we're of course grabbing a screenshot from our window capture and then we're using our cascade classifier to detect the objects in those screenshots and then right now we're just showing those results so this would be a good spot right after we show those results to take some actions with their bot so if we have some detection results let's go ahead and just try clicking on the very first one and of course we need to remember to import pi out of gui and then from this list of rectangles we need to find a place to click and we've already written the code to do that in our vision class we wrote git click points it will take this list of rectangles and return a list of midpoints to click on and then from that list of center points let's just grab the first one in that list and then we'll send it over to the window capture class and the get screen position function because this first position is the x y coordinates on the original image but that might not be the same as the position on our screen so remember we wrote this git screen position function on our window capture class to take care of that and now that we know where we want to click let's just use pi out of gui to click in that position so here i'm just calling power to gui that click and i'm giving it the x and the y coordinates of where i want to click you can also split up the mouse movement command and the mouse click command into two different parts so we can move to that coordinate first and then if we call click with no arguments it's just going to click wherever the mouse happens to be and then assuming that this position that we click on is a limestone deposit let's go ahead and wait five seconds after we click on it to allow our character to actually mine some of that stone and we're just using the sleep function here to wait five seconds and of course we'll need to import that from time so if we think through our code a little bit each time through this loop it's going to grab a screenshot and then it's going to use our cascade classifier to find all the objects we want to detect in that screenshot and then from those rectangles it's going to go ahead and draw the rectangles that are found on the original screenshot image and then we're going to display that as our debug output and then after that we're going to take our bot actions so if we found any objects we're going to go ahead and get the position on the screen where that object occurs and then we're just going to go ahead and move our mouse there and click on that point and then we're going to wait five seconds for our bot action to complete and so one of the problems we're going to have is that this debug output is only going to update once every five seconds so that's not great for seeing how our object detection is working but building your bot like this will still work fine so let's go ahead and try it out so if you give this a try you'll find that not only does the debug output update really slowly when we're in that state of sleeping for five seconds it really locks up our program entirely and of course if i've got my game window behind the opencv window my mouse won't actually be able to click on anything so i'm going to go ahead and move this up to my second monitor wow so i'm going to go ahead and move this up to my second monitor but the good news is is we can fix this problem by using threading and threading or multi-threading is a more advanced programming topic so i want to work up to it in steps and what multi-threading is is the ability that all modern computers have to run multiple processes in parallel at the same time so like i've got vs code open right now that can be considered one process the game that i have open that can be considered another process and when we run our script that's again a third process and right now our script is just a single thread meaning that when it's executed it starts with the top line and it works down through line by line and each piece of logic is executed as it's supposed to when it gets to it until finally it gets to the end of our script and this happens sequentially all is one continuous thread from the first line of code to the last with multithreading it allows us to run multiple threads at the same time so we can sort of branch out and do multiple things at once and we can have those threads come together or interact at a later time so in our case what it'd be great to do is we have our main script running in the main thread as it's going through getting those screenshots and detecting objects it'd be nice to branch off our bot actions into a separate thread so that it can go and do the things that needs to do without blocking the execution of our main thread so right here i'd like to just trigger these about actions and it can go and do that in a separate thread somewhere else and then once that thread is triggered it can just move down to the next line immediately and loop back around and get the next screenshot and do the next detection without waiting for that bot action to complete so to do that we can import thread from threading which is just part of the standard python library and then let's take our bot action code and let's move this to a function so i'm going to do this outside of the main loop and i'm just going to call this function bot actions and then our plan is to run this function inside of another thread and then inside our main loop where we want to take our bot actions is where we want to start this new thread so i've created a new thread object and what we need to do is we need to tell it what function we want it to run and we do that by using the target keyword and then we just give it the name of the function that we created and this syntax is a little bit weird this spot actions right here is literally just the name of what we called our function and notice there's no quotes around it even and bot actions takes one argument the rectangles so when we create our thread we also need to pass over that argument and to do that we use the args keyword and we have to send it over a tuple with all of the parameters and here we just have one the rectangles but we still put parentheses around it and still have this one comma here just so it is a tuple and now that we've created this new thread we just need to run it so to do that you just call start on the thread object and before you run this script there is one issue that we have to account for if you think through what our code is doing each time through this loop it's going to get a screenshot do the detection stuff do the debug show stuff and then it's going to trigger one of these threads and it's going to create that and start it and then it's going to quickly go back through the loop again so as we have it now it's going to spawn a lot of these threads 10 or 20 or so every second and that's not what we want what we want is to start another one of these threads only when the previous one has stopped so while one of these bot action calls are running we don't want to start another thread and to do that let's just create a global variable so it'll be accessible by both our main loop and then also our bot actions function and we'll just set this variable to false when our bot isn't running and then when it is going through the actions it'll set it to true so up at the top of our script i've created this isbot in action variable and i've set it to false and then down in our main loop let's only start one of these threads if isba in action is false if not if the bot is not in action let's go ahead and run this code and just before we start this thread is also a good place to set this is by in action to true because we know we're about to start it so if the bot actions aren't running we'll say it is running and we'll start that new thread and now in bot actions we just need some way to set that variable back to false when these actions have completed so to do that we can just set isbot in action to false after our actions have completed but we've got a scope problem here this is by an action variable right now is just local to the spot actions function and we actually want it to refer to this global isb inaction variable that we defined earlier and to do that we can just use the global keyword and this will let this function know that we want the global version of this isb inaction variable we don't want to create a local variable here and so now every time through our main loop if our bot is in action it won't execute this code but as soon as it's not in action once it reaches this place in our code it'll trigger a new thread and start our by actions again so let's run this and see what kind of improvements we get all right so immediately our debug output is looking a lot better and i can see up here on my second monitor that the script is moving around and clicking on these different optic detections every five seconds one thing you'll notice is that your script no longer immediately stops once you see done in your console if your bot thread is still running it's going to continue until it completes and only after all the threads in your program are done does your program actually exit so using a global variable like this to communicate between your threads and your main thread is fine for smaller scripts but as your bot gets more complex this will quickly become less than ideal and most teachers will tell you that using global variables like this is something that should be avoided unless you absolutely have to do it if you see some code with a lot of global variables in it it's probably not going to be very fun to work with so let me show you a better way to do this threading stuff without needing to use global variables so let's make one thread for getting the screenshots and another thread for doing the object detection and then we'll do our bot actions in a third thread so i guess we'll have four threads total we'll have the main thread the screenshot thread the detector thread and then the bot actions and let's actually start with the object detection we'll move this out of the main script and we'll put it into its own class all right so here i've created a new class called detection and in the constructor we're just going to pass it the name of the cascade classifier that we want to load and then we'll go ahead and create a threadlock object which i'll talk more about in a minute and then we'll load the cascade classifier model and then i've written a method here to start the thread and i've created a property on this class called stopped and when we start the thread we'll flip this over to false and then we'll create the thread and start it just like before except the function that we're going to run is called this run function i've got a simple method here called stopped which just flip stopped over to true and then the run method itself this is the function that's being run inside the thread it's just going to go on an infinite loop as long as we haven't stopped it and then as long as we have a screenshot to do the detection on we're going to go ahead and call detect multiscale on our cascade classifier just like before and we're going to get that list of rectangles and then we're going to save those rectangles to another property on this class and so when our bot thread wants another list of object detections so when it wants more of these rectangles it's going to go ahead and grab that from the detection object that we create and that can be a problem because remember these will both be running in separate threads and if that bot thread tries to get a list of rectangles while this thread is in the middle of updating those rectangles that's going to cause a huge conflict and it's going to break your program so to prevent those sort of issues from happening we're going to use that lock object that we created in the constructor and we're just going to call acquire on that lock so we're acquiring a lock on this thread and then as soon as we're done updating that list of rectangles we're going to go ahead and release that lock and we're doing the exact same thing in this update method that i wrote so our detection object is going to need to get an updated screenshot from our window capture and to do that we'll use this update method and while we're saving the screenshot to this object we'll make sure to lock it down while we're updating that screenshot and then releasing it when we're done and that's really all there is to this class we've got these properties at the top of course the stopped property and the lock property that we already talked about and then the rectangles that we're going to be outputting from this object that's going to be initialized to an empty list and then of course the cascade object that we initialized in the constructor and the screenshot that we're going to be getting from window capture are also properties on this class all right so now that you've seen how this class is written let me show you how to use it so in the main file we need to import it and then we need to initialize it and of course we won't need to load the cascade classifier like this anymore instead we'll create an object that i'm calling detector using the detection class and we'll give it that file name of the classifier model we're using and then just somewhere before the main loop we can go ahead and start that thread so that'll just be detector.start and that'll call our start method which will create a thread that's running this run function and then inside the main loop we don't have this cascade object anymore but what we do need to do is we need to give our detector the updated screenshot so that was detector was the name of that object and we wrote that update method and it just needs the screenshot so every time through this main loop we'll refresh the detector with the updated screenshot and then the rectangles that represent the objects we detected we no longer have this rectangles variable instead this is a property on the detector object so that'll be the detector.rectangles and then finally when we stop our script we also need to stop that detector thread and so to do that we'll just call stop on detector that'll call this stop method that we wrote and we'll just set stop equal true and so the main run method that's being run inside that thread and it'll hit this while statement and it'll see that stopped is now true so it'll stop this infinite loop which then ends the thread because there's nothing left to do inside of the run method and i missed this other spot down here where we use rectangles that needs to be updated to be detected at rectangles and when you run it you see that it works just as well as before except now all of this detection stuff is going on inside of a separate thread so now let's do this for our window capture class that creates our screenshots all right so the changes to the window capture class are all very similar to what we did in the detection class i've imported thread and lock from threading and i've created these stopped and locked properties on the class and also the screenshot that it outputs will now be a property on the class and in the constructor we initialize that lock object and all this other stuff is the same as before except at the bottom i've created these threading methods so it has start stop and run just like the detector class has and in the run method all it's going to do is while it's going through this infinite loop until it's stopped it's going to call get screenshot on itself to get that screenshot and then it's going to update that property and of course we're locking the thread whenever we're updating that screenshot property so now let's update our main script to start that thread so initializing the window capture is going to be exactly the same as before but now we do need to call the start method on the window capture object in order to start that thread and then inside the main loop if we don't have a screenshot yet from our widow capture then the rest of the code below here is kind of pointless and i was running into some issues if we didn't have a screenshot yet at this point in the code so if we don't have a screenshot yet i just call continue which will bring us back around to the start of the loop and this will just kind of keep us in a holding pattern until we do have a screenshot and then we don't need to call getscreenshot here anymore remember that's going to be happening now inside of the run method of the window capture class and so this screenshot is actually a property on the window capture object so whenever we want the latest screenshot we can just call wincap.screenshot so we'll pass that over to the detector and we'll also use that to draw the rectangles and by the way i tried doing this draw rectangles and i am show stuff in a separate thread as well but it didn't really work for me i just got like a great image in the i am show window whenever i tried that so maybe it'll work for you but i was fine with just leaving this debug stuff in the main loop anyway but maybe i'll just add a flag here so we can turn this part of the code off when we don't need it i need some vocal training oh yeah don't take your shirt off on camera so if we don't need this debug output maybe we're running our bot and we're stepping away for a while we just go ahead and turn it off with this debug flag and up at the top of our code i'll just set this to true for now and if you run this you should still see that it works exactly the same as before but you might actually notice that the performance is a little bit better because remember now we're doing that screenshot and we're doing the object detection in separate threads from our main thread so you might find that these results appear a little faster and a little smoother now all right now let's actually write some bot code and mine some limestone deposits and i'm going to write a byte class that works just like the detection and widow capture classes so i'm going to go ahead and tear out all this test bot code that we have and for this one why don't we work backwards so i'll outline it in the main script what we want our bot class to do and then after that we'll go back and we'll write our bot class so i'll create a file called bot.py and in there i'll create a class called alpinebot and then we'll need to initialize the bot object and just like with the other threaded classes we'll need to call start on that bot thread and then inside the main loop we know that we're waiting until we get a screenshot and then once we do the detector will get the updated screenshots and then we're doing that debug output to show where our matches are and of course down here when we stop our script we also need to stop our bot thread and then somewhere in this loop we'll need to update our bot with the latest screenshots and the rectangle detections and you could write something like this where you have an update method on your bot object and you just pass over the screenshot and those rectangles but remember that the vision class has this git click points method on it for converting those rectangles into places to click on the screen so maybe we call that first because we have easy access to the vision object right here and then instead of passing over the rectangles we can pass over those targets and so something like this would work but when i see this code what i'm thinking is you know every time through this while loop it's going to be giving the bot an updated screenshot and that target list but depending on what our bot is doing you know it might not need that data right now and maybe we could get some performance increases if we only pass that data over when our bot actually needs it so maybe something more like this would be better maybe our bot has different states that it's in so when the bot is starting up it's got this initializing state and then when it's searching for the next limestone to click on it'll be in this searching state and then after we click on the limestone but before we get to it we'll be in the moving state and then while we're at the limestone we'll be in the mining state and so depending on what state our bot is in we only need some of this data so while it's searching for the next limestone deposit it needs both the list of targets and the current screenshot and i want to give it access to the current screenshot because i've got this idea where if i move the mouse over potential detection one of the potential limestone deposits albion will show these little tooltip messages when you hover over something my plan is to use that to determine whether we have an accurate detection or not and so to verify that the bot will need access to the screenshot and then also wallorbot is in the process of moving i plan on sending over those screenshots as well and we can compare the current screenshot with the previous screenshot to see if we actually stopped moving or not but while we're mining we don't need any information we're just waiting for the about to finish and while our bot is initializing you know when it's starting up it doesn't really need any information then either but i did find that it worked slightly faster if i gave it that list of targets to work on right away and of course these bot state values need to come from somewhere and i'm just going to create another class inside the bot file that defines those values all right so here's what the beginning of our bot script looks like we're importing pi to gui and sleep here and we no longer need those over in the main file and here's that bot state class and this is a data structure known as an enum and newer versions of python have the ability to create an actual enum but in the past we had to just create a class like this and then give it some properties and set some values on those properties so here i'm showing that older way of doing it just in case you are using an older version of python but just know that in other programming languages a data structure like this is known as an enum and then here's our bot class i've got some constants here for how long we want to wait when our bot first starts up and then also how long we want to wait while we're mining a limestone deposit i've got the stopped and the lock properties just like before the bot has a property to keep track of its current state the targets and the screenshot will be data that's incoming from the main loop and then i've got a timestamp property here that we can use to measure elapsed time so in the constructor we'll go ahead and create the lock object we'll set the state to initializing and then for the time stamp we'll set that equal to the current time for the threading methods we've got two methods here to update the targets and the screenshots separately and then the start and the stop methods look just like the other classes i showed you and so the run method is the main controller for our bot we've got this loop that'll run as long as our bot isn't stopped and if it's in the initializing state it'll go ahead and look at the current time and compare that to the timestamp that we saved when we called the constructor and if that's greater than six seconds we'll go ahead and switch over into the next state which is the searching state and remember because our main loop is interested in the state of the bot we gotta go ahead and acquire a lock and then release it every time that we change this value and then once we're in this searching state we'll go ahead and click the next target and this is a method that i still need to write so i'll do that next and if we were able to successfully find a target and click on it we'll go ahead and change to the moving state otherwise we'll just stay in the searching state and we'll go back around and we'll use a new list of targets and try to find it again once we are in the moving state we'll have some sort of method for figuring out you know if we're moving still or not and so this have stopped moving function is another method that i need to write and once it's detected that we have stopped moving we'll go ahead and move over into the mining state and we'll update our timestamp when we do that because once we're in the mining state we want to wait for that 14 seconds and then once that 14 seconds is over we'll go ahead and switch back to the searching state so now let's go ahead and write those two functions the click next target and then also have stopped moving let me show you the half stopped moving function first because it's a little simpler so first i added this property called movement screenshot which is just the previous screenshot and then the function itself you can see that if we don't have that previous screenshot saved yet it will go ahead and just save that from the current screenshot and by returning false here down in the run function you can see that if this function returns false it's going to go ahead and wait for half of a second before it goes back through this loop and it calls this half stop moving function again so the second time that have stopped moving is called we now have that previous screenshot saved and we've waited half of a second so we have the previous screenshot and then also an updated screenshot now we're just going to compare those two screenshots using our old favorite match template so remember that match template is just going to take the needle image that you give it and it's going to overlay it on top of the haystack image and then it's going to give you some value for how closely the pixels match in both images so here i've given it the current screenshot laid on top of the previous screenshot and because those images are exactly the same size we're only going to get a single result returned from match template so i just save that here to the similarity variable and if that value is over some threshold we're going to go ahead and say that those pictures are pretty similar and if the images are really similar then we've probably stopped moving and of course i need to define this constant here somewhere so i'll go ahead and do that up at the constants section of our class definition and i tried this out earlier and i found that this 0.975 value works pretty well you know the two images might not always match exactly for example if another character is running by the screen next to you the two images will be different even though your character isn't moving anymore and then to wrap up this function if it looks like we're still moving we'll go ahead and update that previous screenshot with the current one and then we'll return false to let the run method know that we need to call this function again next time through the loop can you focus yeah pro level now i'm ready to write this click next target function here's what i want the logic for this function to look like so we've got a list of targets from the detector and i want to sort that list to find which target is actually closest to my character that way i can start by checking objects that are really close to me rather than clicking on the edge of the screen somewhere and so once i have that list sorted i'm going to loop over it and i want to hover over the nearest target that i found and then once my mouse is hovered over that position i want to go ahead and check that tooltip that pops up and if we do confirm that it's a limestone deposit we'll go ahead and exit this loop but if it's not we'll go and continue through the loop and check the next object that was detected so once we've gone through that list and we've hopefully found an object to click on then we want to actually click on that found target and return true but if we didn't find anything we'll go ahead and return false here that way our run method will know to stay in searching mode and it'll call this method again so here's the code i came up with to do that we're going to need some sort of targets ordered by distance method in order to sort that list and then once we have that list sorted we'll go ahead and loop through it and if at any time through this loop our script has asked us to stop we'll go ahead and break out here because i found that this is a place where this thread can get stuck for a while and so by adding this breakpoint here it'll just help our program stop sooner when we ask it to now we want to take the next target in our list and go ahead and convert that coordinate into an actual position on our screen and we've written this method previously on our window capture class and i debated just passing over that object to this thread as well but i thought that was a little messy i kind of want to keep these threads and these classes completely separate so i'm just going to go ahead and duplicate this method here on our bot class so once we have that screen positioned we can go ahead and move our mouse there and then we want to wait a second or so to give time for that tooltip to appear and then we'll write a method for confirming that tooltip whether it's there or not that says limestone and if it is we'll go ahead and click on it and we'll return true otherwise we'll go back through the loop and we'll grab the next target and test that one all right so now we've got more methods to write so let's start with that git screen position method that we're just copying over from the window capture class we're going to need the window offsets here so let's go ahead and pass that over back in the constructor so when we initialize our bot object we'll go ahead and give it those offsets from the window capture and then i've created a property for it and we'll go ahead and add that parameter to the constructor and then we'll save it in the constructor next let's write this sorting function so for this one first we need the character position and then we need to figure out how far away each object's position is from our character's position and we know in albion our character is always in the middle of the screen so we can just divide the window height and width by two in order to get that center position and then we can use the pythagorean distance formula in order to figure out how far away our character's position is from some of those object positions so you've got the object x position here and you subtract it by the character x position then you square it and then you do the same for the y you've got the object y position your character's y position you square that you add those together you take the square root and that'll give you a value for how many pixels there are between those two positions so with this little function here we can use that with the sort method pass that in as the key that we want to use and this will sort our targets list by the targets that have the shortest distance between it and our position at the top and the ones that are farthest away will appear at the bottom of the list and these window width and height those are also new parameters that we can get from the window capture class so in main when we initialize this bot we pass over not only the offsets but also the width and the height of the window and then just like with the offset we've got properties for those and then we'll save them in the constructor and before we move on from this targets order by distance function when i was testing the spot i found that i really like to click on the same limestone deposits that it just harvested which makes sense because those limestone deposits that it's practically standing on top of are going to be nearest to our character so to help solve for that i decided it'd be nice to have a little exclusion zone around our character where we didn't look at objects detected right on top of us and i decided to do that right here in this function any targets that are within 130 pixels of our character i'm just going to go ahead and remove those from the targets list and so that's what i'm doing right here with this list comprehension and this ignore radius is another new constant i've got that defined up top with the other constants and i've just set that to 130. so something like this may or may not apply to the bot that you're building but in my case it made my bot a little bit better and for confirm tooltip i'm just going to use match template again i'm going to use this needle image that i cut out from that tooltip and if i can find this anywhere on the screenshot i'll go ahead and say that we're hovering over a limestone so that tooltip image is just going to be another property in our class and in the constructor that's where i'll go ahead and load up that image so i'm just using opencv i am read and then the confirm tooltip function itself is pretty straightforward if you've seen the other videos in this series we're just calling match template using the updated screenshot as the haystack and then that tooltip image as the needle and then we're using min max location to find the best match and if the similarity value of that best match is above a certain threshold we'll go ahead and say that we've seen it so if we've seen that tooltip in the screenshot will return true otherwise we'll just return false we'll say that that tooltip doesn't appear on the screen and for the threshold amount i found that 0.76 worked pretty well for me so if i was writing this code in real time i definitely want to run lots of tests as i go through this you know it's not a good idea to write a 200 something line class without testing it once in a while but i've written all this previously and i'm just kind of going over how it works with you so with that in mind there's a couple other improvements that i ended up making wow why don't why did i do it like that maybe i should just scrap that whole section all right got live coding here yes we check false no so one issue i found is that there was no limestone deposits on the screen my bot would just get stuck and it wouldn't move anywhere so i had this idea of if there's no limestone deposits on the screen you can go ahead and backtrack and move back to where it was when it started so i'm always going to start my bot in a location where there is some limestone on the screen and then it's going to go about its merry way and click on limestone deposits whenever you can find one and then if it gets itself in a position where it can't click on anything it'll go ahead and reverse all those mouse clicks and it'll just go back to where it came from so to do that i'm here in the run function and then when it's in the searching state i'll go ahead and call that click next target still and if it wasn't successful i decided to go ahead and call that a second time you know that targets list that we get from our detector it's always kind of in flux and there might be a good limestone on the screen but maybe when you grab those rectangles it just happened to be one of those split seconds where it wasn't detected so by running this through twice if we didn't find it on the first try that helps to correct for some of those situations so we're not backtracking where we don't need to so if we did click on an object we detected we'll go ahead and change it to moving just like we did before otherwise if we perform the click in the pass we'll go ahead and call clickback track to kind of undo that click and then we'll switch over into this new backtracking state so we need to do a few things here we need to create this backtracking state we need to populate this click history whenever we click on a deposit and then we need to write this click backtrack function so adding that new state is simple enough same with creating that click history property as an empty list and then in click next target down where we perform the click that's a good place to save that target position to the click history and i decided to save the position on the screenshot itself rather than the screen position just because that math is a little bit easier to get that position flipped around and then here's how i wrote that click backtrack function so i'm treating this click history list as a stack and the stack is the data structure you can think of like you're stacking a bunch of books together on a table and the book that's on top of the stack is the first one you're going to take off of the stack so the last item that you added to the list will be the first one to come off the list so this data structure is perfect for things like if you're building an app and you want to keep track of what screens the user has clicked on so that they can reverse those clicks they can backtrack through your app or like in this case we're keeping track of all of the clicks that our bot has done and in order to undo those clicks we got to start with the most recent click first so when you're treating a list like a stack you'll use append to add new items to the end of the list and then you'll use this pop method to remove items off of the end of the list so pop does two things here it removes the most recent item from the click history list and then it returns that item and saves it in this last click variable that we created and then once we have that previous click we can't just click on that same spot again we actually need to do the opposite of that click we need to perform the click that will undo that click so if you think about a video game window that's 200 pixels by 200 pixels and your character is right in the middle at the 100 by 100 position if you clicked at the 120 by 120 position so that would be further to the right and then further down in order to undo that click you actually need to click on the position that's to the left and then up from your character so that would be at the 80 by 80 pixel location because our first click was 20 pixels by 20 pixels away from our center to the lower right and so in order to undo that you need to click 20 pixels by 20 pixels to the upper left hopefully that all makes sense but that's what this code here is doing again our character position is at the center of the screen and then we calculate that mute click location and then once we have that position on the screenshot that we want to click we've got to convert that over into a screen position and so of course we've got a function to do that and then once we have that position we can go ahead and call move two on piyo to gui and then click on that location alright so that was an epic coding session i don't have any more code to show you today let's go ahead and run this spot and see how it does error number one forgot to import square root all right so when i run my bot i need the game in the foreground and i want to show that to you guys so i've got to hide this uh matches debug screen but you can see here at this location the object detection is working pretty well and there we've successfully clicked on our first limestone and so once we've clicked on one and we're mining it's going to wait that 14 seconds and then once that's done it's going to go back into searching mode so it'll move the mouse all over the screen checking different positions to see if we can get that limestone tooltip there we go i found a second limestone you notice i did hover over a couple limestones and uh it didn't detect it there so i wonder if i need to adjust my threshold a little bit on the match template or maybe i'm just not waiting long enough for the tooltip to pop up and of course the searching here isn't the fastest because it's getting a lot of false positives and the best way to correct that would be to train our model better so our cascade costs 5 it's actually doing the object detection the better you can get that the quicker your bot's going to operate all right here i've moved to an area with fewer limestone deposits because i want to see if the backtracking works so now that we've mined that deposit it shouldn't detect it again because it's within that 130 pixels around our character that it won't check again and after it goes through this loop twice of checking all the different positions there we go did that the back tracking click for us and now that we're not so close to that limestone deposit it should pick it up again and click on it again alright so we found a little bug there once it's in the back tracking state we don't have any way currently to switch back to the searching state so here in the bot run function i've modified this moving controller to also work for the backtracking state so again it'll call this have you stopped moving function and then once it has if the bot is moving it'll go ahead and switch it over to the mining state but if we were just backtracking we'll go ahead and switch it back over to the searching state let's give this another try all right so we're mining the limestone now the mouse is moving around so we're back in the searching state and then there's our back tracking click and then the mouse is moving again so we must be back in the searching state and there we go we were able to pick up that limestone again man i don't even know how to uh unload my inventory boom not overloaded anymore there are obviously a ton of things we could do to make the spot better but this should be enough to get you started on building your own bot from here it's a process of testing finding issues and correcting them and just doing that over and over again until you've got a refined bot that you're happy with if you've been following along and writing code now's the time to set out on your own path and just explore and have fun make what you want and make it your own just keep building keep coding and you'll be learning and getting better and if you've watched this far and still haven't started writing code yet man now's the time we only get better through practice and there's no way around it you've got to put in the hours for yourself if you want to be a programmer and a project like this is the perfect way to build that experience so if you're having any troubles and especially if you're having troubles just getting started let me know in the comments or shoot me an email and i'll do my best to help alright that's all i got take care and thanks for watching you

Channel: Learn Code By Gaming

Views: 65,380

Keywords: opencv, python, bot opencv, opencv bot game, opencv game, opencv game control, pyautogui, pydirectinput, pyautogui image recognition, pyautogui game bot, python multithreading, python threading, python threading tutorial for beginners, python thread, mouse click automatically, mouse click automation, python threads shared variable, computer vision projects, python automation projects, how to write a bot in python, python game bot, opencv full tutorial, opencv projects

Id: FylRUEEWhCo

Length: 41min 32sec (2492 seconds)

Published: Sat Sep 12 2020