How to make advanced image recognition bots using python

hey so today i'll teach you how to make some bots we'll be covering how to make a simple piano telespot how to make an aim booster bot that can track an object okay so let's get started step one download python and remember to check the add to path checkbox when installing step two open the console with administrator privileges and run the following commands you can find them in the video description these are all python libraries that we are going to use throughout the tutorial i'll explain what they do a bit more into the video step three now that we have completed this setup create a new folder and create a text file named bot.py you can name it to whatever you want as long as it ends with the dot py extension now begins the interesting part of the tutorial right click your newly created python file edit with idle edit with idle and paste this into the code editor you can find this in the description these are the libraries we downloaded a second ago let's find piano tiles online this one should do for this game we have four rows where black squares will come down from each row the fastest and easiest way to about this game is to check a single pixel from each row we will check if that pixel is black and if it is we're gonna click it okay so first we need to know the rgb value of black which is the tile's color for this open a new python shell by pressing windows search python and open write import pi auto gui and then pi auto dot gui dot display mouse position the shell will display the pixel you're pointing at and its rgb value hover over any black part of the screen you can see the color for black is zero red zero green and zero blue go back to the shell and press ctrl c to stop it from running we now want to figure out the x ak horizontal position for each row open the python shell again and type import pioto gui and then pay auto guide on display most position hover over the first second third and fourth row go back to the shell and copy the data from each respective row we will only be using one y vertical position so i'll put mine to around 400 which is around the middle of the game for me the first house pixel is x 597 second x 676 and so on it will be different for you but don't worry open the pod.pi file there are two ways to perform a click the first is by using pyota go i click function which looks like this the the downside for this is that it's pretty slow so instead we will use the win32 api which runs much faster to do this we start the script by creating a click function like so this uses the win32 api library to move the mouse to xy position with set cursor pause pressing down holding for 0.01 seconds and releasing we want the 0.01 second delay since clicks sometimes fail to execute if you click too fast we then write while keyboard is press q equal equal false this line checks if the key queue is pressed down it returns 2 if it spreads down and false if it's not we're going to use this to stop the bot when we want to stop it okay so now what we'll do is check the rgb value of the first rows pixel to do this we use the if pi auto gui dot pixel and then your pixel's location 0 equal equals 0 this will return the rgb value of the pixel located in x 597 and y 400 the zero checks for the r value whereas if you were to switch the zero for a one it would check for the g value and if you were to switch it for two it would check the b value there's no need to check for more than one in our case so the r value is enough now that the script can tell when the pixel is black we want to click it immediately when it does for this we call the click function we created and make it click the first tile's position which is the same position we scanned we do this by writing click and then your pixel's position alright so repeat this process for the remaining three tiles and don't forget to add the correct pixel location and there you go press run run module switch back to piano tiles and press start so as you can see yay it works [Music] [Applause] [Music] [Applause] now that you know how to make a piano tile spot let's move on to the aim booster bot start by going to aimbooster.com enable flash player if it's not enabled and press reload in this game you have to press the circles that appear randomly in this area of the screen there are two approaches to solving this game the first is to take a picture of the circle and locating it on the screen this approach would be too slow so instead we will take a screenshot of the game area and check every one out of five pixels to see if that pixel is the color of the center of the target before doing this however i'll demonstrate how to do image recognition okay so through my amazing artistic talent i have made what appears to be a stickman and paint what i'll teach you now is how to detect if the stickman is on the screen or not go back to the folder you created with the bot.python file create a new file and name it stickman.py edit the file without it like we did before and paste in the same libraries as in the piano tile spot go back to the stick menu drew in my case this one on windows press shift windows s this will open up a screenshot tool that allows you to select a part of the screen to screenshot drag over the stick manager to save the screenshot open a new paint file and paste the stickman you now want to trim it as small as possible by selecting the area that contains the stickman and cutting excess like so once you have cut the excess press file save as and save it as stickman.png in the same folder as the script saving it in the same location as the script is important open the stickmanscript you just created we will now create a while loop with while one inside this loop will write the script that is actually going to find the image if piotrgy.locateonscreen stickman.png is not equal none this line uses pi outer gui to locate the stickman on the screen if it doesn't find anything the output of this line will be none so if it's not none it means it found it so if you write the following script if it finds the image on the screen it will print out i can see it and then wait for 0.5 seconds the delay so that the script doesn't spam the console and if it doesn't find the image it writes i am unable to see it run the script by pressing run run module and as you can see the script can see the image but if you move the window over the stickman it stops working now one of the questions you might have is what if the background of the image i am trying to detect changes for example if i were to put this disturbance here the script wouldn't detect the stickman since it's not an identical match to fix this we simply add the confidence parameter and the locate on screen like this let me explain when the bot looks at the screen it checks every pixel and gives a percentage chance probability of finding a match to the image you're asking to search for by default pyro gui only gives an output when it's 100 sure of having a match but by using confidence 0.8 we tell it that if it's 80 sure or more to have found the image it finds it you can of course adjust the value to whatever you need but wait there's more now you might be asking yourself how can i make the script run faster the first and most obvious way is by checking a smaller region of the screen so instead of searching for the image on the entire screen we have to tell it where to look finding the correct region is a bit of a process but i'll explain as best as i can create a new python script in the folder copy the path to the folder by clicking here and copying this open the script and write the following code it's in the description as well but remember to change this path to your folder's path you copied a second ago this script saves the image on the region 0 0 100 100 as savedimage.png the first number is the x position that the screenshot starts on the second number is the y position that the screenshot starts on the third number is the width and the fourth is the height so let's say to region 100 200 500 500 the script will take a screenshot starting 100 pixels from the right 200 pixels down and have a width and height of 500 pixels now we just need to tweak the values manually run the script look at the output until we find the region of the stickman so as you can see running the script saves an image with the region i gave it i found that the region for the stickman in my case is around 150 175 350 and 600 keep in mind it doesn't have to be too close to the object it's fine to check the area around the stickman as well open the stickman script again and write region equal u region and then grayscale equals true the grayscale parameter gives a speed boost of around 30 so it's always a nice thing to have do not use grayscale if you're getting fake results but in our case it's fine run the newly modified script and as you can see it can detect the stickman even though parts of it are hidden and or modified alright so back to the aimboosterbot create a new python file in the same folder as all the other files we made name it aimbooster.py paste in the libraries we used in the two last examples and then rewrite the same click function we used in piano tiles remember to add a time.sleep2 to give you two seconds to switch over to aimbooster when starting the script so as i explained a bit earlier we'll be taking a screenshot of this region of the screen where the circles appear and then check every one out of five pixels to see if the color of that pixel matches the color of the center of the target so first we want to figure out what the region for the game area is to do this open the script you created during the stickman demo that takes a screenshot of a region and saves it and tweak the region values until you get this region which is the region where the circles are going to be appearing now that you know the game region let's start the script with the while keyboard is pressed equally glue false so that we can stop the bot from moving the mouse when we want it to stop write the following line this line uses payout guy screenshot function to take a screenshot of the screen and store it in the pick variable in our case it takes a screenshot of the region we give it which for me is 660 350 600 400. your region will be different so don't mind it if it's not exactly the same we then use the size function to get the width and height of the pick variable like so this will store the width and the variable width and height in the variable height pretty self-explanatory now we want to check what the rgb value of the center of the target is for this i took a screenshot of the game while playing pasted the picture in microsoft paint open the python shell wrote import pi out of gui and pi auto gy the display mouse position i then hover over the center of the target and the console will print out the rgb value of the pixel i am pointing to the center of the target has an rgb value of 255 219 and 195. copy the rgb values go back to the script and paste it anywhere as a comment note to make a comment in python use the hashtag symbol ok so since checking every single pixel in our screenshot for a match is a bit overkill we will instead check for the first pixel then the fifth then the tenth etc etc but first let me explain how a for loop works let's take this example for x in range 0 10 1 print x this will output every single number starting from 0 up to 9. this means that the script will set x to 0 print out x so 0 and then increase x by 1 which is the third value here it will do this while x is smaller than 10. the maximum value discount output is 9. if we were to run 4x in range 0 15 1 print x the script will print out every number from 0 to 14. if we were to increase the third value here to 2 for x in range 0 15 to print x we will instead get increments of 2 instead of 1. so the output would be 0 2 4 6 8 10 12 14. so as you might have already guessed we will be using increments of 5 for our image so for example let's say that our image has a width of 500 pixels and a height of 600 pixels we write for x in range 0 500 5 for y in range 0 605 rgb equal pick dot get pixel x y the script will save the rgb values of the pixel and position x y of our picture pick thanks to the get pixel function so for example the script will first check the pixel x0 y0 then y 5 then x 0 y 10 then x 0 y 15 etc etc until it hits the maximum value of y 595 since we have increments of 5 and y has to be smaller than 600 and store the red value in r green value in g and a blue value in b alright so your script should now look like this since my game has a region width of 600 and a height of 400 i will use the width variable here and the height variable here instead of hard-coded numbers you can use hard-coded numbers if you want but for simplicity i will use the variables so now that we can successfully check the color of every fifth pixel we want to perform it click if the color matches the center of the target's color you can code this in two different ways in my case the center of the color of the target is always going to be the same so we can add a if b which is the blue value equal equal 195. this checks if the blue value of the pixel located in x y is equal equal to the value 195 which is the blue value of the center of our target to check for the red and green values change it to either r for red and g for green but in this case blue is more than enough but if you wanted to create a bot similar to this one where the center is not always going to be the same color then you can use range again if you were to use range the loop would look like this if b in range 180 210 this will check if the blue value stored in b is between 180 and 210. in my case checking for a range is unnecessary so i will just stick for if b equal equal 195. okay so now that the script can tell when the color of the pixel is the color we want it to be we want to click for this simply call the click x plus 660 y plus 350. if you wonder why i have these values remember that the click function will click a pixel based on your screen resolution so a click 0 0 will click at the top left of the screen but in our case we want to perform a click relative to the screenshot's region since our region's first pixel aka pixel 0 0 of our screenshot starts at 660 pixels to the right and 350 pixels down we want the click function to also be 660 pixels right and 350 pixels down since the script runs a bit too fast it might click the circles a bit too fast so we added time dot sleep after the click to prevent miss misclicks i'll put mine to 0.05 we then end the script with a break to stop the script from clicking the same place again save the script run the script tab over to aim booster and press challenge and voila there you have it a working aim booster bot remember that you have to hold the cue to stop the bot if you experience accuracy issues just slightly tweak the delay after the click so the bot doesn't have the time to click twice so before ending the video i just wanted to ask if the fast pace of this video was okay or if it was way too hard to follow and if you want a more detailed in-depth video about botting alright so i hope you enjoyed this fast paced tutorial if it helped you out please consider subscribing and leaving like if you have any tips on how i can improve the video do leave them in the video description have a great day

Channel: Kian Brose

Views: 844,411

Keywords: How to, tutorial, bot, bots, python, image, recognition, image recognition, pyautogui, 15, minutes, smart image recognition, kian, brose, how to make, smart bots, in 15 minutes, short, botting tutorial, smart botting, bot tutorial, python bot tutorial, python botting tutorial, python image recognition, python smart bot, python bot, how to make python bot, how to make a python bot, python bot tutorial fast, how to make smart bot with python, machine learning, computer vision tutorial

Id: YRAIUA-Oc1Y

Length: 15min 1sec (901 seconds)

Published: Wed Mar 11 2020