Control your Mouse with PyAutoGui using Python 3.10 Tutorial (Windows / Mac OS)

how's it going guys in today's lesson we're going to be going over how we can use pi auto gui to create some automated tasks on our computers and the first program we're going to be creating today is going to sort our files on our desktop so it's going to grab them and it's going to drag them from any location into the folder that we choose which is going to be my audio files folder and it's going to grab all of the files that we've specified so in this example they're all going to be audio files and yeah that's actually all we're going to be doing in this lesson and as you can see it's quite effective and quite simple but the concepts you're going to learn in this lesson will be transferable to any program you want to create so the very first thing you want to do as usual is go ahead and create a new empty project in pycharm and we want to go ahead and open up the terminal because we want to install pi auto gui and opencv python so you want to install these two packages and then close the terminal now the very first thing we're going to get out of the way is the snapshots because we need to be able to recognize our master folder and these files over here so that we can click on them and drag them so go ahead and just take snapshots of these two i'm going to take a snapshot of the audio files like this and i'm also going to take a snapshot of this music playing symbol over here and i will rename the audio file to target folder here i'll just type in music image then with that being done we can go ahead and drag them inside our project so go ahead and drag your snapshots into your main project file so that they are next to your main script and then we can maximize this program once again and go to our main file now inside here we're going to go ahead and import pi outer gui as pag and we do not need to import opencv to make this work it's going to work under the hood but the very first thing i want to show you is that we can provide a fail safe and we're going to set this to true and what failsafe does is that if your program is doing something it's moving the mouse around at any point of the program you can move your mouse to this top left hand corner and it's going to end the program it's going to throw an error and end the program so this is important in case your program starts opening links and logging into your facebook and messaging your friends with unwanted messages you might want to be able to step back from that so it's going to give you a way to exit out of it and there's another one you should be aware of and this is pag dot pause and this is just a timer between each function call so if you put one second it's going to automate the mouse and then it's going to pause for one second until the next function call so this just gives it a big delay in case it's doing something you don't want it to do you can stop it ahead of time we're not going to be using this in this example but it's good to know that you have these options now let's go ahead and create a function that can get the position of any image so here we'll call it def get position and it's going to take an image of type string and here we're going to create a try and accept block because first we want to try and see if we can get the position so it's going to be pag dot locate center on screen and we want to insert the image here and we also want to give it a confidence of 80 so if the image looks 80 as the one that we've provided it's going to recognize it as the image we're looking for otherwise it's not going to find anything and this confidence parameter is only available if you have opencv installed so that's why we installed it with pip and this program actually works without this as well it's just in general if you don't have a pixel perfect image it nearly never recognizes the image you give it so confidence at 80 or 90 just makes your program a bit more fluent and helps with ignoring those very small imperfections now below that we want to check if the position is none then we're going to go ahead and print a formatted string that says that the image is not found on the screen and we're going to return none else it means there is a position and we have found the image so we're going to go ahead and grab those positions so the x is going to be the position at the index of zero and this is also another thing i need to clarify because i'm running this on a retina display on a macbook and retina displays have about double the pixels so you're going to get some really funky behavior the positions are not going to look the same as on windows and for mac users if you do have a retina display you need to divide all of these positions by two but if you're on windows you don't have to do this unless you have a retina display but this is just something that happens on mac and that's why i'm dividing by two over here and i need to do the same thing for the position at the index of one which is going to be the y-coordinate and with that being done we can return both x and y and we still need to take care of the accept block which is going to be an os error and we're going to accept it as e so that we can just raise an exception which is going to be e and you can enter whatever kind of code you want in here i just want to make sure that it stops the code if something's wrong and in general you're going to get this error if you insert an image name that it cannot find in your folder so it's definitely a logic error and if you can't find the image in your folder then your program won't work no matter what you do so it's good that it raises this error but let's go ahead and test this out so if we go ahead and create let's say a main check so if name is equal to main and we're going to get the position for one of the images that we've provided and we'll just pick the target folder so here we'll type in target folder dot png and we can close this then i need to open my screen a bit so we can actually locate this over here and the final modification i need to do is type in position here because we want to get the coordinates and print the position so if we go ahead and run this it's going to return these coordinates of where this folder is and one thing you can do that's really cool right now is go ahead and type in png move to and we can just insert that position and let's also give that a duration of 0.5 and if we run this program one more time it's going to move your mouse pointer directly to the audio files so so far so good we've got everything working as we wanted to so let's go ahead and close the terminal and open this window once again and just remove all of this because the next function we need to create is the drag to function so def drag to target and it's going to take an image of type string followed by a target of type string and a speed of type floats and the position is going to be equal to get position of the image that we will pass in and the target will also be equal to get position of the target image now we want to check that both of these are not none so if we go ahead and type in if none not in position and target so if none of these are none then we can go ahead and continue with the program so pag dot move to and we want to move it to the position with a duration of speed and then we want to call p8g dot drag to the target position and the button has to be specified or this will not work we're just going to use the left button with a duration of the speed so as you noticed earlier this performs a move to action and this performs a drag action and if you go ahead and type in phd you'll notice that there's a lot of these actions you can perform such as click and double click and triple click and so on there's a lot of these actions you can play around with but here we need to go ahead and create an else statement which will raise an exception and if it returns none we want the program to stop completely so i'm just throwing an exception here that says could not complete the request because one of the images returned none and this just means there's no more of the images that we're looking for on the screen so that's how we're going to exit out of the program next we're going to go ahead and create a main function just to throw in all of the code we have and we're going to call drag to target so now we need to fill out these fields and if we open our project over here you can see that the first one we need to fill out is the music underscore image dot png followed by the target image which is going to be the target underscore folder dot png and the speed will be set to 0.5 so let's go ahead and test that this actually works so let's go ahead and create our main check and inside here we're going to go ahead and just call main now go ahead and close this slightly because we need to test it out by grabbing one of these and putting it here so if everything works perfectly it's going to call this function which is going to grab this over here and drag it over here so let's go ahead and try it and click on run main and perfect it dragged it directly to the folder and you might be wondering so how do we do that with many files and first we need to arrange the files in some random order we'll just put it in a circle this time and to actually make it work in a circle all i did is go ahead and insert a while true loop which i recommend you never do when it comes to using automated controls because this can lead to something infinite that makes it really hard to control your mouse and if it does something wrong there's not much you can do to recover from it other than turning off your computer or hoping that you can go to this left-hand corner and stop the program but as you can see all i did is include a while true here and it's going to continue looping through this until it raises an exception and the exception will be raised when it finds no more of these music files but if we go ahead and run this program one more time it's going to do its best to drag all of the folders or all the files to the folder and as you can see it's doing a wonderful job and yeah that's actually all i wanted to cover in today's lesson we had a very simple introduction to how to use pi auto gui and how to recognize images so with that being said as always guys thanks for watching and i'll see you in the next lesson

Channel: Indently

Views: 8,987

Keywords: code palace, cde palace, code palce, palace code

Id: cZQDO0ktnrw

Length: 11min 18sec (678 seconds)

Published: Thu Aug 04 2022