The most important Python script I ever wrote

I used to Lo Monday afternoons at work as Not only was it the beginning of the week but it was also the time I had set aside to spend a couple of hours what I could only describe as the most mind-numbing task I had to do in an otherwise interesting job data entry it was a sort of task that didn't take long enough to complain about although I did obviously but was tedious enough to Warrant me thinking that there must be a better way to do this I can't seriously have to sit here for hours and hours my first approach was to try and palm it off onto someone else that failed quickly so with that feeling of it shouldn't be this hard I carried on spending my Mondays copying and pasting and copying and pasting and not always getting it right fast forward a few months and my journey into python began I was looking for something new to get into my age and failing reflexes were condemning me to lower and lower ranks in Counter Strike and with my dream of going pro fading away I thought I'd apply myself to something else whilst looking for learning resources I came across automate the boring stuff by alswager it came highly recommended and I found the topics extremely enticing so I got it and started working through the examples whilst it wasn't the be and endle it was my Catalyst into my Rush B approach to learning as it really opened my eyes as to what was possible with python and programming in general after all you don't know what you don't know but now I did know know I was hurted I felt like a lot was possible a lot I saw how easy it was to read and write CSV files take data from one file and add it to another once I saw that you could control a browser my my mind was blown it sent me down a path of ideas and was convinced that within a few months I'd be able to automate all my work and I'd spend my 9 to-5 drinking coffee and boring my colleagues of with stories of how I was once in a top 20 ukcs Clan so off I went down the rabbit hole with beginner's Vigor and I started to climb that initial peak of the done in querer curve with my sights firmly on a particular Monday afternoon task but of course it wasn't all that easy I fell into every single trap and snake pit I wanted to give up but I didn't I took my limited knowledge banged my head against a few walls copy stack Overflow and eventually hack something together some Jacky ass code that would just about do the task for me just but it did they did it more quickly more accurately and even better than I could even when it inevitably crashed this would become the most important automation script I ever wrote and in the rest of this video I want to share you share with you a better more modern example of what I did so hopefully you can replicate it in some way and find your own path into automation with python I really think you'll benefit from trying to do this yourself first if you've never done this before go to playright use the documentation copy the example and build upon it and try to build a script that will visit a set of order numbers from a list and go to each page and try and pull some arbitrary data from that page do that first and then come back and see how I did it and you'll be able to compare and maybe pick up some ideas or maybe yours will be better than [Music] mine so the first thing that we want to do is we want to have a look at the site that want to automate because we need to sort of figure out the best way that we're going to do this obviously we need to log in which is actually demo and demo for the username and password then we click log in then we're going to have this box that's going to come up that we're going to need to deal with so that's worth keeping in mind then we want to come over to the sales the order section you can see all the orders have this order ID now if I click on one you'll notice that the URL changes the middle part change with the user token and then we have this and with the order ID parameter out here so now if I change this to three it changes the order so this means now I know how I'm going to Loop through all the orders when I import them and we'll just pull some arbitary information off this page simulate trying to get something like maybe we'll grab the flat shipping rate thing here or something like that to automate you know what we may or may not want to do from this um from this from this from this project so now I've got that open I'm going to come over to my edit my uh folder where I'm working and I'm going to create a new virtual environment d mvnv v MV this is how I do it you can use um virtual M if you want to make sure you do use a virtual environment for this we need to activate it I'm going to type act which is my shortcut otherwise it's Source VMV bin activate um and then I'm going to do pip three install and we're going to be using playright um I actually started with uh selenium um playright wasn't as widely available then but this is a much better option nowadays I think for this sort of thing although selenium will work just as well I'm just going to create a couple of new windows just my workflow if you're using vs code or something you obviously don't need to do anything like this you can just use vs code and use the terminal at the bottom so now I'm going to create a new file I'm going to call this main.py and we're going to open that with my code editor here I'm going to come out of full screen because we want to go over here I'm going to go to playright dodev have a look at the website we're going to go to get started and get started library and right here we can literally copy all of this and paste this into our code to get us started so I'm going to move this window off onto another screen just so I have this one open only I'm going to save this and I'm going to come and I'm going to run my main.py file and we can see that it works now that's because I already have the chromium installed if this is the first time you're going to want to do play right install chromium you can do install and install all of them but I would just recommend using chromium you don't need to use the other browsers this one will work just fine save yourself some hard drive space so make sure you do that step if this is the first time that you're working with this so now that I can see that that works I want to wa one change up here and we're going to do headless is equal to false we want to see the browser when we run it and now when we do so we can see that that works worked just fine so the first thing that we want to do is we want to come back and grab the actual URL that we want to work with which was this one here and it was the admin page cool so let's copy this and we'll put this into here instead of this URL there we go I'm going to remove this as well and whilst we're working I'm going to import in time this is just an easy way I find to um put some sleeps in so I can kind of really see what's going on cool so let's save that and I'm going to run this now on the openart website we can see that it's popped up here and I have the logins Pages etc etc and because I had that sleep it's done cool so now I'm going to bring the my actual browser back over and I'm going to minimize the code a little bit hopefully you can still read it and what we want to do is we want to open up the inspect element tool I'm going to move this to the bottom and we'll move it over here like this so what we want to do is we want to actually now use the um use playright to type the username and password in for us and uh then hit submit so we do page Dot and I'm going to do get by placeholder in this instance and this is going to find the element for us there's a few different ways you can do find element but because we have a placeholder value here I'm going to do it like that so username like so I'm going to do fill to fill it in with the word demo I'm going to copy this line and we'll do the same for password let's save and now we'll do uh time sorry we'll do time dot sleep just temporarily so I can show you and let's go ahead and move this back over and run it again and we should hopefully now see that the demo has been filled into that browser on the right hand side which it has perfect so we can now go ahead and do clicking that button so we want to do page. get by and I'm going to do get by roll in this instance and thank you to my LSP which makes it quite difficult to see it is button and the name is equal to login is a capital l I'll show you where this comes from in just a second but here we can do dot click so hopefully you can see how you can start to um click on things on the page so if I go back to this there's a button button and the text uh was login I think name was login um I don't see the name on this one this might work anyway let's try it and we're going to make the Sleep five just for the moment and now let's see if we can click that button we can this is going to give us this Cloud faving verifying we're human this is just this probably won't happen for you it's probably not going to be an issue it just happens on this website for whatever reason so now that we're here what I'm going to do just to make sure we get through that I'm going to make this 15 um just to make sure that we're not getting stuck here and we do actually get to the next page although we do have that URL you can see up there which has our good token in okay so that does work perfect so now we just need to get through that um next thing here so I'm going to let's make this 10 for the moment and now again I'm going to do page. get roll get by roll and again it's a button and we want to select this um button here that comes up let me just log back in over here this one so what I'm going to do is I'm going to hover over this little cross mark here we can see that we have this this button here and I'm going to go ahead and do copy selector and put this in here I don't really like the look of the selector but what I can actually see here is that this is going to be um the end button now quite often when you do copy the selector it should look it might not look like this it might look a little bit bit neater depending on um yeah see we we're not we're not we're not in the right place okay so I think this one will work if I do nth one do click because this is probably this will be the first index of the button so I'm just going to check that that will work for this instance and we'll do do time. sleep again so we should now be able to log in just wait for the uh verification thing to happen this this is strange and do we click on the button no yes we did click on the button you can see it worked so this did work um again you'll have to play around with the selectors for the buttons you want to click on just to make sure that you're in the right place and we'll do this one now sleep by one and we'll chop this one down to five and that should be good for us at the moment so now we're actually loading up that page and we're inside the main sort of bit so what I want to do from here is I want to get our page uh we'll call this order no call this page URL and that's going to be equal to page.url like this and uh no it should be like that and I'm going to print out the page URL and I'm just going to show you how I go about constructing the URL for this order page um gu it's slightly slightly complely different depending on like how things go about okay so here we go here's the URL now if we compare that to our order page URL where we can actually get the orders let's move them side by side you can see we have this root common dashboard and here it's rout sale SL order and then at the end of it we actually get info and then the order at the end so what I want to do is I want to come over here and I want to say sale page it's going to be equal to our page. URL if I can type which I cannot I'm going do do replace so we want to replace the part here root common dashboard in fact we just need to replace this part don't we I want to replace that with this after the root this like so okay that should be good so I'm going to move this back over so now we have they're going to have a decent URL constructed for us we can actually start to um let's move that there uh we don't need that now let's do um for order in orders I'm going to sort that out in just a second we'll do page. go to because we want to go to a new URL that's going to be our sale page URL so let's go ahead and let's put put a quick orders list in uh 3,000 3001 3,2 so we can Loop through these and but but this is going to give us that URL but we need to add on the end of it are what was it order ID equals cool so we can put that in and then I'm going to do plus the string of the order this is going to be our full URL for that set that sales page so we can go there so I'm going to do time do sleep is equal to two so we should have now two seconds on each of those pages when it loads up when we when we get in and we just need to put the Amper sand in there like that so now this should work okay so we can see we're on pay order 3000 3001 3002 and we can see that that's how we're going to be able to Loop through these orders now from here we want to grab some information so I'll just I will go for let's just get the shipping method uh this could be anything you could be inputting data you could be collecting it from anything like that um but to do that we're going to do page. query selector and this is going to be give us access to um the sort of the CSS selector on this page so I'm going to come over here and we're going to grab this and this is now going to be um one of these here so input shipping method like so so this is going to get us to this this element then we want to get to this one here the option value uh the option one the second one down so to do that we can get uh this needs to be an ID so this needs to be hash then after that we want to go option but we want the nth child child of two I believe should get us there so now we have this shipping method what we want to do is we want to say if shipping method because you know some orders might not have it then we'll do print shipping method. text text content and we're also going to now append this to a results list so we call this results and we'll do results. append and we'll put in a twole so we'll do order and the shipping method do text content then after our browser closes we'll just print out our results so this looks pretty good so far I'm going to save and I'm going to run this and hopefully we can have pull out all those shipping method uh text information now I think think on this website they're all the same so you know we're just going to get flat export or whatever each time flat shipping rate but you can see it's coming through like so and then at the end we have our list of tupes with each one available in it and as I said before you can easily go ahead and do anything else you want to do on this page we could you get any bits more bits more of information or if you wanted to add and add something there's a comment you could type into this box by finding the selector for this box like this and type text in here and then add that would work too no not a problem um so there's quite a lot you can do with that the one the last thing I want to show you is is we're going to import CSV and we're going to do uh with and we'll have an orders. CSV file CSV and we'll have this is read as F we'll have our reader is equal to CSV do reader and then we'll have our orders it's going to be equal to uh order index of zero for order in reader like so I believe what have I done wrong here with open cool so now I have uh the opportunity now I can create an orders. CSV file and we can open it up here and now we can have our 3,000 3001 3002 33 let's go for that save this and close this buffer so now what we're doing is we're just going to import our orders from a CSV file which you know obviously if you were doing this for your job like I was I had a massive long CSV file so I just imported it I just put them all into the orders. CSV in the main folder with this and I would just set this code running and instead of printing the results I would um export them to CSV or whatever and um yeah that would work so let's just run it let's just uh run it one more time and we'll just see and double check this all works fine now this is a little bit fickle so we're we're going for direct selectors which means you know if something changes it might not work but you'll find as you go along that you'll need to tweak it you'll find where bits stop working um maybe certain orders don't have that information on so you'll need to change the selectors I'm using time. sleep Which is far from ideal but it does work there's all sorts of Weights that you can use for um playright so if we look here it's waiting for events waiting for State all this sort of stuff Alo waiting so it's worth looking into that and uh and choosing and sprucing this up a bit but I really wanted to show you sort of the most basic version and sort of how you can really easily do stuff like this across websites um for your job or for whatever you need to do and this is 38 lines of code and this could be less or certainly much better in some cases but hopefully you've enjoyed this video if you have like comment subscribe it really does make a difference to me um come and join the Discord we got big Community there have a chat and come talk to me um yeah thanks for watching if you've enjoyed this and you want to know a bit more about Automation and how you can get playright to actually write some of this code for you I'm going to direct you to my other video right here

Channel: John Watson Rooney

Views: 15,104

Keywords:

Id: q1GDSHhaH0E

Length: 19min 57sec (1197 seconds)

Published: Sun Apr 28 2024