How I use SELENIUM to AUTOMATE the Web with PYTHON. Pt1

hello everyone welcome John here today's video is the first in a short mini series on selenium and Python what selenium is it is a way that we can control our browsers using our code it's generally used for testing but it does do other things and we're going to look into those as well so this is the first video and what we're going to be doing is we could be getting installed taking a look at the webdriver opening pages filling out forms and looking at the weight function it does a lot more but we'll get stuck into that in native videos including web scraping so the first thing that we need to do is always to check out the documentation it's the best place to start and here we can see that we've got some demo code the first thing you want to do is to make sure that you've got selenium installed so that would always be a pip install so do pip install selenium always do - test user - it's sort of my separate user account only the second thing we need to do is we need to install the webdriver now there's two main webdriver ones that I use one is called gecko driver and that is for Firefox and the other one is the chrome driver for Chrome there's two things that you need to make sure before you do this is you make sure that you have the browser that is respective to that driver install so if you have the gecko driver and you want to use that you must have Firefox installed otherwise you're not going to get anywhere and the same with Chrome the second thing is to make sure that the web driver you download works with the version of the browser that you are using so check the version of the browser by looking at the about page or whatever and making sure you get the one that works with that so if we go back to the docs and have a quick look at what it says about the web driver we can see that here it says the web driver is a browser natively as a user would either locally on the rowing machine using the selenium server so we need to make sure that we have the web driver installed as appropriate to our browser version as I just mentioned so we go back to the main page and we click here on the selenium webdriver button we can see right away it tells you what languages it supports down here is his browsers and this is the ones I was talking about here Firefox on the gecko trial driver and chrome and Chrome driver so please go on to the documentation and download whichever one is appropriate to you if you're a Linux user check under your package manager and one might be in there for you that'd be nice and easy to install if not you can follow the downloads and you can download it that way it's basically just an executable file that needs to be in your path to be able to control the browser so once you've got it downloaded and you've got selenium installed we can go to our Python scripts and we can check that everything is working so now that we've got the chrome driver or the gecko driver downloaded and saved on our system and we've got selenium installed using pip we need to go to our python scripts our PI file and test to see that it works so to do that I will do let's make this bigger so you can see from selenium import webdriver like this you save that if we run that we get no errors we can see right here that means that's working the next thing we want to do is we need to set our webdriver that we've just imported so all the way to call it driver so we know what we're looking at is equal to webdriver dot and in this case I wouldn't going to use Chrome doc chrome just like that and what that's going to do is it's going to initialize the Chrome browser and open it up so if we run that now we should get a Chrome browser open here we go if everything's work properly at this stage you'll see that it says chrome is being controlled by automated test software and it's open this and then nothing has happened if you've BOS to get this far then everything's working fine if you're having problems with the webdriver not being in the correct path the default path for Linux is in here get this on the screen and it is that's the default and that should be the same for Mac for Windows it might be a bit difficult a bit more in-depth there are other guides out there but one thing you can do is you can put it directly into here so you could do C and then double backslash for Windows paths and then type the path to your driver here so if it was in your C Drive you could just do something this and that should work for you okay so now we want to make it so they opens up a URL for us so if we set a URL here and we'll just do google.com like so what we want to do is we need to do driver get and then our URL well this is going to do is it's going to go to this page for us we open that up you can see it's taking us to the Google home page now other things that we might want to do is perhaps fill out forms or click buttons or something like that to demonstrate that I'm going to use the same dummy login page that I used before and the URL for that is this one right here so what we're going to do is we're going to double check that our page opens up that you are correctly for us and I'll explain how to work out how to get it to put data in for you so now we've got this our page open here we can see that we have two fields and a username and password again we're going to need to use the inspect element function because we're going to be looking to find out the name or the ID of these form fields so we can input data into them so if we right click in here and go inspect you can see that go here we've got the field here now it has got a name and an ID but I prefer to use the XPath or cross path and that you can find here by going copy and copy XPath like that so I'm just going to put that into our script there so we know what that is and the same again for the password copy there that in and also we're going to need to be able to click on this button here so I'm going to find out where the button is now this looks like the class for the text or something like that I think this is the button here so you said button class so again save the thing great now we've got that saved there I'm going to close this now so we can get working here there's one other thing that we need to import them the selenium library if we want to be able to input keystrokes into our fields I our text so that's from selenium dark web driver common dot keys is quite long-winded but just copy and paste most of the time import Keys now what this will let us do is it will it should be a capital I think okay this will let us input our characters in so now what do we need to do to get our form filled out well we know that the paths of the boxes and the button to click on so we need to find those we need to tell the driver to find those and to do that we would do driver dot find element by path like this now what that means is it's going to use the XPath to find this on the on the web page you can do it by ID you can do it by CSS selector have a look on the selenium docs to see which one will work best for you I tend to go for this because I just find it easier so we know that the first one was the username so let's put that in there and now the same again but for the password and let's put a password in there like this and then we're going to need to find the button again so saying for that but it was again button like this okay so now when now our selenium is going to find these elements on the page but we need to tell it to do something with them so we can do dot send underscore keys and not this will do is will send the keys that we type in after this to this form and I know that the username for this is Tom Smith and the same for this which is super secret password and to click on a button is just simply dot click like this so hopefully need to put this above here so we get the URL before we try and find elements that aren't on the page so hopefully when we run this is going to click on put the text in the keys and click on the button for us and there we go it's logged in so what we've successfully done is we've opened up the browser automatically with our code we've sent in we've sent keystrokes to both of the form fields and we've clicked on the login button is us to the next page that's really useful we could log into loads of different websites this way we can test forms etc etc one other thing that you might find useful is to close the browser once you're done and we'll do that with driver dot quit like this now this is gonna flash by real quick I just run it again because it's got a longer sitting that's gonna close a browser logged in and done again another thing that we'll find useful is to use selenium to load dynamic content for us so we can access it this could be text or something loaded by JavaScript that we can't scrape the normal way I'm gonna go a bit more into web scraping in the next video in this series but right now I just want to show you how you can use the weight function in selenium to wait until an element is loaded so I'm going to clear this a little bit out here keep that the same and I'm going to change the URL again to this website but it was a different page as a dynamic loading page so if we open this up I'll show you what it looks like you can see now that we have when it goes going oh it's because I haven't I had taught care URL if you don't tell it to go somewhere it won't do anything so now we've gone to this website you can see we've got a button that is click you click on and then after a short loading screen it will load up some text so I'm going to do the same thing as I did before and I'm going to find the path of this button so if I click on this here we can see button so let's copy the path again because we want to click on this button so now we're going to put that there so when we actually click on this button and get the loading screen the information that comes afterwards as well we want that right here we are so now this is loaded up in our browser we can see that this text is here so we want to get this as well because we want to tell selenium to wait until this this is popped up before we do the next thing so if I put this in here so we've got that there as well okay that's close so what we want to do is to do again driver find element by ah and we know that this was our button here and as before we want to click on this button dot click like so and then driver by path and the text here and because we want the text of that we just do dot text at the end and we'll store that in text variable and then print the text at the end so we run this now we're going to get an error as you'll see because it's going to click on this button and it's going to immediately look for this element which isn't there because our simply loaded up yeah I move this you'll see the error okay there we go we got the error even though still working but we got the error stopping our code now to do a weight we have to import a few more things so we need to import selenium webdriver support you I import webdriver weight with a T on the end so in between these two pieces of code we want to put our weight so it ways to find this element before before our code ends there are two different types of weight and the one we're going to be looking at is a weight is an implicit weight which basically means that it's going to keep polling until it finds that element that within the allotted within the time we've given it I find that this is the best one to use for this sort of example so we were going to do driver dot implicitly can I spell weight and we'll give it a ten seconds so what that's going to do is going to wait ten seconds for this to for it to find this element which is plenty of time so now if we run it we should hopefully get our text value out because it's going to wait for it to no errors it's waiting it's waiting is waiting as soon as this pops up it finds the element that we were looking for and we've got our hello world out here what we've done in this is that we've used two lynnium to load up some dynamic data called a little wait for it to wait to find the element which was behind a click and then got that information so hopefully you found this this video useful get you going with selenium so go out there and try and open up some web pages let me know if you've got any cool uses for this I'd like to hear that and don't forget to like the video if you liked it and subscribe for more the part two and three will be coming shortly we're going to be looking at doing some more web scraping with selenium and then part three we're going to be looking at running selenium headless Lee on a Linux server which is a really cool thing cheers guys bye

Channel: John Watson Rooney

Views: 21,086

Rating: 4.9578948 out of 5

Keywords: python, learnpython, selenium, python selenium, browser automation, chrome webdriver

Id: pUUhvJvs-R4

Length: 13min 58sec (838 seconds)

Published: Wed Feb 05 2020