Python Selenium Tutorial #1 - Web Scraping, Bots & Testing

[Music] hello everyone and welcome to a brand new series on selenium in Python now for those of you that don't know what selenium is this is a framework that allows us to interact with HTML elements from any website so rather than just scraping information which we can do with selenium we can grab information from web pages we and also interact with it so we could in theory make things like BOTS we can have you know like an automated script that goes through and test different aspects of our website it's actually quite useful and it's pretty easy to get set up which we're gonna be doing in this video so some common examples of things you could do with selenium just basic movements is like drag-and-drop clicking a button filling in a form entering something in a search field grabbing data from some specific tags finding elements in the source code reading the entire page source code pretty much anything you want to do with a web page you can automate that with this Python selenium stuff the scripting which I'm going to be showing you which i think is pretty cool so we need to get into the setup here I will say first if you guys do appreciate this type of content consider becoming channel members by hitting the join button that you should be beside my channel name is a little promo there that tells you about what you get if you decide to become a channel member but I definitely do appreciate the support so with that being said let's get into the setup now the first thing we actually need to do is we need to install selenium so I'm gonna assume you have Python installed for those of you that are asking me this is a subline text window this is just a code editor it's a way that we can write Python code you don't have to use it you can use the standard ideally that I assume most of you probably have but what we need to do is make sure we have Python installed some version of that and then we need to get pip working which I'll talk about in a second install the selenium module and then install what I'm gonna call the Chrome web driver and I'll talk about that more when we get to it but the first step is to get pip working so pip stands for package installer Python and many of the popular Python modules are installed and kind of configured throughout that or through pip now pip is installed and comes by default with a Python installation so you do have pip on your computer sometimes it can be a little bit finicky though and not work earlier you can't access it so I'm gonna go through those mistakes quickly but first if you're on Windows open up a command prompt window if you're on Mac or Linux open up your terminal window and if you're on Windows so you're gonna type pip and hit enter if you're on Linux or Mac you're gonna type pip 3 and hit enter it may be PIP you might be pipe 3 try both of them if you're on Mac or Linux but the idea is that we hopefully want to get some output that looks something like this it doesn't have to be identical but something like usage pip command options you just don't want to get this error pip 3 is not recognized as an internal or external command so if one of these two work for you pip or pip 3 good pretty much ready to go if neither of them work on Mac Linux windows whatever then you're gonna need to configure pip and make sure that that's set up properly on your system now some of the common reasons for this it's not on the system path that's why you can't access it you do how PIP you just can't find where it is so I have a video called how to install PI game this teaches you how to fix pip all the common issues with it there will be a card and a link in the description click that make sure your pip is working and then come back here and start following along so once pip is working we have everything good all we're gonna do is in our terminal or command prompt window type pip install selenium in mind if I could spell this correctly so pip install selenium that if you're working in a virtual environment you're gonna have to make sure it's installed in that virtual environment I assume you guys know how to configure that if you are using a virtual environment but for most of us is just gonna be pip install selenium and assuming that works fine we're ready to go a selenium is installed now it's just time to get the webdriver so again make sure this works we can test if this is working by going into our Python window or python script and typing selenium so import selenium if we run that and we don't get any errors then we're good to go you know the program is fine if you do get an error then you got to make sure that you have this package installed in the right Python interpreter which is a common error that I can't really go over here because that's a whole video on its own but anyways once selenium is up and running I'm gonna assume you guys are at that stage now we're actually gonna have to go to this website which you can see I have here and it's gonna be in the description too the Chrome web driver so for this tutorial series I'm gonna be using Google Chrome as our main interface the thing we're gonna be working with I would highly recommend you follow along using Google Chrome so that's gonna mean download the Google Chrome browser if you don't have it and then come to this webpage which there'll be a link to in the description and we need to download the Chrome web driver associated with our google chrome installation so these are two different things the Chrome browser is different than the Chrome web driver you need both of them for this to work so first chrome second browser our driver sir so what we need to do is figure out what version of Google Chrome we're using and then download the appropriate web driver version so to find that out if you go to google chrome you can hit these little dots at the top right hand corner go to help and then about Google Chrome and it will show you right here if your chrome is up-to-date and what version it is you guys are most likely the same version as me but you may be a different version find that number and then download the correct version from this page so you can see that it says if you're using a t1 download this so will download a t1 but if you're using an older version you can scroll down and they have the older versions here so 79 78 all of that it's probably worth it just to update to 81 or 83 but it's not a huge deal so click that link and then you'll see that there is three or four files here and you're gonna download the one corresponding to your operating system so in this case we'll do win 32 so download that this is gonna be a zip folder I'm gonna put this in my downloads folder notice I already have that downloaded I shouldn't take very long and once we have that downloaded so you've moved past that stage you found the right version we need to extract that and place that in a specific location on our operating system so make sure you guys are following along here this is important this is hopefully save you if you headaches and some selenium errors extract this file so just double click into it if you're on Windows you know uncompress on Mac whatever it is you need to do and copy this file so I'm gonna do control C or actually control X cuz I'm just gonna cut it out of there but copy this file on your clipboard because we're about to move it to an easier location to access so I'm gonna go to local see drive here so local disk C Drive you can get that from going to this PC local disk C Drive if you have it there and all I'm gonna do is go into Program Files x86 and paste this file in in fact you can see I already have it here but for you guys that's where I would recommend you place this now what we're doing right now is picking the location that we want this file to live in so where we want chrome driver to sit so you can place this anywhere you want just make sure you know where you place it so that we can access that location because we're gonna have to reference the path to that file so the path to the file can be found by simply hitting the bar up here you just click on it and then copy what's here so in this case C Drive Program Files x86 I'm gonna copy that because that's where I place my chrome driver and now I'm gonna go on my Python file I'm going to say path equals this like that so the idea is take that Chrome file place it somewhere on your operating system find out the path to that file which you can find just by going up here or whatever directory the file is in copy that put it inside your Python file and now we're gonna add something to this which is the name of the file so chrome driver exe so the idea here is that we need the path to the file so that we can run it so make sure you get that correct and you put it in a location you can leave it in downloads if you want but typically I don't just cuz you accidentally delete things from downloads so anyways yes alright so now that we have that we're actually set up and we're pretty much ready to start kind of working with selenium and opening some things up I won't do anything too crazy here I'm not gonna explain a ton because that's gonna be mostly for the next video well the first thing we need to do if we want to actually open up a web page and start working with it is change our imports it from selenium imports webdriver now the webdriver is what is driving these actions it's what you know links up with the browser it's what's able to perform the actions so we need to pick what browser first of all we're gonna be using for selenium so the first thing we're gonna do is say driver and this is always the first step is equal to in this case webdriver dot and then you're gonna pick the browser you want to use so in this case I've been recommending chrome so we're gonna use Chrome with a capital C but notice you can use Firefox ie fan two chrome remote I don't know all these other browsers you can use as well so webdriver dot Crump now what we're gonna do in here is simply put the path variable that we defined above so we're going to say the web browser we want to use as chrome and the web driver that is for this browser is located at path which is here on our computer right so we do that there we go webdriver Chrome path and now what we can actually do is open up a web site so using driver we can say driver dot get and putting the name of any website you want I'm just gonna use my website I'm just to make sure I'm not doing anything illegal throughout these tutorials by running bots on websites we're gonna tech with Tim dotnet so drive it I'll get doc at Tech with Tim net we're gonna run that and notice wait a second data and then it loads up the web page like this and it says Chrome is being controlled by automated test software so that is how this works that's how we open up a web page that's how we get to what we want to do and I'll show you really quickly how we can interact with the web page just to give you a little bit more in this first video but that's the idea that's the setup get the driver downloaded get the pip package installed set up the path for that driver set up this driver and then we can access any web page we want so first thing if we want to close a web page what we can do is use driver duck close what this will do is close the current tab so that may not close the entire browser window it will just close the tab but if you want to quit the entire browser then you can do quit now obviously if you only have one tab maybe you run close it closes that tab it will close the browser but if you want to make sure you're closing the entire browser you can run quit so if I run this now let me give it a second you'll notice right as it gets to the web site it just goes ahead and quits because that's the next line right here now if we wanted to do something different like say we want to get the title of the web page what we can do is say driver dot title like that so we can actually print this out and this will tell us the title so what will pop up if I can find that here so see this tech with Tim Python and Java programming tutorials that's what it says that's get it's gonna show us that in the Python console so if I run this we look give it a second it's gonna quit and then it shows us the title Tech with Tim Python and Java programming tutorials Tech with Tim dotnet so we can go ahead and you know mess with that mess with a title we can check if things are in the title we can interact with specific elements and I'm gonna show you more and more but I think I'm gonna leave it here because we are hitting about the 11 minute mark and the next video will actually interact with this page I'll show you how we can type stuff in how we can hit enter on the page how we can navigate to different pages all that fun stuff and even more stuff to come in the future leave a comment if you guys enjoyed the video like the video of course subscribe to the channel and I will see you guys in another YouTube video

Channel: Tech With Tim

Views: 1,220,617

Keywords: tech with tim, python selenium tutorial, python selenium web scraping, python selenium, selenium python tutorial, selenium python tutorial for beginners, selenium web scraping, selenium web scraping python, selenium tutorial for beginners python, selenium tutorial

Id: Xjv1sY630Uc

Length: 11min 41sec (701 seconds)

Published: Sun Apr 26 2020