Excel VBA Introduction Part 57.5 - Implicit and Explicit Waits in Selenium

welcome to this weissel vba tutorial in this video we're going to cover implicit and explicit weights in selenium basic we'll start with a description of why it's important to wait for elements to exist before you attempt to interact with them and then we'll describe the implicit weight system and how you can adjust that to your own needs we'll explain how you can set a timeout for an individual statement which tries to find an element on a page and then show how you can set a basic explicit weight for a set period of time towards the end of the video we'll look at some of the more interesting ways you can wait for elements including an example of waiting for an element to be displayed on screen so a couple of interesting things to do here let's get started to get started i've created a new excel workbook and i've saved it as a macro enabled file just a quick reminder that if you want to follow along with the video you'll need to make sure that you have selenium basic installed and if you haven't already done that we have a video which explains how to get it set up the first example we'll cover in the video is going to be based on the wikipedia main page and specifically what happens when you type something into the search box you probably already know this but if you type something in you get a list of suggestions and i'm interested in getting that list written out so that i can follow some of those links so while i'm here i've got the url for the main page for wikipedia i'm just going to copy that to the clipboard now i'm going to back to excel and open up the visual basic editor and let's write out the basic code to navigate to that main page so i'm going to head up to the tools menu in the vb editor and choose references and i want to make sure i can set a reference to the selenium type library so i can click into the list and type in the letter s to jump down in the list to find selenium type library and when i've checked the box click ok i can then insert a new module into the project and then at the top of this module i'm going to declare a module level variable which will hold a reference to my chrome driver so i'm going to be using google chrome for this example so i'll say private cd as selenium dot chrome driver i'll get there eventually the subroutine we're going to write it doesn't really matter what it's called i'm going to call mine wait for elements and then the first thing i'll do in here is set a reference to a new instance of the chrome driver so i can say set cd equals new selenium.chromedriver and then i can start that browser and i can say to get the url for the wikipedia main page so in a set of double quotes i can just paste in what i've copied from my open instance of chrome and then i can give that one a quick test to make sure that part's running and when the page finally appears there we go okay so we're good to go ready to start interacting with that search box in order to interact with the elements on the page we need to have some way to identify them and a nice way to find out how to identify an element is to inspect it in the web browser so i'm going to head back over to my open instance of chrome and the first thing i care about is the search box itself so i'm going to right click on the search box and choose inspect in chrome that opens up a panel on the right hand side of the screen which shows you the element you've just inspected and provides several possible options for identifying it i'm going to use css selectors for this particular video if you've watched the previous video in the series you'll be aware that there are several ways to identify a particular element so what i'm going to do here for the sake of convenience is to right click on that element and choose copy and then copy selector that will automatically pick the best way to identify that specific element and because this element has a unique id called search input that's what will be copied to the clipboard so at this point i'm just going to head back to the vb editor i'm going to add a quick comment that says interact with search input and then on the next line add another comment and then paste in what i've just copied i'm not actually going to perform the interaction just yet i want to focus on the list of suggestions first before we interact with the search input so i'm going to head back to wikipedia again and then i'm going to make my suggestions list appear i'm going to type in something into the search box and then i can right click on any of these suggestions and again choose to inspect it so that jumps down in the list now to the list of suggestions that have appeared and it's highlighted the one that i right clicked on i'm more interested not in a single individual suggestion but the container which holds all of them if i just scroll back up a tiny little bit i'll find this element a div element with a class called suggestions there's no unique id for this particular element but again i can still right click on the element and choose copy and then choose copy selector i'm then going to head back to my vb editor and i'm going to add a quick comment get list of suggestions and then on the next slide i'm just going to paste in what i've copied from google chrome to see what we get so there's an example of a css selector it's looking in the body element of the page and then for a direct child of the body element which is a div element whose class is suggestions now this is a little more elaborate than it needs to be to get what i want i can get away in this particular case i'm pretty confident that there is only one suggestions element on the page so i don't actually need all this qualifying code here i can just look for something with a class of suggestions i am going to add a variable which will allow me to capture a reference to that object then i'm going to say dim suggestions if i can spell that correctly suggestions i'll try again i'll get there eventually i promise there we go suggestions div as selenium dot web element so a single element and then i'm going to say set suggestions div equals cd dot and i'm going to use the find element by css method so inside the round brackets and the double quotes all i need to do is pass in the name of the class i'm interested in preceded with a full stop of course that's important to identify a class i can then close a double quotes and close around brackets just get rid of that comment that i put in and there we go we've identified the elements we're interested in at this point i'd just like to run the subroutine to see what happens so i'll hit the executor run button i'll see a new instance of chrome opens up i can see my swirly blue mouse cursor staying there for a few seconds and then eventually ends but if i switch back to the vb editor at this point i get a runtime error telling me that it couldn't find an element with this with the class suggestions which is funny because i'm pretty sure i just identified that from my other open instance of chrome so what's going on here if i just zoom back out again and then i'm going to click debug just to show that it is that line which has caused the problem of course i guess that's fairly obvious and i'm going to have a look at my instance of chrome that's been opened up by this code and if i just right click anywhere on the background of the page i want to not right click on the input box this time so i'm going to right click somewhere on the background of the page and choose inspect i'll just maximize that window there as well what i'd like to do now inside this elements panel on the right hand side is try to find something called suggestions so i'm going to press ctrl and f to open up the find box and i'm going to try to type in suggestions and you can see that it hasn't suggested any elements have the word suggestions as part of it and the reason for that is that the suggestions list doesn't get generated when you first launch the wikipedia page the suggestions list only gets created when you interact with this search box so if i click into the search box i don't know if you can see now already in the background it's generated at the class called suggestions so if i click back in the search box here or the find box and press enter now it's found multiple elements with suggestions and i can scroll through them to see that it's definitely found them so what does that tell us well let me just close down this instance of chrome and i'll stop running this procedure and so that tells us that there is a potential that we might not find a particular element when we first uh try to look for it i want to look at how you can control how long it takes for selenium to search for something before it fails i'd just like to add a bit of extra code that will tell us whether or not the suggestions element is present i'm going to do that first of all just after we've loaded the main page of wikipedia so i'm going to declare a new variable called find by and it's going to hold a reference to a new instance of the selenium dot by class i can then use that to print some information into the immediate window so i'm going to say debug.print and then i can use the is element present method of the chrome driver so cd dot is element present i can then open up some round brackets and then i need to pass in a reference to a by object so i'm going to say find by and then i'm going to use the css technique again and then in some round brackets and some double quotes i'm just going to copy and paste the dot suggestions class and in fact i might as well copy everything to the end of that line and then close the extra set of round brackets so that method returns true or false depending on whether that element has been found or not but there's no delay on this one so you notice when we get the find element by css there was a delay this one detects at any given point whether that element exists it's probably worthwhile just adding a bit of extra text to the beginning of that so i could say suggestions present and then concatenate that with the result of the the function let's just display the immediate window as well so that we can see the results of this when we do run the subroutine and if i do run it one more time we'll see that we get the same pause again so waiting waiting waiting and then when it's finished we'll end up with the same runtime error but it has told me before it tried to find that element or get a reference to that element that the suggestions present is false just as we'd expect the delay that we're experiencing when we can't find the element we're looking for is due to the built-in or implicit weight of the web driver we're using by default all of the find element methods in selenium basic have a built-in delay of three seconds to give the element we're looking for chance to be created and this is a good idea in websites whose content is generated dynamically we've already established that the suggestions div doesn't exist until we interact with the search input box so if you imagine if we had filled in this bit of code here to search for something there could be a delay between typing in what we're looking for and the list of suggestions being created if our code ran without delay it's possible that we could try to capture a reference to the object before it had chance to be created which would lead to a runtime error so that three second delay is designed to give the element a chance to be created you can write code to modify the duration of the implicit weight in case you wanted to make it longer or shorter to do that i can head back up to the top of the subroutine just after i've created the new instance of the driver and then i can say cd.timeouts dot implicit weight so this property can be set to be the value of the delay in milliseconds that you want the default is three seconds if i wanted to make it much shorter i could type in 100 so a tenth of a second and then if i were to run the subroutine again we'll see the same thing happens but after the web page is loaded i don't get the long three second delay you can see it's finished virtually instantly i'll still end up with the same end result here because we know at this point the suggestions div isn't being created but the code takes a lot less time to run you can also change the timeout for each individual find element method if i click back into the round brackets where we're trying to find the suggestions div if i type in a comma after the css selector parameter i can see that i've got an optional timeout parameter its default value is -1 which means it will just use the the default or the implicit wait time but if i wanted to say for example the suggestions dip is one that may take a long time to be generated i could bump this up to say 3 000 milliseconds which sets it back to the default and then if i were to run the subroutine again i'll see that i this time when the page is loaded i do get back to that three second delay before the circular mouse cursor disappears again so you can either set a default a default implicit weight by changing the timeout's implicit weight property or you can do this on an individual basis each time you attempt to find an element let's move on and add some code to actually interact with our search input box i'm going to add a bit of code where i've left my comment earlier on i'm going to declare a new variable called search invert and that will be a selenium dot web element and then going to set the search input to be equal to cd dot find element by css and then in some round brackets and some double quotes i'm just going to copy and paste the id that i copied earlier on and in fact i can just then get rid of that comment so what i want to do once i've captured a reference to it is i'd like to click on it so i'm going to say search input dot click i'm just going to tidy up and get rid of the implicit weight and the explicit timeout that i added on earlier on so i'm going to remove the code which modifies the timeout implicit weight property and i'm also going to get rid of the parameters value for the timeout in the find element by css method so that's now everything working back on the default value of three seconds i'd then just like to quickly copy and paste the line which tells us whether a suggestion whether the suggestions element is present so we've done this at the very beginning of the procedure before we've done anything on the page just after it's been loaded so that will always return false because we haven't done anything on the page yet i'm going to copy that line and i'm going to paste it in just after i've clicked on the search input box and then i'm going to paste it in one more time after i've captured a reference to the suggestions div having done that i'm going to clear the contents of the immediate window and when we run this we should end up with three messages telling us whether the suggestions element is present at three different points once we load the web page after we click the search input and then after we capture the reference to the suggestions div so if i run the subroutine at this point see the page loads i don't get the three second delay this time so that's a good sign if i click back into the background of the vb editor i don't have any runtime errors it's reached the end and you can see this is the importance of having that built-in weight for elements to exist clicking the search input does generate the suggestions div but when we try to print out whether it exists or not immediately after the click the page hasn't had time yet to generate that div element we have then the code which finds the element with its built-in three-second implicit weight it doesn't wait the full three seconds of course once it can retrieve the element it will then proceed with the rest of the code so you can see finally on the last line there that the suggestions element is indeed present let's move on and try to list out the suggestions that wikipedia comes up with based on what we type in so the idea here is that if i go back to my open instance of chrome each of the suggestions generates an anchor or a element and i want to write out the text of that element and the href to get the actual url that it's pointing to so all of those a elements exist within the suggestions div let's tidy up the code a little bit first i'm going to get rid of these debug.print statements just to clean things up a bit so i've got three of those to remove and i'll get rid of the one at the top there as well i would also then like to modify what i'm doing with the search input rather than just clicking on it i'd like to search for something so i'm going to change the click method to the send keys method instead and once we search for let's search for vba why not it seems appropriate what we can then do is declare a couple of new variables so after we've captured the reference to the suggestions div i'm going to say dim suggestions i don't know why i'm trying to return that from the intellisense this is a brand new variable so dim suggestions as selenium dot web elements so that's going to be a collection and then another variable to hold a single suggestion so dim suggestion as selenium dot web element to capture the entire collection of suggestions i'm going to look in the suggestions div element for any elements with an a tag so i can say set suggestions equals suggestions div dot find elements by css and if i want to find any elements with a particular tag i can just write that tag name in in a set of double quotes so in this case it'll be the a tag what i can then do is loop through the collection of suggestions i can say for each suggestion in suggestions and then next suggestion and then inside this loop i can say debug.print suggestion dot and i can type in i can refer to the text method to return the text of that a element and then a comma and i can say suggestion dot attribute and then in some round brackets and double quotes i can return the href attribute of that object as well okay now this is where things get a little tricky to test and if i run the subroutine i genuinely don't know whether or not i'll get a list of suggestions printed out so let's find out i'm just going to run it and see what happens when we search for vba so we can see that we load the web page and vba gets written out into the box and then if i look back at the vb editor i can see that oh it didn't actually write out those suggestions it didn't fail to find something you can see that i didn't get any runtime error there was no delay the code just ran really quickly and didn't have chance to capture the list of vba suggestions the trouble is that that's not always necessarily the case we've got a bit of a race going on between wikipedia loading that list of suggestions and our code running to print out the list again it's my codes run too quickly this time for wikipedia to catch up so again i didn't get the list of suggestions but we can keep on trying this and sometimes you'll see and i say this is tricky to test but sometimes you'll see in this case it has worked so i got two failures and one success just by running the exact same code three times without making any changes so what can we do about this first of all it's worthwhile explaining why we didn't see either a runtime error or a delay when we were trying to find the a elements but they hadn't been generated yet the reason is we're using the find elements method rather than a find element method when the find elements methods run they can return an empty collection so if there are no a elements it will still return something without failing it'll just be an empty collection so we're essentially here looping through a collection of no items we can demonstrate that fairly clearly actually we could do a debug.print of how many elements the collection contains so we could say debug.print suggestions dot count and then if i clear the contents of the immediate window i think what i'll do is make sure that we're looking for something which we know won't return anything at all so let's change the the suggestions to look for something with any old class name so i know i'm pretty confident that wikipedia doesn't have any elements that have that particular class so we should return a collection of zero elements but without a delay and without a runtime error so we can run that and we'll get the page loaded up and we'll get some suggestions but nothing gets printed out but we get zero uh elements listed out it tells us that that collection didn't contain anything okay so how can we solve this problem i'm just going to clear the contents of the immediate window and get rid of my debug.print statement as well and i'll revert this find elements back to looking for a elements one thing we could do is set an explicit weight after we've captured the suggestions div we could tell the code to pause before it continues now there's a bit of guesswork involved here how long do you tell it to pause for well let's just show you the practical application of this let's say cd dot wait and then you can specify a time in milliseconds i'll go for let's say half a second let's say 500 milliseconds so if i run this one at this time we'll see we search for vba and it prints out the list and you saw there was a short delay there before it actually printed out that list you can see this even more clearly with a much longer delay if i stick in let's say let's go for two seconds let's go for 2000 i'll just clear the contents of the immediate window again and then we can run the subroutine one more time and you'll see there's a clear delay there of a couple of seconds before anything happens in the immediate window so the problem with this is that there's a bit of guesswork involved and of course no matter what conditions are true that delay will always happen which is a little bit irritating when you know it could potentially work a bit faster fortunately selenium provides a range of ways to test a variety of states of an element a good one to check for this particular example would be whether our list of suggestions has appeared on screen yet so there's a potential delay between typing something in and that list of suggestions being displayed so we can check that using a property of the element let's head back to the visual basic editor and i'm just going to get rid of the cd.weight line and i'm going to replace it with a debug.print statement first of all so let's say debug.print and then i can say suggestions displayed and then i can concatenate that with a reference to the suggestions div and then i can check the is displayed property a simple boolean property which tells you whether or not the item is displayed on screen there are a couple of other useful ones worthwhile mentioning there's an is enabled property so whether you can interact with the element there's an is present property you may have captured a reference to an element when it existed at some point but then the web page may have changed so that that element has been removed so you can check whether the element still exists or not then you can also check if it's selected so suggestions dave dot is displayed i then actually like to wait until the suggestions div definitely is displayed on screen and tied in with the is displayed property there's an option to wait for the element to be displayed to do that i can simply say suggestions div dot weight displayed and you can see again there are several other options there for waiting until a specific condition has been met so for this example i'm going to say suggestions div dot wait displayed and then i'd just like to print out the status of its is displayed property again so i'm just going to copy and paste that line okay so when we wait for it to be displayed we can pretty much guarantee that this one will always return true but the first printing of the is displayed property might not return true because it might not have had time to be displayed on screen yet let's see what happens if we search for vba again again it's difficult to predict exactly what will happen when we're testing these sorts of procedures so let's close down that and see what we got so we got two trues so our our div was displayed in time this time i got the other the true result from the displayed property let's maybe change the suggestion that we're searching for so let's say selenium this time i'll clear the contents of the immediate window and then run the subroutine again let's see what happens this time good good so we'll always get our list of results we're waiting for that list to be displayed and it came out with true and true again uh shall we give it one more try tell you what let's search for something completely different let's search for let's search for where wise owl is based a little town of gloss-up there you go learning something about wise owl so i'll run that one one more time so of course we'll get the list printed out because we're waiting for that list to be displayed ah there we go good so in this case we uh we ran this instruction before the div was displayed but then this one of course waited for it to be displayed before it printed out results so there we go for the next part of the video i thought it would be good to show a slightly more practical example of some of the weight techniques we've seen here this example actually comes from a question asked by someone who is watching one of the much earlier videos i created on web scraping using internet explorer and what you want to do is capture the flight information at cairo airport so when unloading this page you land on the arrivals for today and what you want to do is scrape that table into an excel worksheet and then click on the link for tomorrow's flights and get those and then head to the departures section and grab the table for today's departures and then likewise get the departures for tomorrow and you can see as we're clicking between these different versions of the different tables that we're getting a short delay in loading that information so we need to make sure that we're waiting for the appropriate time to capture the information in the table so while i'm here i'm going to copy the url from the address bar and then head back to the visual basic editor and i'm going to create a new subregion i'll add this up at the top of the the existing module and i'll call this something like cairo flights and then i'll just quickly paste this in as a comment first of all and then i'm just going to copy and paste the first few parts from my original subroutine so creating a new instance of the chrome driver starting it and getting a url so i'll copy that and then just paste that in at the beginning of that new procedure so that we're basically navigating to the flight information for cairo airport now let's look at how to identify the elements we want to copy if we switch back to chrome and then i'm going to right click on the top left hand corner cell of the table and choose to inspect it you can see it opens up a fairly deeply nested set of elements and the one i'm interested in is the table element so you can see that if i hover the mouse cursor over that it highlights the entire table it's got a class a compound class in fact it's got table table dash striped and fl table as it turns out there's more than one table on this page which has that collection of classes assigned to it in fact if i double click just to highlight that and then copy that text then i can press ctrl and f to find and i find that's pasted into the to the search box and i can see that i've got two different elements on the page which have that collection of classes so the one that i'm looking at at the moment let me just collapse the header row of that you can see that corresponds to the table body with the unique id of arr short for arrivals i'm assuming if i head down to the next version of that i can see another table i can't see it highlighted on the page because it's not actually visible at the moment but if i expand that i can see that it's got its own body with its own unique id called departures so you'll see that if i switch to the departures page i can now highlight that table because it's visible on the screen if i switch back to arrivals that table is no longer visible so i can't access it so that's useful information let's head back up to in fact let's just use the the search box let's head back up to the original table the arrivals table if i expand the body of that we can see all the individual rows in there and i can see those being highlighted if i switch between today and tomorrow you can see that the structure of the table doesn't change it's still the arrivals table that's being displayed on screen it's just changed the information inside that table so it's updated the actual data using some javascript uh yeah i'm not quite sure how that works but anyway we can see that the contents of the table change that's the important part so with that table class available it's probably a good idea to get a reference to all the tables with that collection of classes assigned to it so i'm going to right click on that table element and choose copy and then copy selector i can then head back to the visual basic editor and let's declare a variable let's call this let's call it flight tables as selenium dot web elements as we know there's going to be more than one of them and then we can set flight tables to be equal to cd dot find elements by css and then in some round brackets and double quotes i'm just going to paste in the selector that i've copied hmm that's not actually the selector i wanted i actually wanted the names of the classes so i'm going to head back to the flight information page and i'm just going to double click to highlight the three classes in that table attribute copy those to the clipboard head back to vba and then paste those in to reference compound classes with css selectors you just put them all together separated by single full stop characters again we covered that in the previous video in the series where we talked about css still actors so at that point it's probably worthwhile quickly checking that we've captured some tables so we can say debug.print flight tables dot count and then if we run the subroutine we should find that when the page loads the tables appear and if we look back at the background of our vb editor we have captured references to two tables in that collection okay so writing out the contents of one of those tables to an excel worksheet is pretty straightforward we've got a reference to a table element and there's a method built into selenium basic that lets you convert that into a table and then write it directly out into an excel sheet to do that let's refer to the first table in our collection so i'm going to say flight tables open some round brackets and then refer to the first element in there i can then apply the as table method to that single table and then i can say to excel and at that point all i have to do is specify where i want that table to go i'd like it to go into cell a2 of a brand new worksheet in this workbook so i'm going to say this workbook dot worksheets dot add and then say dot range a2 and the reason i'm using range a2 is so that i can then say range a1 so when you insert a new worksheet it becomes active so i can say range a1 and refer to cell a1 of that sheet and say dot value equals arrivals today okay so at that point i'll just comment out or in fact delete the debug.print statement for the time being and then run the subroutine again i will load the page load the tables and when i get back to excel i'll see that i've written out all the arrival information from today so just a quick sense check there just a quick visual check that we're looking at the correct information that all looks pretty good to me okay next i'd like to get the arrivals information for tomorrow so i'll just close down this instance of chrome and i'll also delete that worksheet so that we don't end up with loads and loads of extra worksheets and then switch back to my open instance of chrome and i want to find out what the name of that button is or that that link element i'm going to right click on it and choose to inspect it and i can see that the individual element is an a tag an anchor tag like those that we saw for the wikipedia example it doesn't have a unique id but what i can see is that that a element is nested inside a list element so you see an li element there and that does have a unique id t-o and there's another one for i guess today's which is tt and yesterday's which is yy so i'm going to rely on using the list element and i can find this element by its unique id and then click on that to load the next table so i can although it's not too much effort to type in to i can right click on that element and choose copy and then copy selector and then i'm going to head back to the visual basic editor and then after i have pasted in today's arrivals i'm going to say cd dot find element by css and then open some round brackets and double quotes paste in the unique identifier that i've just copied and then close the double quotes and close around brackets and then say dot click so i will click on the tomorrow button and load the new table what i'd like to do then is just retrieve the contents of flight table 1 again so the first table is the arrivals the second table is the departures so i'm looking at the arrivals for tomorrow so the simplest thing to do here is just copy and paste those two lines of code and then update the range a1.value to say arrivals tomorrow instead so let's give this one a quick test if we click somewhere inside the subroutine and choose to run it we'll see it loads up the page and we know that it already captures the first table so today and we can see that it has indeed clicked on the tomorrow button and it will have generated a new worksheet with tomorrow's information so let's compare the results let's head back to excel so i'm looking at currently i'm looking at the arrivals for tomorrow but if i look at that i mean it's fairly obvious at this point already that apparently some arrivals or tomorrow have already arrived quite a lot of them in fact and if you compare this page here with what's displayed on the actual web page we're not getting the same results if you look at sheet 3 which is the arrivals for today and sheet 4 the arrivals for tomorrow you can see that we've failed to capture the new information our code ran too quickly the code won the race between the website updating and getting the new information so this is the point of course at which we need to think about how we can pause or wait until the new table has updated let's just clear up the worksheets i'm going to highlight and then delete both of those new worksheets i'll close down that extra instance of chrome as well and head back to the visual basic editor and i guess a nice simple way to do this is just with a basic explicit weight i don't think it takes too long for that table to be updated i think a delay of maybe a second or perhaps half a second would be sufficient to make this work so let's say cd.wait and then i'm going to pass in a wait time of 1000 milliseconds one second so if we run this one again now we will see chrome is interacted with in the same way so we'll end up with a start with today and then click the tomorrow button i think we could have gone away with a lot less time than just one second let's have another comparison now if you look back at the results we can see sheet six that looks more promising already so nothing has arrived yet for tomorrow which is a good sign and if we compare that with what's displayed on the actual web page that looks like we've got what we want just to quickly reassure ourselves i suppose that sheet 5 the arrivals for today is still returning the correct information that all looks pretty good to me now let's look at capturing the departure information i've already tidied up my workbook by deleting those extra worksheets and i've closed down the other instance of chrome we saw when we printed out the number of tables we had captured that we got two tables so even before we display the departures table we do actually have access to it in our flight tables collection let's see what happens if we attempted to just print that information out so if i just copy the line which prints out today's arrivals and then paste that in down towards the bottom i could attempt to change the flight tables to two and then the text here it doesn't really make much sense let's just change it to departures and then if i attempt to run this subroutine i'll end up with three worksheets in the output which is what you'd expect and the code is running i get no runtime errors but if i look back at excel you'll see that in the departures table it's completely empty um the problem with this is that i can't get the information from the departures table until it's visible and to make it visible i need to click on the departures button back in chrome so let's close that sorry back button let's get rid of these extra worksheets first of all and then i'll get rid of this extra instance of chrome that we've opened up and then head back to the previous instance of chrome and i'm just going to right click on the departures button and choose to inspect it to find out how i could reference it so i've found a span tag but that's nested inside an a element which has a unique id h r f departures that's a promising sign any unique ids are always a good thing so we can see that if we click on the departures table oh in fact let me just quickly select the tomorrow table for arrivals when i click on the departures link you'll see that it loads departures but it also flicks back to today so if i click on departures that gives me today's departures so i'm going to right click on this a element in the elements panel and use copy and then copy selector and then head back to the visual basic editor and then i'm going to say cd dot find element by css open some round brackets and some double quotes paste in the selector i've just copied close the double quotes in the round brackets and apply the click method to that i'm then just going to update this departures today and then let's have a look if i run the subroutine at this point we'll end up with three new worksheets eventually so we can see it clicking through the various elements and if i switch back to excel i'll see that i do indeed get my departures today now in this case we were lucky that the web page won the race between the vba code and the departures table so the departures table was updated and made visible before we attempted to get the information out of it in the vba code but that's not always going to be the case um it's difficult to demonstrate again it is possible that if i run this subroutine again to do the exact same thing without making any changes whatsoever that the next time i run it i don't actually get the departures information because the table hadn't appeared before my code executed as it turns out it's worked again perfectly happily but i can't always guarantee that so it's always worthwhile thinking about whether an element will be visible or not or whether it will be available to your code before you attempt to do something with it let me just tidy up this set of worksheets by selecting and then deleting them all and then i want to make sure that the departures table body is visible before i attempt to copy the information out of it into an excel worksheet so with departures today selected i can right click on any element within that table and choose to inspect it and i'll see that below the head section there's the t body section with a unique id of departures so i'm going to right click on that id and choose copy and then copy selector and then back in the vb editor i want to make sure that that element is displayed before i attempt to copy anything from it so to make that work i after i've clicked my departures button i'm going to say flight tables 2 dot find element by css so i'm going to look for a specific element inside that table and then in some round brackets and double quotes paste in the selector i've just copied close the double quotes in the round brackets and then a full stop wait displayed just as we saw earlier on with the wikipedia example so that gives me that reassurance that the table will definitely be visible before we attempt to copy out of that table into excel and just one last little test of that it gets a bit boring waiting for it to run through all of its different clicks but once again we end up with a worksheet with today's departures and we can have a quick visual check that that has the correct information displayed on it okay so we have one more table to get and that's tomorrow's departures and we should be able to use the the same basic technique we used for getting tomorrow's arrivals we can find the button to click on maybe wait for a second to give the table chance to update and then just copy the information from version or from the second instance of the table in the flight tables collection so in fact let's just do a quick copy paste of a bunch of that code i'm going to do that now so we can copy and paste that down to the bottom after tabat departures for today and then i'll update the flight table to the number two and then the word arrivals to the word departures if we attempt to run that subroutine at this point we'll see and it goes through its clicks gets to tomorrow goes to departures and today and then it doesn't appear to click on tomorrow's departures there's a good reason for that um if i click into the background of my code i'll see that i've got a runtime error it's telling me that there's an element not visible error and an element not interactable if i click debug it shows me that it's where i'm trying to click on the to button the tomorrow button and this is not immediately obvious what's gone wrong here if i just stop running the subroutine and just head back to my original instance of chrome when we're on the arrivals page the tomorrow button as we saw and we've correctly identified is nested in an ally element with an id of two when we're on the departures page however it isn't obvious that the web page has changed but this is a actually a different set of buttons the tomorrow button here isn't the same one this one is nested inside an li element whose id is 2 2. so as well as the departures table being invisible the buttons which control the departure table are also now invisible thus i can't click on them to interact with them and of course the one that i want to click on has a different id that the change to make to make the code work is incredibly simple but the cause of it is a little less obvious if you were just trying to observe what's happened on the web page as you click between buttons so the solution nice and simple i can change two to two two if that made any sense whatsoever and then if i just clear up the worksheets that have been generated and then run the subroutine one more time we'll go through all the clicks we get today's arrivals first of all we get tomorrow's arrivals today's departures and tomorrow's departures and if we switch back to excel we should find tomorrow's departures today's departures tomorrow's arrivals and today's arrival so we have all four sheets that we wanted to get so there we have it the basic techniques for weighting implicitly and explicitly in selenium of course there are some more complicated techniques we can use which we hinted that earlier on but at this point i think i'll save those for a later video hope you found this one useful thanks for watching see you next time

Channel: WiseOwlTutorials

Views: 4,668

Rating: 4.9191918 out of 5

Keywords: vba, selenium, visual basic for applications, chrome, web scraping, scraping, websites, crawling, web crawling, wait, implicit wait, explicit wait, wait for element, wait for display, excel, microsoft excel, wise owl, webdriver, chrome driver, google chrome

Id: ii1LxfEfY44

Length: 48min 0sec (2880 seconds)

Published: Thu Jan 28 2021